Gautoedit Documentation

  • Pronounced “Gee! Autoedit!”
  • a graphical editing tool for CODAS ADCP data

Note

The editing paradigm has undergone a slight change. Details of the new scheme, which includes “Reset Some Editing”, are located here

Introduction

Python “gautoedit” is a program for extracting and displaying velocity and other ancillary data from a codas database in panel plots, selecting data to flag as ‘bad’, and applying those flags to the CODAS database. This is the third generation of graphical editing tools for CODAS data. In all cases, ascii files are generated, which contain information to flag data as ‘bad’. Information from these files is then applied to the database so that further data extraction will not include those flagged values.

This html documentation is meant to give you an introduction to the gautoedit package and provide some guidance as to its use. Figures may not look exactly as they do on your computer: some screenshots are taken under Mac OSX or Linux interfaces, and some figures illustrating editing concepts are retained from the earlier Matlab gautoedit tutorial.

Gautoedit Overview

The original “Gautoedit” was written in Matlab. The present version is written in Python. It has the same basic components:

Component: Windows  
CODAS ADCP EDITOR: a GUI (Grahical User Interface): display and editing parameters are controlled here control
PANEL PLOT (typically u,v, percent good, amp) panels
VECTOR PLOT of surface velocity over topography topo

Component: editing tools:

  • manual selection (Plot Tab)
    • profile
    • rectangle
    • polygon
    • bottom identification
  • threshold editing (Threshold Tab)
    • adjust until you like what you see
    • save flags to disk

Component: Work flow strategy:

  • Edit a chunk of data, Apply Edits, go to the next time range
    • repeat for subsequent time ranges, working through the whole cruise
  • After going through the cruise
    • review the data from the whole cruise again.
    • repeat as needed.

Gautoedit Windows

When gautoedit is invoked, three windows appear. Here is an example of the window layout for editing (click thumbnail for a larger image):

typical layout of gautoedit windows

CODAS ADCP editor:

Tabs on Gautoedit GUI Editor window
The CODAS ADCP Editor window has a collection of tabs:
  • Plot: ( control what is viewed)
  • Thresholds: control threshold editing
  • Log: show time ranges requested so far (**)
  • Show Zapped: shows time ranges (**) for manual editing
  • Show Applied: shows time ranges (**) when “Apply Editing” was clicked
  • (**) NOTE These pages only update when you click to view them, they do not update while you watch (sorry)

Editing ADCP data

Starting out

  • Run gautoedit from the edit subdirectory of your adcp processing tree
gautoedit.py

## or if you want 5 panels, do this
gautoedit.py -n5
  • Choose the number of days to display (based on your screen resolution), make sure you can see individual profiles.
  • Press “Show” to load the data from that time range

Typical Editing Steps

ACTION TO TAKE WHAT IT DOES
(1) view a time range: (click “Show” or right arrow
shows a time range with thresholds applied
  • present time range
  • next time range (overlaps previous)
  1. Reset Some Editing
resets flags for selected profiles to minimum Percent Good only (in database); Shows effect of thresholds in the view
  1. adjust thresholds
if needed; click “show” to see effect
  1. click Show
look at the effects of the thresholds
  • experiment with “Masking”:
    • no flags (none at all)
    • codas (what the database has)
    • all (show effect of thresholds)
  1. manually edit
select additional bins (or profiles) - profile - rectangle - polygon (double-click to end selection) - lasso (freehand: cute, not very useful) - bottom (double-click to end selection)
  1. click Apply Editing
applies thresholds and manual editing for present time range
  1. show next
move forward in time by “step” (with a slight overlap
NOTES
  • When Reset Some Editing is clicked:

    • flags for the selected profiles are reset in the database to

      minimum Percent Good only

    • what you see does include the effect of the threshold editing

  • When Apply Editing is clicked:

    • flags from files ending in *.asc are applied to the database.
    • flags from all *.asc files are then appended to *.asclog
    • all *.asc files are deleted
  • The *.asc files with flagging information can come from

    • earlier editing session (automatic editing during at-sea processing, older gautoedit session)
    • manual editing, if the “Save Manual Editing” button is clicked
    • the present state of threshold settings

Note

If you want to know more about what is happening behind the scenes, look on this page.

Typical sequence of steps

  1. Look at the data with different masking (select different collections of editing flags to display) This example is a newly-loaded LTA dataset; no editing has occurred except for a percent-good cutoff of 50% (PG)

    • diagnostic: no flags (everything in the database)
    • diagnostic: codas (whatever codas database flags have been set)
    • usual: all (codas database flags as well as thresholds; this is what your dataset will look like if you click “Apply Editing”
  2. Adjust threshold editing if necessary (Threshold Tab. see below); click “Show” to renew the plot

  3. You might have data within range of the bottom.

    First see how much the Threshold editing can remove

    Then manually edit the rest

    • Manually edit out the rest of the bad data associated with the bottom.
      • The bottom editing tool always uses “amp” for editing
      • The bottom edit tool takes the beam angle into account (side-lobe interference) so all you should have to do is choose the maximum amplitude (or a bin above that).
  1. Other manual editing: in all cases, choose the editing type, perform selections until finished, and click “Save Manual Edits”

    • profile: click and drag (left to right) – the time range will be implemented when the mouse button is released
    • rectangle: click and drag to select the rectangle of bad bins – the rectangle will be selected when the mouse button is released
    • polygon: acts like bottom identification: single clicks are vertices of a polygon, doubleclick joins the last point with the first point to close the polygon
    • lasso: freehand draw

Note

In all manual editing cases

  • you can add multiple segments before you click “Save Manual Edits”
  • you can close a manual editing window without saving (if you make a mistake).
  • the flags are not applied to the database until you click “Apply Editing”

Note

After “Apply Editing” you can revert to just minimum Percent Good by using the button to Reset Some Editing (selected profiles are reverted).

Threshold editing

  • Wire interference and ringing

    wire interference - set the number of bins to evaluate - set the ship speed (only looks when ship is slower) - enable/disable: set Error Velocity cutoff (best

    signature for wire interference) – this value is smaller than the general error velocity criterion below

    • This algorithm finds bad bins, and then also flags each bin to the right and left, and above and below. It is designed to hollow out a little patch around a peak of high EV. It is not simply a threshold.
    • wire interference example

    ringing

    • set the ship speed (only looks when ship is faster)
    • enable/disable: set number of bins to reject
    • ringing example
  • Thresholds for excluding individual bins

    These are all disabled be default. None of them is good enough to really do the job, but they can be used together or as indicators of a problem

    • error velocity: (good indicator of a problem)
    • vertical velocity: (specialty: sometimes useful for bizarre cases of electrical interference)
    • shallow low Percent Good: (can be useful for cases of ringing, bubbles, or ice). It cannot get rid of everything but is sometimes a good indicator (example)
    • resid_stats_fwd: a measure of how consistent the averages are in the direction of ship’s travel. This is a diagnostic for intermittent underway bias at the single-ping level.
  • Percent Good in Reference Layer

    This is a way to require a minimum number of high Percent Good bins in a reference layer in order to accept the profile. It can be useful in situations with intermittant bad weather and periods with few, isolated profiles

    • choose the range of bins to asses
    • choose the Percent Good cutoff for bins in that reference layer
    • choose how many good bins are required for a profile to “pass”
  • Identify Bottom

  • Thresholds for excluding whole profiles

    • jitter: somewhat arbitrary indicator of “data smoothness” (was more important in Matlab processing; Python processing is better and produces less “jittery” data
    • neighbors: if profiles are are gappy, but profiles adjacent to the gaps seem bad, this will trim them
    • ship speed: discard if ship speed exceeds this; useful if the heading device has poor accuracy

Interpretation: things to keep in mind

finishing up

After you have reached a stopping point, eg. “the whole dataset”, run this command to ensure a clean sweep of any remaining ascii files, rerunning the navigation steps and updating calibration: (This is done from the processing directory)

quick_adcp.py --steps2rerun --apply_edit:navsteps:calib --auto

Now look at the dataset again dataviewer.py to be sure you edited out all the bad parts. There’s often something left, so it’s worth checking.