The editing paradigm has undergone a slight change. Details of the new scheme, which includes “Reset Some Editing”, are located here
Python “gautoedit” is a program for extracting and displaying velocity and other ancillary data from a codas database in panel plots, selecting data to flag as ‘bad’, and applying those flags to the CODAS database. This is the third generation of graphical editing tools for CODAS data. In all cases, ascii files are generated, which contain information to flag data as ‘bad’. Information from these files is then applied to the database so that further data extraction will not include those flagged values.
This html documentation is meant to give you an introduction to the gautoedit package and provide some guidance as to its use. Figures may not look exactly as they do on your computer: some screenshots are taken under Mac OSX or Linux interfaces, and some figures illustrating editing concepts are retained from the earlier Matlab gautoedit tutorial.
The original “Gautoedit” was written in Matlab. The present version is written in Python. It has the same basic components:
|CODAS ADCP EDITOR: a GUI (Grahical User Interface): display and editing parameters are controlled here|
|PANEL PLOT (typically u,v, percent good, amp)|
|VECTOR PLOT of surface velocity over topography|
Component: editing tools:
- manual selection (Plot Tab)
- bottom identification
- threshold editing (Threshold Tab)
- adjust until you like what you see
- save flags to disk
Component: Work flow strategy:
- Edit a chunk of data, Apply Edits, go to the next time range
- repeat for subsequent time ranges, working through the whole cruise
- After going through the cruise
- review the data from the whole cruise again.
- repeat as needed.
When gautoedit is invoked, three windows appear. Here is an example of the window layout for editing (click thumbnail for a larger image):
CODAS ADCP editor:
- Run gautoedit from the edit subdirectory of your adcp processing tree
gautoedit.py ## or if you want 5 panels, do this gautoedit.py -n5
- Choose the number of days to display (based on your screen resolution), make sure you can see individual profiles.
- Press “Show” to load the data from that time range
Typical Editing Steps
|ACTION TO TAKE||WHAT IT DOES|
|(1) view a time range: (click “Show” or right arrow||
||resets flags for selected profiles to minimum Percent Good only (in database); Shows effect of thresholds in the view|
||if needed; click “show” to see effect|
||select additional bins (or profiles) - profile - rectangle - polygon (double-click to end selection) - lasso (freehand: cute, not very useful) - bottom (double-click to end selection)|
||applies thresholds and manual editing for present time range|
||move forward in time by “step” (with a slight overlap|
When Reset Some Editing is clicked:
minimum Percent Good only
what you see does include the effect of the threshold editing
When Apply Editing is clicked:
The *.asc files with flagging information can come from
If you want to know more about what is happening behind the scenes, look on this page.
Look at the data with different masking (select different collections of editing flags to display) This example is a newly-loaded LTA dataset; no editing has occurred except for a percent-good cutoff of 50% (PG)
- diagnostic: no flags (everything in the database)
- diagnostic: codas (whatever codas database flags have been set)
- usual: all (codas database flags as well as thresholds; this is what your dataset will look like if you click “Apply Editing”
Adjust threshold editing if necessary (Threshold Tab. see below); click “Show” to renew the plot
You might have data within range of the bottom.
First see how much the Threshold editing can remove
Then manually edit the rest
- Manually edit out the rest of the bad data associated with the bottom.
- The bottom editing tool always uses “amp” for editing
- The bottom edit tool takes the beam angle into account (side-lobe interference) so all you should have to do is choose the maximum amplitude (or a bin above that).
Other manual editing: in all cases, choose the editing type, perform selections until finished, and click “Save Manual Edits”
- profile: click and drag (left to right) – the time range will be implemented when the mouse button is released
- rectangle: click and drag to select the rectangle of bad bins – the rectangle will be selected when the mouse button is released
- polygon: acts like bottom identification: single clicks are vertices of a polygon, doubleclick joins the last point with the first point to close the polygon
- lasso: freehand draw
In all manual editing cases
After “Apply Editing” you can revert to just minimum Percent Good by using the button to Reset Some Editing (selected profiles are reverted).
Wire interference and ringing
wire interference - set the number of bins to evaluate - set the ship speed (only looks when ship is slower) - enable/disable: set Error Velocity cutoff (best
signature for wire interference) – this value is smaller than the general error velocity criterion below
- This algorithm finds bad bins, and then also flags each bin to the right and left, and above and below. It is designed to hollow out a little patch around a peak of high EV. It is not simply a threshold.
- wire interference example
- set the ship speed (only looks when ship is faster)
- enable/disable: set number of bins to reject
- ringing example
Thresholds for excluding individual bins
These are all disabled be default. None of them is good enough to really do the job, but they can be used together or as indicators of a problem
- error velocity: (good indicator of a problem)
- vertical velocity: (specialty: sometimes useful for bizarre cases of electrical interference)
- shallow low Percent Good: (can be useful for cases of ringing, bubbles, or ice). It cannot get rid of everything but is sometimes a good indicator (example)
- resid_stats_fwd: a measure of how consistent the averages are in the direction of ship’s travel. This is a diagnostic for intermittent underway bias at the single-ping level.
Percent Good in Reference Layer
This is a way to require a minimum number of high Percent Good bins in a reference layer in order to accept the profile. It can be useful in situations with intermittant bad weather and periods with few, isolated profiles
- choose the range of bins to asses
- choose the Percent Good cutoff for bins in that reference layer
- choose how many good bins are required for a profile to “pass”
Thresholds for excluding whole profiles
- jitter: somewhat arbitrary indicator of “data smoothness” (was more important in Matlab processing; Python processing is better and produces less “jittery” data
- neighbors: if profiles are are gappy, but profiles adjacent to the gaps seem bad, this will trim them
- ship speed: discard if ship speed exceeds this; useful if the heading device has poor accuracy
After you have reached a stopping point, eg. “the whole dataset”, run this command to ensure a clean sweep of any remaining ascii files, rerunning the navigation steps and updating calibration: (This is done from the processing directory)
quick_adcp.py --steps2rerun --apply_edit:navsteps:calib --auto
Now look at the dataset again dataviewer.py to be sure you edited out all the bad parts. There’s often something left, so it’s worth checking.