.. ...................................... .. UHDAS/CODAS restructured text document .. ...................................... Understanding Editing (Gautoedit + CODAS) ------------------------------------------ Gautoedit contribution ~~~~~~~~~~~~~~~~~~~~~~~ Gautoedit (like the original "waterfall plots") has two kinds of editing: thresholds, and manual. Both append data to ascii files. These files contain the timestamp of the bad profile and information about the flagging. **Gautoedit output** +--------------+-----------------------------------------+ | file | file contains ... | | | | +==============+=========================================+ | abadbin.asc | timestamp and list bins to flag | +--------------+-----------------------------------------+ | abadprof.asc | timestamp of whole profile to flag | +--------------+-----------------------------------------+ | abottom.asc | bin identified as 'bottom' | +--------------+-----------------------------------------+ Gautoedit appends flag information to these files. But the information is not transferred to the database until you run :: quick_adcp.py --steps2rerun apply_edit ... .. There are two types of editing available: Threshold Editing ^^^^^^^^^^^^^^^^^^ "Threshold editing" refers the adjustment and application of values in the colored boxes along the left side of the gautoedit window. To see how this works, try adjusting "jitter": - decrease "jitter" - click "show now" - repeat Any of the colored values on the left can be adjusted and when you clock "show now" it will show you the effect of those thresholds, i.e. what you see (from the gautoedit editing) is what you will get (in the database) if those thresholds are applied. You **MUST** click "list to disk" or the flags will not be appended to the ascii files. Click thumbnail for a larger picture .. ------------------------------------------- .. |threshold_editing| image:: threshold_editingT.png :alt: scattering layer bias .. _threshold_editing: threshold_editing.png .. ------------------------------------------- |threshold_editing|_ Manual (mouse-driven) Editing ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ There are basically 4 manual editing buttons: - **del bad times** -- flag bad profiles * click individual profiles; hit to accept * click left and right time range - **rzap bins** -- rectanglular region slection - **pzap bins** -- polygon region selection * click slowly to mark vertices * double-click to finish (connects that vertex to the first) - **bottom** -- identify bottom (use amplitude to see the bottom) * click slowly to identify the bottom * don't guess about bottom blanking; just identify the bottom * double click to finish In all cases, when the tool is finished and the window exits, the flag information is written ascii file. Click thumbnail for a larger picture .. ------------------------------------------- .. |manual_gautoedit| image:: manual_gautoeditT.png :alt: scattering layer bias .. _manual_gautoedit: manual_gautoedit.png .. ------------------------------------------- |manual_gautoedit|_ Gautoedit notes ^^^^^^^^^^^^^^^^ - Only one gautoedit window can be open per matlab session - As long as the gautoedit window is open, it "remembers" the manual editing that has been done. "Show Now" applies the present thresholds in the boxes when it draws - The data are extracted to disk when the start time or time step is changed. All other "show now" operations on that collection of data simply read the data, but do noe extract it again. - Always kill and restart the gautoedit window after you apply the editing to the database. This will make the gautoedit window more in line with the match the flags in the database. - click the gray bar "do not show autoedit editing" to see what the flags are that com only from the database - click the gray bar "do not show effect of profile flags in database" to see the data that would otherwise be flagged in the database - (you can ignore the "bias parameter"; it was experimental) - (there is a bug when using depth as the vertical coordinate: the flagged rectangles show up in bin space, not meters; sorry) - You can use the following matlab command to read the data being shown: :: data=agetmat('a_'); .. **Remember** gautoedit only appends to ascii files; it does not put flags in the database. Getting flags into the database ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The quick_adcp.py step called **"--steps2rerun apply_edit"** performs several steps. These are performed in the edit/ subdirectory: (1) **dbupdate ../adcpdb/dbname abottom.asc** This step takes any bins identified as the bottom (stored in "abottom.asc") and identifies them in the database (2) **dbupdate ../adcpdb/dbname abadprf.asc** This step identifies bad profiles in the database by setting the "last good bin" value to -1. (3) **badbin ../adcpdb/dbname abadbin.asc** This step identifies bad bins in the database. A decimal value of 1 is put into "profile_flags" at this stage. (4) **set_lgb ../adcpdb/dbname 20** This step takes the bottom (from #1) and makes flags (decimal flag value of 4) below the bottom. It also masks data close to the bottom if the data are subject to side-lobe contamination (depends on beam angle). The default beam angle is 30 degres. If another number is specified, that is used (eg. "20" in this example) (5) **setflags setflags.tmp** This does two things: (i) takes the bad profile identifier and and gives every bin in that profile a "bad bin" flag (decimal value of 1) (ii) flags all values with percent good below the specified threshold (usually 30% or 50%) using a decimal value of 2 for the profile flag. Try using "showdb" to look at a database. The variable "profile_flags" shows the editing status of a given bin or profile. Bins are flagged in the database with a binary bit, depending on why they were flagged. This is a useful way to see whether data have been flagged or not. +--------+---------+---------------+---------------+-----------+ | binary | decimal | below bottom | percent good | bad bin | +========+=========+===============+===============+===========+ | 000 | 0 | | | | +--------+---------+---------------+---------------+-----------+ | 001 | 1 | | | bad | +--------+---------+---------------+---------------+-----------+ | 010 | 2 | | bad | | +--------+---------+---------------+---------------+-----------+ | 011 | 3 | | bad | bad | +--------+---------+---------------+---------------+-----------+ | 100 | 4 | bad | | | +--------+---------+---------------+---------------+-----------+ | 101 | 5 | bad | | bad | +--------+---------+---------------+---------------+-----------+ | 110 | 6 | bad | bad | | +--------+---------+---------------+---------------+-----------+ | 111 | 7 | bad | bad | bad | +--------+---------+---------------+---------------+-----------+ "Unediting" scenarios ----------------------- (1) starting over ~~~~~~~~~~~~~~~~~~ Example: You have a UHDAS dataset in which too much data were edited out If you need to start over, you will have to clear all profile flags, remove ascii files associated with editing, and do the editing from scratch. If all you need to do is add flags, just run gautoedit and remove the offending data (i.e. add flags to the existing ones). Here are three examples when you might need to return the flags to zero and start over - At-sea WH300 data were collected at 2-minute averages, but the default jitter threshold is for smoother data than that. The data in gautoedit will show lots of little white stripes where profiles were thrown out by the jitter criterion that was too low for this dataset - The gyro was replaced without any change in calibration and the headings are off by several degrees, so profiles at acceleration points (stop/start or turn) are flagged as bad because of the jitter criterion. - You are training someone else to process ADCP data and they flagged way too many things as bad. You do not need to redo all their processing, just the editing To start over with editing: - go to the edit directory - remove all \*.asc files - copy **setflags.tmp** to **clearflags.tmp** - edit **clearflags.tmp** as shown below - run "setflags clearflags.tmp" - start editing. Be sure to "list" before moving to "next" :: original (setflags.tmp) new (clearflags.tmp) --------------------------- --------------------------- dbname: ../adcpdb/a_demo dbname: ../adcpdb/a_demo pg_min: 50 pg_min: 50 set_range_bit clear_all_bits time_ranges: clear_range all time_ranges: all .. (2) change percent good ~~~~~~~~~~~~~~~~~~~~~~~~ You can change the percent good used by quick_adcp.py (eg. change to 30 instead of 50) by specifying **--pgmin 30** but you must do it every time you call quick_adcp.py. Otherwise it will revert to the default. To change percent good once, - go to the edit directory - **do not** remove any files - copy **setflags.tmp** to **setflags30.tmp** - edit **setflags30.tmp** as shown below - run "setflags setflags30.tmp" :: original (setflags.tmp) new (setflags30.tmp) --------------------------- --------------------------- dbname: ../adcpdb/a_demo dbname: ../adcpdb/a_demo pg_min: 50 pg_min: 30 time_ranges: time_ranges: all all .. Then run :: quick_adcp.py --yearbase YYYY --steps2rerun apply_edit --pgmin 30 --auto (3) Workhorse: (bugfix) recover a few deep bins ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ "We" (a vigilant user) discovered that bottom blanking was hardwired for 30deg, which is not correct for most Workhorse instruments. This bug only affects data collected when the bottom was in range. Data processed with code prior to May 1, 2009 will have had 15% of the range flagged near the bottom (cos(30deg)), but if the instrument has 20deg beams, the flagging should have been more like 5% (cos(20deg)). To recover the range 85%-95% of the water column depth from such a dataset - make sure your executable "set_lgb" accepts an argument for beam angle - if it does not, follow `these instructions <../codas_setup/hg_install.html>`_ to install new programs. - go to the edit directory - remove all \*.asc files - copy **setflags.tmp** to **clearflags.tmp** - edit **clearflags.tmp** as shown below - run "setflags clearflags.tmp" :: original (setflags.tmp) new (clearflags.tmp) --------------------------- --------------------------- dbname: ../adcpdb/a_demo dbname: ../adcpdb/a_demo pg_min: 50 pg_min: 50 set_range_bit clear_all_bits time_ranges: clear_range all time_ranges: all .. Now you have a dataset with PG<50 flagged as bad. - edit astup.m and add the following line to the bottom **config.beamangle = 20;** Then run :: quick_adcp.py --yearbase YYYY --steps2rerun apply_edit --auto (4) during a gautoedit session, too much was flagged ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Example - "list to disk" was clicked with the wrong thresholds and to much was flagged - the mouse slipped and the manual (graphical) editing took out too much In this case the flag information has been appended to the ascii files but they have not been put into the database. Recommended procedure: - figure out (in the three files) which lines are the flags that you changed your mind about. - open the file in an editor and take out those lines - apply the editing you have done so far :: quick_adcp.py --yearbase YYYY --steps2rerun apply_edit --auto .. - quit gautoedit and start again. Now gautoedit is consistent with the database.