. ...................................... .. UHDAS/CODAS restructured text document .. ...................................... .. _dataviewer-edit-mode: Edit Mode ---------------- This is the fourth generation of graphical editing tools for CODAS data. In all cases, data are flagged as 'bad' based on one of three criteria: - bad (due to some threshold) - below the bottom - manually excluded Once the 'bad data' are flagged in the graphical editor, a separate step to "apply editing" to the CODAS database involves writing the flags to ascii files and applying the information in those ascii files to the CODAS database so that further data extraction will not show those flagged values. This html documentation is meant to give you an introduction to the ``dataviewer.py`` and provide some guidance as to its use. Figures may not look exactly as they do on your computer: some screenshots are taken under Mac OSX or Linux interfaces, and some figures illustrating editing concepts are retained from the earlier Matlab ``gautoedit`` tutorial. This is a work in progress; some of the screenshots are still from the python2 version. Overview of the Windows ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The original editing tool was written in Matlab. The original Python version only works in Python2.7, because of the graphical library used (Wx). This version works with Python 3.6 using the Qt graphical library. It is more sophisticated than the Wx version, but retains the same strategy as its ancestors. | +----------------------------------+---------------------------------------------------------------------------+ | **CONTROL PANEL**: | .. image:: /images/dataviewer_edit_mode/edit_mode_dataviewer_controls.png | | display and editing | :alt: dataviewer control panel | | parameters are controlled here | :scale: 35 | +----------------------------------+---------------------------------------------------------------------------+ | **PANEL PLOT** (typically u,v, | .. image:: /images/dataviewer_edit_mode/edit_mode_dataviewer_panels.png | | percent good, amp) | :alt: dataviewer panels | | | :scale: 25 | +----------------------------------+---------------------------------------------------------------------------+ | **VECTOR PLOT** of surface | .. image:: /images/dataviewer_edit_mode/edit_mode_dataviewer_topofig.png | | velocity over topography | :alt: dataviewer topography | | | :scale: 35 | +----------------------------------+---------------------------------------------------------------------------+ The **work flow** below assumes each sonar processing directory has already had: - heading correction patched (if required using :ref:`patch_hcorr.py `) - calibration (phase and possibly amplitude using :ref:`quick_adcp.py `) applied - transducer-ADCP offsets applied (using quick_adcp.py) **Overview of editing work flow**: * For each sonar (eg. wh300 and os75nb): - Edit a chunk of data, click Apply Edits, go to the next time range. - Editing uses these two schemes: + manual selection (**Plot Tab**) + threshold editing (**Threshold Tab**) - Repeat for subsequent time ranges, working through the whole cruise - After going through the cruise + review the data from the whole cruise again. + repeat as needed. * After editing, adjust the phase amplitude calibrations again so they are as good as possible * Then step back and use **compare** mode to flag more data if that is necessary ----- More Details about each window ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ When ``dataviewer.py -e`` is invoked, three windows appear. Here is an example of the window layout for editing (click thumbnail for a larger image): .. image:: /images/dataviewer_edit_mode/edit_mode_km1001_postproc_layout_annot.png :alt: typical layout of dataviewer windows :scale: 30 | **Control Window**: The Dataviewer Edit Mode has 4 tabs. - **Plot**: (:ref:`control ` what is viewed) - **Thresholds**: control :ref:`threshold editing ` (see below) - **Log**: show time ranges requested so far - **Help**: a little help (under construction). Most help is in the web pages you are reading right now .. image:: /images/dataviewer_edit_mode/edit_mode_gui_plot_tab_annot0.png :alt: Tabs on Dataviewer Editor window :scale: 50 **Panel Window**: The only thing you can do in the :ref:`panel window ` is pan or zoom, but if Multicursor is enabled in the Control window, the red line in the panel plot matches the corresponding position in the topography window. .. image:: /images/dataviewer_edit_mode/edit_mode_dataviewer_panels.png :alt: Panel window :scale: 30 **Topography Window**: The :ref:`topgraphy window ` is designed to view ocean velocity vectors over ocean topgraphy. You can change some of the display settings (vector averaging in vertical and time, and length of vectors). You can pan and zoom (but the topography does not increase in resolution). If Multicursor is enabled in the Control window, the red line in the panel plot matches the corresponding position in the topography window. .. image:: /images/dataviewer_edit_mode/edit_mode_dataviewer_topofig.png :alt: Topgraphy window :scale: 50 ------ Work Flow (Editing ADCP data) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Starting out ++++++++++++++++ * Run ``dataviewer.py -e`` from your ADCP processing directory or the ``edit`` subdirectory. :: dataviewer.py -e .. * Choose the number of days to display (based on your screen resolution), make sure you can see individual profiles. * Press "Show" to load the data from that time range ,,,, **Typical Editing Steps** .. NOTE:: Note that the following are often equivalent: - clicking **Show** - after entering text, hitting the Return key | +-------------------------+--------------------------------------------+ | ACTION TO TAKE | WHAT IT DOES | +=========================+============================================+ | (1) View a time range: | Shows the chosen time range with | | (click **Show** or | thresholds updated | | right-arrow button) | | +-------------------------+--------------------------------------------+ | (2) Adjust thresholds | You can enable/disable threshold editing, | | if needed | and you can adjust values | +-------------------------+--------------------------------------------+ | (3) Click **Show** | Look at the effects of the thresholds | | | - experiment with "Masking": | | | - no flags (none at all) | | | - codas (what the database has) | | | - all (show effect of thresholds) | +-------------------------+--------------------------------------------+ | (4) Edit out data below | Thresholds: enable, then tweak | | the bottom using | Seabed Selector: click on the bin with | | thresholds or | hightest amplitude; doubleclick to stop. | | Seabed Selector | Sidelobe effects **ARE** accounted for. | +-------------------------+--------------------------------------------+ | (4) Manually edit | Select additional bins (or profiles) | | using Selectors: | - rectangle | | | - horizontal span (move left-right) | | | - vertical span (move up/down) | | | - one-click (zoom, then center of bin) | | | - polygon (click starting point to end) | | | - lasso (freehand; lift finger to end) | +-------------------------+--------------------------------------------+ | (5) click Apply Editing | Applies thresholds and manual editing | | | for present time range | +-------------------------+--------------------------------------------+ | (6) Reset Some Editing | Resets flags for selected profiles to | | | minimum Percent Good only (in database); | | | Shows effect of thresholds in the view | +-------------------------+--------------------------------------------+ | Repeat above | Assume this will take two passes. Flag | | if needed | obviously bad values; do not yet flag | | | subtle problems. Investigate later | +-------------------------+--------------------------------------------+ | (7) Show next | Move forward in time by "step" (with a | | | slight overlap | +-------------------------+--------------------------------------------+ **NOTES** - When **Apply Editing** is clicked: * flags from files ending in ``*.asc`` are applied to the database. * flags from all ``*.asc files`` are then appended to ``*.asclog`` * all ``*.asc`` files are deleted - The ``*.asc`` files with flagging information can come from * earlier editing session (automatic editing during at-sea processing, older editing session) * the present state of threshold settings Manual Editing Examples ++++++++++++++++++++++++++++++ (1) Look at the data with :ref:`different masking ` (select different collections of editing flags to display) This example is a newly-loaded LTA dataset; no editing has occurred except for a percent-good cutoff of 50% (PG) - diagnostic: *no flags* (shows everything in the database) - default: *codas* (shows data in CODAS database that is not flagged) - diagnostic: *all* flags are applied, CODAS flags and threshold flags. This helps visualize what your dataset will look like if you click "Apply Editing". (2) Adjust **threshold editing** if necessary (see below); click "Show" to renew the plot (3) You might have velocities **below the bottom**. First see how much the *Threshold* editing can remove - Try turning on :ref:`Bottom Identification ` - Try :ref:`adjusting ` the bottom-detection parameter. Or you can use the :ref:`Seabed Selector ` to manually identify the bottom: - Manually edit out the bad data associated with the bottom. + The bottom editing tool always uses "amp" for editing + The bottom edit tool takes the beam angle into account (side-lobe interference) so all you should have to do is choose the maximum amplitude (or a bin above that). 4) Editing by bins or profiles using **Selectors** .. NOTE:: Select a bin or profile by grabbing the **middle**, not a corner. The :ref:`Selectors ` window has - dropdown menu for variables plotted, including ocean **u**, **v**, percent good, signal return, and others. - dropdown menu for editing tools: - **Rectangle**: click and drag to select the rectangle of bad bins -- the rectangle will be selected when the mouse button is released. - **Horizontal Span**: click and drag the mouse horizontally to select profiles - **Vertical Span**: click and drag the mouse vertically to remove all the bins in a specific range - **One click**: click in a bin - **Polygon**: Use single clicks to create a polygon. To finish, go back to the first point and click (it will "grab" that point if you get close enough) - **lasso**: freehand draw. Lift the mouse button to release, and it will connect to the beginning of the curve. .. NOTE:: In the Selector window - you can add edits using any of the tools or variables. * use **Mask** mode to ADD flags * use **Unmask** mode to SUBTRACT flags - If you don't like any of your editing you can * close the manual editing window without doing the Stage Editing step * click "Clear Edits" to reset to its initial values - the flags are not applied to the database until you click "Apply Editing" in the Control Windwow. .. NOTE:: After "Apply Editing" you can revert to just minimum Percent Good by using the button to :ref:`Reset Editing ` (selected profiles are reverted). :ref:`More detail here` .. NOTE:: Sometimes you need to reset editing manually. Instructions for that are in :ref:`this link` Threshold editing ++++++++++++++++++ * **Wire interference and ringing** *wire interference* - set the number of bins to evaluate - set the ship speed (only looks when ship is slower) - enable/disable: set Error Velocity cutoff (best signature for wire interference) -- this value is smaller than the general error velocity criterion below - This algorithm finds bad bins, and then also flags each bin to the right and left, and above and below. It is designed to hollow out a little patch around a peak of high EV. It is not simply a threshold. - :ref:`wire interference example ` *ringing* - set the ship speed (only looks when ship is faster) - enable/disable: set number of bins to reject - :ref:`ringing example ` * **Thresholds for excluding individual bins** These are all disabled be default. None of them is good enough to really do the job, but they can be used together or as indicators of a problem - *error velocity*: (good indicator of a problem) - *vertical velocity*: (specialty: sometimes useful for bizarre cases of electrical interference) - *shallow low Percent Good*: (can be useful for cases of ringing, bubbles, or ice). It cannot get rid of everything but is sometimes a good indicator (:ref:`example `) - *resid_stats_fwd*: a measure of how consistent the averages are in the direction of ship's travel. This is a diagnostic for intermittent underway bias at the single-ping level. * **Percent Good in Reference Layer** This is a way to require a minimum number of high Percent Good bins in a reference layer in order to accept the profile. It can be useful in situations with intermittant bad weather and periods with few, isolated profiles - choose the range of bins to asses - choose the Percent Good cutoff for bins in that reference layer - choose how many good bins are required for a profile to "pass" * **Identify Bottom** - Try turning on :ref:`Bottom Identification ` - Try :ref:`adjusting ` the bottom-detection parameter. * **Thresholds for excluding whole profiles** - *jitter*: obsolete: a somewhat arbitrary indicator of :ref:`"data smoothness" ` (was more important in Matlab processing; Python processing is better and produces less "jittery" data - *neighbors*: if profiles are are gappy, but profiles adjacent to the gaps seem bad, this will trim them - *ship speed*: discard if ship speed exceeds this; useful if the heading device has poor accuracy Interpretation: things to keep in mind ++++++++++++++++++++++++++++++++++++++++++ * :ref:`rotation misalignment ` * :ref:`scale factor ` * :ref:`underway bias ` * :ref:`horizontal offset between ADCP and GPS ` Finishing up ~~~~~~~~~~~~~~~~ After you have reached a stopping point, eg. "the whole dataset", run this command to ensure a clean sweep of any remaining ascii files, rerunning the navigation steps and updating calibration: (This is done from the processing directory) Now look at the dataset again using ``dataviewer.py`` to be sure you edited out all the bad parts. There's often something left, so it's worth checking. (Return to :ref:`TOP`) .. toctree:: :hidden: .. phase .. amplitude .. underway_bias .. manual_undo_editing .. xducer_dxdy