.. _PostProcessing_index: Demo: Post-processing ~~~~~~~~~~~~~~~~~~~~~~~~~~~ **Scenario**: - A disk exists with ADCP data collected at sea by UHDAS. - The :term:`processing directory` is ready for the next steps - Everything seems to have worked; the data look good. All that remains is to edit out in-port data and a few bad profiles, and apply a small calibration correction. **Resources** - `command-line help <../../../txtfiles/quick_postproc_help.txt>`_: ``quick_adcp.py --commands postproc`` - `text file <../../../txtfiles/km1001c_postproc_os38nb.txt>`_ (with metadata) from actual `final processing directory <../../../../adcp_py3demos/adcp_pyproc/km1001c_uhdas/os38nb_postproc>`_ - the rest of this web page - Additional resources (links in this documentation) are: + `quick_adcp.py detailed documentation <../../CODAS_docs/CODAS_quickdoc.html>`_ + `dataviewer.py documentation <../../dataviewer/index.html>`_ (viewing and editing) ---- .. NOTE:: **Directory Strategy** To simplify the documentation, these instructions assume you have set up your directories as in the :ref:`Directory-Setup` section, and you are doing your processing demos in **my_adcp_py3demos/adcp_pyproc** :: your practice directory source ----------------------- ----------- my_adcp_py3demos my_adcp_py3demos/adcp_pyproc (new, WORKING IN HERE) my_adcp_py3demos/uhdas_data copy of adcp_py3demos/uhdas_data my_adcp_py3demos/vmdas_data copy of adcp_py3demos/vmdas_data my_adcp_py3demos/uhdas_style_data (new, empty, for VmDAS conversion) ---- (1) Identify the :term:`CODAS Averages` from the disk. These directories are usually located under the ``CRUISE_NAME/proc/`` folder and are named after their associated sonar (e.g. ``km1001c/proc/os38nb``). (2) In your work area (``adcp_pyproc``), make a :term:`Project Directory` for all tutorials related to the cruise km1001c data.:: cd ~/my_adcp_py3demos/adcp_pyproc mkdir km1001c_uhdas cd km1001c_uhdas pwd The result should be:: /home/adcpproc/my_adcp_py3demos/adcp_pyproc/km1001c_uhdas (3) This the postprocessing demo, so we want to use the at-sea preliminary processed data from the cruise data. Copy the whole ADCP processing directory for this example (``os38nb``) to your postprocessing location **Copy this directory**:: ~/my_adcp_py3demos/uhdas_data/km1001c/proc/os38nb **to this location**:: ~/my_adcp_py3demos/adcp_pyproc/km1001c_uhdas/os38nb_postproc .. NOTE:: We will use the **cp** (copy) command. If you want to see what is happening in your command, add the **verbose** option:: cp -av .. From the **project directory** (**km1001c_uhdas**) run this:: cp -a ../../uhdas_data/km1001c/proc/os38nb os38nb_postproc ls The result should be:: os38nb_postproc .. Now you should have: .. image:: ../procfigs/postprocessing_step3.png :alt: copy preliminary processing directory to project directoru :scale: 60 (4) For each **processing directory**, keep notes in a **text file** (not a word-processor file) about your processing, eg. ``os38nb_postproc.txt``. This should be **adjacent to** the processing directory. (5) We will pre-fill the text file with contents of the file ``os38nb_postproc/cruise_info.txt``. Valuable metadata reside in the cruise_info.txt file. It's not complete, but it's a start. From the **project directory** ``km1001c_uhdas``, type this:: cp os38nb_postproc/cruise_info.txt os38nb_postproc.txt ls The result should show:: os38nb_postproc os38nb_postproc.txt Now you should have: .. image:: ../procfigs/postprocessing_step5.png :alt: copy metadata file "cruise_info.txt" from processing directory :scale: 60 You can use a variety of editors, but one which we install on all CODAS virtual machines is ``emacs``. Emacs is a powerful editor that includes a standard graphical interface. The **&** at the end gives you the command-line prompt back. :: emacs os38nb_postproc.txt & .. **Instructions** **start making notes** Write this in your text file: :: Cruise km1001c processing: The os38nb at-sea processing directory was copied from the "proc" directory of the km1001c cruise disk. All postprocessing is done with full codas+python (i.e. using numpy+matplotlib) (1) make a **project directory** ``km1001c_uhdas`` for work done related to this cruise - copy km1001c/os38nb directory from uhdas_data directory into km1001c_uhdas - rename it os38nb_postproc (to distinguish it from the other demos) - start editing a text file called os38nb_postproc.txt, with the contents of os38nb/cruise_info.txt at the top. - keep notes down below, and fill in above later Look at the evidence in the processing directory to see if this is what we expect, look for problems, evaluate what needs to be done. Write this down: you want to be able to remember what you did. Your notes should be inside the **project directory** so if you need to delete the processing directory, you don't lose your notes as well! This is our goal for the final text file: (`here <../../../txtfiles/km1001c_postproc_os38nb.txt>`_) ---- The last thing we will do is to make a **COPY** of the processing directory before we start changing things, so we can compare the effects of our work:: cp -a os38nb_postproc os38nb_postproc_orig .. .. NOTE:: Now we switch to working inside the **processing directory**. All commands related to editing or calibration of the dataset are run from the processing directory, ``os38nb_postproc`` unless otherwise noted -- go there now .. :: cd os38nb_postproc pwd The result should be:: /home/adcpproc/my_adcp_py3demos/adcp_pyproc/km1001c_uhdas/os38nb_postproc (1) Look at the cruise track to see what we're expecting:: plot_nav.py nav/a_km.gps This is a cruise with a "patch test" to calibrate the multibeam. The ship started this "cruise" near the patch test site, did the patch test, then came down towards Newport, Oregon and did a reciprocal run in towards shore and back out, then proceeded to port. There should be ample watertrack calibration points from the patch test. **Click on the image to enlarge it** .. image:: km1001c_nav.png :alt: postprocessing demo: nav plot :scale: 50 (2) Check accurate heading device (POSMV): Look at figures in ``cal/rotate/*hcorr.png`` for gaps (gaps would be red "+" signs -- there are none) :: figview.py cal/rotate/*png .. .. NOTE:: If your cruise is old enough that it does not have png files, you can generate them using ``quick_mplplots.py``. It is often a good idea to do this. To practice on this directory, and to **correct a bug** in the watertrack calibration, run the following command:: quick_mplplots.py --yearbase 2010 --plots2run all --noshow and then run this command to view all the PNG plots:: figview.py .. Check processing: (1) look at all figures (``figview.py``) (2) look at heading correction: are there gaps? does it need to be patched? (3) look at watertrack and bottom track calibrations: what can we expect? **Are there gaps in the heading device?** .. NOTE:: Gaps are in the heading correction are indicated in the plots (``cal/rotate/ens_hcorr*.png``) by red ``+`` signs, and good data are green dots. We use ``patch_hcorr.py`` to interpolate heading corrections. This cruise does not need the correction, but the example for patching the heading correction is `here <../../patch_hcorr/index.html>`_ | There are no gaps in the heading correction file. Add this to our notes: :: Conclude: no action needed regarding heading correction .. **Figures to look at** +------------------+-----------------------+--------------------+ | **directory** | **files** | information | +==================+=======================+====================+ | nav | ``nav_plot.png`` | cruisetrack | +------------------+-----------------------+--------------------+ | nav | ``reflayer_*.png`` | reference layer | +------------------+-----------------------+--------------------+ | edit | ``temp_plot.png`` | temperature | +------------------+-----------------------+--------------------+ | edit | ``nping_plot.png`` | pings per average | +------------------+-----------------------+--------------------+ | cal/rotate | ``hcorr_mpl_*.png`` | heading correction | +------------------+-----------------------+--------------------+ | cal/watertrk | ``wtcal*.png`` | watertrack calib | +------------------+-----------------------+--------------------+ | cal/botmtrk | ``btcal.png`` | bottomtrack calib | +------------------+-----------------------+--------------------+ ---- Based on the cruise track, we expect watertrack calibration values from the multibeam patch test. We probably do not have any bottom-track data. (3) check calibration: (a) run this command, then add part (or all) of the record displayed, to the documentation:: tail -20 cal/watertrk/adcpcal.out The important part of the results are: :: **watertrack** Number of edited points: 32 out of 36 amp = 1.0080 + -0.0027 (t - 30.3) phase = -0.01 + 0.0589 (t - 30.3) median mean std amplitude 1.0060 1.0080 0.0085 <--- slight scale factor phase 0.0775 -0.0058 0.4343 <--- no phase adjustment ----------------- .. Look at the watertrack figures:: figview.py cal/watertrk/*png .. There is no bottomtrack data - action: watertrack suggests a slight scale factor might be applied after editing - calibration is "close enough" to allow editing at this point .. NOTE:: HINT: The CODAS virtual machine has three aliases to help speed up the discovery process looking at the calibrations. `This link <../../../txtfiles/calibration_aliases.txt>`_ shows the actual bash aliases. You can type these shortcuts to look at the calibrations, which really speeds things up. .. :: shortcut: looking for what it does (but prettier) -------- ------------ ----------------------------- catwt watertrack cal tail -20 cal/watertrk/adcpcal.out catbt bottomtrack cal tail -6 cal/botmtrk/btcaluv.out catxy ADCP-gps offset tail -6 cal/watertrk/guess_xducerxy.out .. ---- View the data using:: dataviewer.py ---- **Compatibility with older cruises** Because we are post-processing an older directory (Matlab processing), there is a file which contains crucial metadata, and it is missing. `This link <../../older_cruises/postprocessing/index.html>`_ explains in more detail. Before we can edit or correct calibrations, we need to bring the dataset into compliance with modern tools, and generate the metadata file ``dbinfo.txt``. This is the command we want to run, but it will fail because ``dbinfo.txt`` is missing: ``quick_adcp.py --steps2rerun calib:navsteps --auto`` .. NOTE:: The program ``quick_adcp.py`` will complain about missing some information. At each error, include the information in the quick_adcp.py command line until it runs. This note explains the missing pieces; a completed command to run is at the bottom. :: Each time you try to run quick_adcp.py, it will complain. To speed things up, the errors and answers are summaraized here: ERROR solution: add to commands ERROR -- must select "datatype" "--datatype uhdas" ERROR -- must set "sonar" "--sonar os38nb" ERROR -- must set "yearbase" "--yearbase 2010" ERROR -- must set "ens_len" "--ens_len 300" Look in the file adcpdb/a_km.cnh to determine the number of seconds used for the averaging. We need the column "SI" for "sampling interval" ERROR -- must set "cruisename" "--cruisename km1001c" For post-processing, the cruise name is just used for titles ERROR -- must set "beamangle" "--beamangle 30" Usually, beamangles are: os38, os75, os150 30 wh300, wh600, wh1200 20 bb75, bb150, bb300 30 nb150, nb300 30 - now try again. We are running this from the post-processing directory: (e.g. pwd: /home/adcpproc/my_adcp_py3demos/adcp_pyproc/km1001_uhdas/os38nb_postproc) .. In the end, this is the command we need to run:: quick_adcp.py --steps2rerun navsteps:calib --datatype uhdas --sonar os38nb --beamangle 30 --yearbase 2010 --ens_len 300 --cruisename km1001c --auto That's the only time we need to run such a long command for this cruise. ---- **Dataviewer.py edit mode** For this cruise, the main thing is to remove the profiles at the end, when the water is very shallow. To do so, use dataviewer.py in edit mode:: dataviewer.py -e .. For more details on how to use dataviewer in edit mode, follow `this link <../../dataviewer/index.html>`_. After editing, we want to know what the new calibration is. Did it change? Is there still some phase to apply? or scale factor? We need to run `quick_adcp.py <../../quickadcp_overview.html>`_ to do the rest of the postprocessing, but this is an old cruise and there is a file with metadata that is missing. When you are done editing, run this command to recalculate the calibrations:: quick_adcp.py --steps2rerun navsteps:calib --auto .. ----- Now look in ``cal/watertrk/adcpcal.out`` for the calibration: ``tail -20 cal/watertrk/adcpcal.out`` Here is the new block (your results might be slightly different, depending on what you edited out) :: **watertrack** ------------ Number of edited points: 32 out of 35 amp = 1.0080 + -0.0027 (t - 30.3) phase = -0.01 + 0.0589 (t - 30.3) median mean std amplitude 1.0060 1.0080 0.0085 phase 0.0775 -0.0058 0.4343 ------------ Now we are ready to apply the final calibration: no change to phase, amplitude (scale factor) of 1.007. 5) Apply final calibration. Write this in the text file:: There are 32 points, enough for reasonable statistics # A phase correction is not warranted because the # mean and median are under 0.1 degree. If the phase # values above had said X.YY, then we would include # this in the quick_adcp.py command: # # --rotate_angle X.YY # # But a scale factor ("amplitude") should be applied Apply the correction:: quick_adcp.py --steps2rerun rotate:apply_edit:navsteps:calib --rotate_amplitude 1.007 --auto Check again to make sure we did not apply the wrong values. Now: - phase should be very close to zero (within 0.05deg) - amplitude should be very close to 1.00 (within 0.3%):: **watertrack** ------------ Number of edited points: 32 out of 36 amp = 1.0011 + -0.0028 (t - 30.3) phase = -0.01 + 0.0589 (t - 30.3) median mean std amplitude 0.9990 1.0011 0.0084 phase 0.0810 -0.0081 0.4342 ------------ Looks good. Look at the figures too:: figview.py cal/watertrk/wtcal1.png ---- We're near the end now Review the data: - Check calibration one more time (try looking in ``cal/botmtrk/btcaluv.out`` and ``cal/watertrk/adcpcal.out``. Note that these files might not exist) - Look at the figures again: ``figview.py`` - Check editing again (``dataviewer.py``). You should be able to see every bin and every profile. This is dependent on your screen-resolution: make the time step smaller if necessary. Close the dataviewer Control window to close the application. - Add comments to your processing documentation about + interesting features, problems + final (complete) calibration values used | Now make web plots (`this link <../../quick_web/index.html>`_ has more information). 6) make web plots:: quick_web.py --interactive .. - view with a browser, look at webpy/index.html When you're satisfied, extract the data for other people to use. Although we are not processing the data with Matlab, we can still write simple Matlab files using an older compatible format. For those who use Matlab or are not familiar with NetCDF files, this is the most common mechanism for looking at the data. These files are documented `here <../../../ADCP_INTERPRETATION/adcp_access/READING/MATLAB/index.html>`_, in the CODAS documentation, in the section concerning ways to access ADCP data. | (7) Data products: Extract matlab files:: quick_adcp.py --steps2rerun matfiles --auto You can extract NetCDF files as well:: adcp_nc.py adcpdb contour/os38nb km1001c_demo os38nb --ship_name Kilo Moana To check netcdf file:: ncdump -h contour/os38nb.nc Type this for help:: adcp_nc.py --help | The contents of the netCDF file we generate is documented `here <../../../ADCP_INTERPRETATION/adcp_access/READING/NetCDF/index.html>`_, in the CODAS documentation, in the section concerning ways to access ADCP data. ---- .. NOTE:: CODAS processing uses zero-based decimal days. See `CODAS Conventions <../../../../UHDAS_OPERATIONS/UHDAS_atsea/Conventions.html>`_ ---- .. NOTE:: When you are done with all the processing, with permission from the Principal Investigator, we encourage you to submit the data to the `Joint Archive for Shipboard ADCP `_ so you get credit for your data collection and processing, and other people can use it too. ---- | **Available Demos** - :ref:`Post-processing` - adcp_database_maker.py * :ref:`UHDAS and adcp_database_maker.py` * :ref:`LTA Processing with adcp_database_maker.py` * :ref:`ENR processing with adcp_database_maker.py` * :ref:`LTA Postprocessing` (located here, because it naturally follows the previous [LTA and ENR] demos) - commandline details * :ref:`UHDAS Single-ping Commandline` * :ref:`LTA processing: automated command-line` * :ref:`LTA command-line processing` * :ref:`ENR Single-ping Command Line`