University of Hawaii "currents" group:
UHDAS and CODAS processing links
A CODAS (Common Ocean Data Access System) database is a way to store
and access oceanographic data. "CODAS processing" refers to the steps
used to turn ADCP data into something valuable to science. "CODAS
processing" is like box of tools to manipulate data stored in
the CODAS database. Quick_adcp.py is like a "black box" for running
using these tools in a standard order for a variety of ADCP datasets.
This black box has removable side panel to allow the user to configure
and run all the underlying processing steps. The programs are Open
Source so nothing is hidden.
Documentation has accumulated over the years as acquisition programs,
instruments, and processing programs have changed. Some of the old
(but highly detailed) documentation contains references that are
obsolete, but the old data are still out there, and the same problems
keep occurring. Much of it is still used and highly relevant.
This page is an attempt to make access to the various generations of
docmentation easier and more direct.
-
FAQ (Frequently Asked Questions)
- Software downloading and setup
instructions
- Accessing data in
Matlab
- Of interest to people at sea
- ADCP data processing
- UHDAS links
- Some technical discussions
New Things present or changed in the current distribution:
(2008/06/31)
Changes:
- additional reorganization of python files
- work continues with python tools for access and plotting codas data
Improvements:
- continued efforts to improvement documentation
- added more variables to "getmat"
- added 64bit linux support
- more improvements to single-ping editing
(2007/05/31)
Improvements:
- improvements to documentation organization (including this page)
- added "persistence" of heading correction over small gaps (UHDAS
data)
- added a distinction between yearbase (for date calculation) and
proc_yearbase (in case it was specified badly, or to join two UHDAS
cruises that span a year boundary)
- More improvements to single-ping editing
Bug Fixes:
- rewrote sonar variable specifications for HDSS so they work
with Matlab R2007a
(2006/10/25)
Changes:
- The entire directory tree is being reorganized to allow for
better version control. This slightly changes the installation
instructions (mostly the user's executable and python paths).
- We are embarking on an effort to move away from the use of
Matlab in processing and display software. Over time, routines
presently performed in Matlab will be ported to Python. We will
attempt to keep the Matlab code running as Mathworks changes
Matlab, but some things will falter, and eventually we will not
maintain them. The present examples of this include the placement
of a colorbar with gautoedit, and some aspects of color mapping.
Improvements:
- Gautoedit polygon zapping and bottom editing tool was
improved for use with Matlab R14
- Codas database now stores cleaned averaged amplitude under 'amp'
and stores individual beam amplitudes under 'rawamp'; both
are accessible with "getmat"
- A new build system is being tested that uses python-based
"scons". If you wish
to compile the binaries for your system, download
codas3_declspec.zip and contact us for more information
about gotchas and what to expect. If you embark on this route
you will be a beta tester.
Bug fixes:
- Fixed: a bug in treatment of bottom depth resulted in
databases with noisy bottom depths were sometimes too large.
- Fixed: reference layers depths behaved badly if chosen to be shallow
- Fixed: inconsistent treatment of ensemble length caused problems for some long LTA averages
(2006/03/30)
New Additions:
- UHDAS documentation (acquisition, processing) added
(2005/10/17)
New Additions:
- improved heading-correction code
- split python processing scripts and executables into two groups for cleaner updates
(2005/05/16)
- experimental HDSS tools (see notes)
- UHDAS processing demo (in q_demos.zip)
- improving documentation for single ping data
(2004/12/30)
Major upgrades:
- Support for Matlab R14 (see notes)
- Support for Macintosh OSX binaries (download binosx.zip)
Minor changes:
- fixed a recent bug in quick_adcp.py that made 'steps2rerun' fail
with VmDAS data
- gautoedit buttons are more visible for 1024x768 screens
Features introduced in earlier
distributions:
(2004/05/18)
- Online
documentation of Codas3 and ADCP processing. This documentation
is also present in the zip file distribution "doc.zip", so you will
have it on your disk once you download all the appropriate zip
files. When creating a processing directory, there will also be
an html file "adcp_processing.html" containing a link to the local
processing documentation.
- A new variable-depth raference layer scheme "refsm" is now the
official standard for data with non-dithered GPS fixes and only short
gaps. It works well with cruises that have steep topography or patchy
depth
loss due to bubbles or hull noise. It does not do well if there
are
missing navigation points, as it currently interpolates missing
positions. "Smoothr" is still required for watertrack calibration and
reference
layer plotting, and can still be used for navigation.
- A new python program (quick_adcp.py)
can set
up the gautoedit
files and run through all the basic processing steps. It is useful for
a quick look
at a dataset and is sufficient for processing most shipboard ADCP
datasets. It can handle averaged data from DAS2.48 (pingdata files)
or VmDAS
(LTA or STA files), and can perform configurable single-ping editing on
UHDAS or VMDAS single-ping data (for example ENX). Quick_adcp.py
can be run at sea on a unix machine at regular intervals,
creating an up-to-date codas database for scientists. Editing of the
averaged data in the CODAS database can be automated
under these circumstances. Advantages to using quick_adcp.prl
include:
- standardized names
- impliments the new refsm reference-layer calculation
- automatic setup of gautoedit files
- works with pingdata or LTA files
- works with ENX or UHDAS single-ping files
- consistency in parameters
- steps are recorded in a log file
- many parameters can be changed on the command line or in a
control file
- makes it relatively painless to delete a database and start
over
- runs under MS Windows
- Improvements to the new editing package "
gautoedit" pronounced "Gee!autoedit!
" include improved automatic bottom detection, manual bottom
editing, and rectangle or polygon zap. This tool is a
graphical means of editing CODAS ADCP data (i.e. averaged
velocities) in user-determined time chunks (eg, 2-4 hours for messy
data; 1-3 days for clean data and a high resolution monitor). This tool
was designed to screen data for
things
like ringing, on-station wire interference, jittery navigation, and
bottom
interference when bottom tracking was not on. The old editing
(best done 20-30 profiels at a time) can still be used for fine-tuning
single-bin editing. This is also a useful
tool
for simply looking at a dataset, even if you don't use it for editing.
- Single-ping or averaged VmDAS data can be read directly into
matlab, allowing scientists to access specfic time ranges to comput
shears or to look at the backscatter amplitude. This readme may help.
- The older perl script (quick_adcp.prl) is no longer supported. It
has been ported to Python, and additional functionality has been added.
other tools:
- A fairly comprehensive but undocumented and unsupported plotting
tool
called "gplot" is available in our matlab distribution. Add the
"guiplots"
subdirectory to your path. Default paths are configured with the
expectation
that you will be running it from the adcp processing directory.
Specifically,
it will need to load adcpsect output files (*_xy.mat) from a contour or
vector
subdirectory, and can make nice panel plots of percent good and other
parameters
assuming your database is in an adcpdb subdirectory.
Caveats:
- Both adcptree.py and quick_adcp.py work under Unix and Windows
systems. For Windows users, there are .BAT files supplied which
access these two programs: adcptree_py.bat (for adcptree.py), and
quickadcp_py.bat (for quick_adcp.py)
- Refsm should only be used with 5-minute averages and high quality
GPS data
(i.e. not Selective Availability, and no gaps). Smoothr still works
fine if you are stuck with jittery fixes.
refsm will interpolate across gaps in navigation. This is
probably OK for 1-2 points. If the gap is 3-6 points, and if the ship
is not maneuvering, you can substitute the positions from smoothr.
For longer gaps, substitute navigation from other source. If you have
no navigation, edit the out profiles. Refsm leaves a log
"refsm_interp.log" in the current directory ("nav" if processing,
"edit" when editing)
Updates:
- 2003/02/27: gautoedit now has an additional red button to allow
flagging of the bottom. Output is written directly to abottom.asc
- 2003/02/27:a new setup program, "adcptree.prl" allows
specification of data type so that the directory can be created to
process VmDAS LTA files as well as ENS files. (Processing of ENS
files is not ready).
- 2003/02/27:A new Python program joins the processing toolbox:
"quick_adcp.py". This has all the features of quick_adcp.prl as well
as features such as:
- keywords to rerun specific steps (eg. nav steps, or
calibration)
- can process LTA files as easily as pingdata files
- 2003/03/03: gautoedit now has an additional gray bar which turns
off smoothing of navigational prior to calculating ocean velocity. This
might be useful if you think that ocean velocity artifacts are caused
by navigation or
navigational smoothing.
- 2003/04/03: gautoedit now has one more manual editing buttons,
"pzap", bringing the number of buttons that immediately write to ascii
files to four (the others being "Trange" and "rzap", and "bottom").
Documentation for these buttons was updated.
- 2003/05/05: fixed a bug in which on-station error velocity was
actually
being applied when underway also (i.e. there was no difference between
on-station
and underway error velocity: both were applied always).
- 2003/05/05: added a field to adjust the tolerance when deleting
low
shallow PG. This editing is designed to identify probable ringing and
delete
the offending bins. You may want to adjust it or disable it, and now
you can do that in the gautoedit window. Smaller numbers mean alot of
PG are
deleted. Larger numbers make it less effective.
- quick_adcp.prl batch processing for pingdata (no longer supported)
- using quick_adcp.prl to process ADCP data