CODAS
CODAS (Common Ocean Data Access System) is not a heirarchical database; it is a portable, self-descriptive file format (in the spirit of netCDF), that was designed specifically for ADCP (and other oceanographic) data. For historical reasons it is stored as a collection of files. Because it is an organized body of information, it is referred to as a database.
The term CODAS processing has refers to a suite of open-source programs for processing ADCP data. The CODAS processing suite of programs consists of C, Python, and Matlab programs that will run on Windows, Linux, SunOS, or Mac OSX, and can process pingdata from a Narrowband ADCP, data collected from a Broadband or Ocean Surveyor data by VmDAS, or data collected by any of those instruments using UHDAS (open source acquisition software that runs RDI ADCPs). Present work is focused on eleiminating the need for Matlab.
ADCP processing
Some kind of data treatment is necessary because the acquisition programs write binary files to the disk that are not readable by commercial plotting packages. In fact, there are not actually any ocean velocities stored in the files. A shipboard ADCP reports currents measured along each of its beams. These currents must be transformed into earth coordinates, and the motion of the ship taken out. Ancillary data such as heading and position are used to determine the ocean velocity from the measured velocies.
There are at least four necessary processing steps which are performed by (or made possible) by the CODAS routines. First, an ocean reference layer is used to remove the ship’s speed from the measured velocities. By assuming the ocean reference layer is relatively smooth, positions can be nudged to smooth the ship’s velocity, which directly results in the smooth reference layer velocity. (This was more important when fixes were rare or jumpy (such as with LORAN) or dithered (such as SA GPS signals prior to 2001).
Second, calibration routines are available to estimate the heading misalignment from either “bottom track” or “water track” data. Watertrack calibration routines use sudden accelerations (such as stopping and starting of the ship when doing station-work) to derive a heading misalignment. For a ship travelling at 10 kts, a 1-degree heading error results in a 10 cm/s cross-track velocity error. It is critical that the misalignment be accounted for if one is to avoid cross-track biases in the velocities.
Third, a GPS-derived heading source (such as Ashtech, POSMV, or Seapath) may provide a more accurate (though often less reliable) heading source than a gyro. Routines are in place for pingdata and UHDAS data to correct the gyro heading with the GPS-derived heading, using a quality-controlled difference in headings. An example is available for VmDAS data. Gyro headings may be reliable but they can vary with oscillations of several degrees over several hours, thus creating spurious fluctuations in the ocean velocity that resemble “eddies” , but which are soley the result of cross-track velocity errors (from the associated gyro heading errors).
Fourth, it is crucial that bad data be edited out prior to use. Traditionally, the data available from the DAS2.48 narrowband data was averaged in 5 minute groups. VmDAS and UHDAS also output time-averaged data, which can be loaded into the CODAS database for further processing. With CODAS processing, a graphical interface allows identification of the bottom and selection of bad profiles or bad bins based on a variety of criteria. To some extent this can be automated, for final processing, a person must visually inspect all the averages from a cruise. The graphical interface vastly speeds up editing to the point where it takes only a few minutes of user time per day of data for a typical cruise.
CODAS processing can be used for data that has already been averaged (pingdata, LTA files) or single-ping data. CODAS processing includes routines that extensively screen single-ping data prior to averaging. Under certain conditions, this may be necessary to avoid underway biases caused by bubbles or ice near the transducer, or acoustic interference from other instruments.
Quick_adcp.py is a Python script that runs the usual CODAS processing steps in a predictable and configurable manner. For a clean dataset, it provides a relatively quick and painless way of looking at the data, addressing configuration issues, and editing. If your dataset has problems, you can always run the appropriate steps manually.
CODAS processing of ADCP data consists of three stages.
If dealing with single-ping data:
These steps are already done in PINGDATA and VmDAS LTA or STA data.
This flow chart shows the split between this stage (acquisition + averaging + loading the database) and the latter stages (CODAS processing; manipulating the database)
Load the averages into CODAS:
Editing and calibration:
More notes
If you start with LTA, STA, or pingdata, you are starting at (2).
- Processing that interacts with the CODAS files (“CODAS processing”)
is really steps (2) and (3).
With the access to VmDAS single-ping data, (ENS or ENX) we have the opportunity to do a better job editing at the single-ping stage.
With UHDAS data, we are required to start with single-ping data.
If you start with single-ping data (stage 1), you should already be familiar with stages (2) and (3)
Setup of a processing directory should use adcptree.py. Although there are various versions of adcptree in existence, they reflect older generations of processing. Quick_adcp.py should be able to deal with any of the following combinations of data acquisition, averaging (or not) and instrument, as long as the processing directory was set up with adcptree.py:
Acquisition program | instrument | ping type | Averaged?? | file type | incremental? |
---|---|---|---|---|---|
DAS2.48 DAS2.49 | NB150 | nb | yes | pingdata | no |
VmDAS | Broadband or Workhorse |
bb | yes | LTA or STA | no |
no | ENS or ENX | no | |||
VmDAS | Ocean Surveyor | bb | yes | LTA or STA | no |
no | ENS or ENX | no | |||
nb | yes | LTA or STA | no | ||
no | ENS or ENX | no | |||
bb+nb | yes | first ping ENS or ENX |
no | ||
no | first ping ENS or ENX |
no | |||
UHDAS | NB150 | nb | no | raw | yes |
OS | bb | no | raw | yes | |
nb | no | raw | yes | ||
bb+nb | no | raw | yes | ||
WH300 | bb | no | raw | yes |
At its operational level, CODAS processing consists of a series of C programs or matlab programs that interact with the CODAS database or with files on the disk. C programs usually deal with the database directly, by loading data (eg. loadping.exe), extracting data (eg. adcpsect.ext) or by manipulating the databaes (eg. rotate.exe, putnav.exe, dbupdate.exe). Matlab programs are used to maniplate files on the disk so C programs can use them, or in the case of VmDAS or UHDAS data, Matlab is used to read the original data files and created translated versions (on the disk) that C programs can read.
Quick_adcp.py is designed to work through the standard processing steps, writing control files and running the C programs, or writing matlab files to disk, and running them. Control files for C programs are named with the same base name (such as “rotate”) with a “.tmp” suffix. Matlab (http://www.mathworks.com) files have the same base name as the original matlab file but have “_tmp” in the name (eg. “ashrot.m” becomes “ashrot_tmp.m” when written by quick_adcp.py).
Once your paths are set up (matlab, executable, and Python), you
pick a working area (not in the PROGRAMS directory; that is reserved for UH code)
run adcptree.py with the appropriate options
locate your data files, determine the appropriate switches for quick_adcp.py
- run quick_adcp.py. Arguments can be typed on the command line
or stored in a control file (accessed with –cntfile). Commandline options override control file options.
Manual Processing
Quick_adcp.py provides a mechanism to run all the individual steps from one script. If one needs to tailor a specific step to address a problem, that can be done on the shell command line (or from the matlab command line). Adcptree.py creates a processing directory tree and copies templates or documented, editable files to the various subdirectories, setting up the tree for processing. To process a dataset manually, one would work their way through the directories, repeating (in the proper order) the following steps:
- edit the appropriate file
- run the related program
C programs are almost always called with a control file to specify parameters that the user may wish to configure or change. These include predictable values, such as the database name or yearbase, and configurable values, such as a reference layer depth range. C programs are called on a command line from the relevant working directory as (for example)
adcpsect adcpsect.cnt
The original “.cnt” files are self-documented, showing the various options that can be chosen. The user is advised to leave these fiels as is and name their copies something else, such as “adcpsect.tmp”, and then run it as
adcpsect adcpsect.tmp
Matlab programs are copied by adcptree.py to the appropriate directory and exist as a script (or a stub that calls a script). The matlab program can be edited and then run in the appropriate directory.
This table is the launching point for the rest of CODAS processing documentation.
description | link | data types | best use |
---|---|---|---|
getting started | practice “demos” | all data types | FIRST TIME USER |
quick_adcp.py | quick_adcp.py guide | all data types | quick_adcp.py + CODAS tutorial |
quick_adcp.py | recipe to process LTA data | LTA | short guide |
quick_adcp.py | list of LTA steps | LTA | illustration or reminder of steps |
quick_adcp.py | list of pingdata steps | pingdata | illustration or reminder of steps |
graphical editing | gautoedit tutorial | all data types | gautoedit tutorial (pronounced “Gee! Autoedit!”) |
Some documentation (eg. CODAS file format and strategy) contains useful information in spite of being outdated. Links are provided here for interest and completion.
postscript manual | detailed codas database reference | codas database and pingdata details | database construction and operation (eg. profile flags) |
original pingdata tutorial | very detailed pindata instructions | old pingdata: details and relevant instructions | look up details of a particular command, (eg. calibration) |