CODAS Processing

Overview

CODAS

CODAS (Common Ocean Data Access System) is not a heirarchical database; it is a portable, self-descriptive file format (in the spirit of netCDF), that was designed specifically for ADCP (and other oceanographic) data. For historical reasons it is stored as a collection of files. Because it is an organized body of information, it is referred to as a database.

The term CODAS processing has refers to a suite of open-source programs for processing ADCP data. The CODAS processing suite of programs consists of C, Python, and Matlab programs that will run on Windows, Linux, SunOS, or Mac OSX, and can process pingdata from a Narrowband ADCP, data collected from a Broadband or Ocean Surveyor data by VmDAS, or data collected by any of those instruments using UHDAS (open source acquisition software that runs RDI ADCPs). Present work is focused on eleiminating the need for Matlab.

---

ADCP processing

Some kind of data treatment is necessary because the acquisition programs write binary files to the disk that are not readable by commercial plotting packages. In fact, there are not actually any ocean velocities stored in the files. A shipboard ADCP reports currents measured along each of its beams. These currents must be transformed into earth coordinates, and the motion of the ship taken out. Ancillary data such as heading and position are used to determine the ocean velocity from the measured velocies.

There are at least four necessary processing steps which are performed by (or made possible) by the CODAS routines. First, an ocean reference layer is used to remove the ship’s speed from the measured velocities. By assuming the ocean reference layer is relatively smooth, positions can be nudged to smooth the ship’s velocity, which directly results in the smooth reference layer velocity. (This was more important when fixes were rare or jumpy (such as with LORAN) or dithered (such as SA GPS signals prior to 2001).

Second, calibration routines are available to estimate the heading misalignment from either “bottom track” or “water track” data. Watertrack calibration routines use sudden accelerations (such as stopping and starting of the ship when doing station-work) to derive a heading misalignment. For a ship travelling at 10 kts, a 1-degree heading error results in a 10 cm/s cross-track velocity error. It is critical that the misalignment be accounted for if one is to avoid cross-track biases in the velocities.

Third, a GPS-derived heading source (such as Ashtech, POSMV, or Seapath) may provide a more accurate (though often less reliable) heading source than a gyro. Routines are in place for pingdata and UHDAS data to correct the gyro heading with the GPS-derived heading, using a quality-controlled difference in headings. An example is available for VmDAS data. Gyro headings may be reliable but they can vary with oscillations of several degrees over several hours, thus creating spurious fluctuations in the ocean velocity that resemble “eddies” , but which are soley the result of cross-track velocity errors (from the associated gyro heading errors).

Fourth, it is crucial that bad data be edited out prior to use. Traditionally, the data available from the DAS2.48 narrowband data was averaged in 5 minute groups. VmDAS and UHDAS also output time-averaged data, which can be loaded into the CODAS database for further processing. With CODAS processing, a graphical interface allows identification of the bottom and selection of bad profiles or bad bins based on a variety of criteria. To some extent this can be automated, for final processing, a person must visually inspect all the averages from a cruise. The graphical interface vastly speeds up editing to the point where it takes only a few minutes of user time per day of data for a typical cruise.

CODAS processing can be used for data that has already been averaged (pingdata, LTA files) or single-ping data. CODAS processing includes routines that extensively screen single-ping data prior to averaging. Under certain conditions, this may be necessary to avoid underway biases caused by bubbles or ice near the transducer, or acoustic interference from other instruments.

Quick_adcp.py is a Python script that runs the usual CODAS processing steps in a predictable and configurable manner. For a clean dataset, it provides a relatively quick and painless way of looking at the data, addressing configuration issues, and editing. If your dataset has problems, you can always run the appropriate steps manually.

CODAS processing stages

CODAS processing of ADCP data consists of three stages.

1. If dealing with single-ping data:

   • read the ADCP and ancillary serial data
   • “navigate” the data (find UTC time, add position and attitude)
   • edit out bad single-ping velocities
   • average the single-ping data; write to disk.

   These steps are already done in PINGDATA and VmDAS LTA or STA data.

   This flow chart shows the split between this stage (acquisition + averaging + loading the database) and the latter stages (CODAS processing; manipulating the database)

2. Load the averages into CODAS:

   • find and smooth the reference layer,
   • obtain a gps-based heading correction for the gyro headings,
   • determine preliminary angle and amplitude calibrations from watertrack and/or bottom track data (using corrected headings)

3. Editing and calibration:

   • editing (bottom interference, wire interference, bubbles, ringing, identifying problems with heading and underway bias),
   • final calibration based on edited data

More notes

• If you start with LTA, STA, or pingdata, you are starting at (2).

• Processing that interacts with the CODAS files (“CODAS processing”)

  is really steps (2) and (3).

• With the access to VmDAS single-ping data, (ENS or ENX) we have the opportunity to do a better job editing at the single-ping stage.

• With UHDAS data, we are required to start with single-ping data.

• If you start with single-ping data (stage 1), you should already be familiar with stages (2) and (3)

Quick_adcp.py: Overview

Setup of a processing directory should use adcptree.py. Although there are various versions of adcptree in existence, they reflect older generations of processing. Quick_adcp.py should be able to deal with any of the following combinations of data acquisition, averaging (or not) and instrument, as long as the processing directory was set up with adcptree.py:

Acquisition program	instrument	ping type	Averaged??	file type	incremental?
DAS2.48 DAS2.49	NB150	nb	yes	pingdata	no
VmDAS	Broadband or Workhorse	bb	yes	LTA or STA	no
			no	ENS or ENX	no
VmDAS	Ocean Surveyor	bb	yes	LTA or STA	no
			no	ENS or ENX	no
		nb	yes	LTA or STA	no
			no	ENS or ENX	no
		bb+nb	yes	first ping ENS or ENX	no
			no	first ping ENS or ENX	no
UHDAS	NB150	nb	no	raw	yes
	OS	bb	no	raw	yes
		nb	no	raw	yes
		bb+nb	no	raw	yes
	WH300	bb	no	raw	yes

At its operational level, CODAS processing consists of a series of C programs or matlab programs that interact with the CODAS database or with files on the disk. C programs usually deal with the database directly, by loading data (eg. loadping.exe), extracting data (eg. adcpsect.ext) or by manipulating the databaes (eg. rotate.exe, putnav.exe, dbupdate.exe). Matlab programs are used to maniplate files on the disk so C programs can use them, or in the case of VmDAS or UHDAS data, Matlab is used to read the original data files and created translated versions (on the disk) that C programs can read.

Quick_adcp.py is designed to work through the standard processing steps, writing control files and running the C programs, or writing matlab files to disk, and running them. Control files for C programs are named with the same base name (such as “rotate”) with a “.tmp” suffix. Matlab (http://www.mathworks.com) files have the same base name as the original matlab file but have “_tmp” in the name (eg. “ashrot.m” becomes “ashrot_tmp.m” when written by quick_adcp.py).

Once your paths are set up (matlab, executable, and Python), you

1. pick a working area (not in the PROGRAMS directory; that is reserved for UH code)

2. run adcptree.py with the appropriate options

3. locate your data files, determine the appropriate switches for quick_adcp.py

4. run quick_adcp.py. Arguments can be typed on the command line

   or stored in a control file (accessed with –cntfile). Commandline options override control file options.

Manual Processing

Quick_adcp.py provides a mechanism to run all the individual steps from one script. If one needs to tailor a specific step to address a problem, that can be done on the shell command line (or from the matlab command line). Adcptree.py creates a processing directory tree and copies templates or documented, editable files to the various subdirectories, setting up the tree for processing. To process a dataset manually, one would work their way through the directories, repeating (in the proper order) the following steps:

1. edit the appropriate file
2. run the related program

C programs are almost always called with a control file to specify parameters that the user may wish to configure or change. These include predictable values, such as the database name or yearbase, and configurable values, such as a reference layer depth range. C programs are called on a command line from the relevant working directory as (for example)

adcpsect adcpsect.cnt

The original “.cnt” files are self-documented, showing the various options that can be chosen. The user is advised to leave these fiels as is and name their copies something else, such as “adcpsect.tmp”, and then run it as

adcpsect adcpsect.tmp

Matlab programs are copied by adcptree.py to the appropriate directory and exist as a script (or a stub that calls a script). The matlab program can be edited and then run in the appropriate directory.

Quick_adcp.py Instructions and guides

This table is the launching point for the rest of CODAS processing documentation.

description	link	data types	best use
getting started	practice “demos”	all data types	FIRST TIME USER
quick_adcp.py	quick_adcp.py guide	all data types	quick_adcp.py + CODAS tutorial
quick_adcp.py	recipe to process LTA data	LTA	short guide
quick_adcp.py	list of LTA steps	LTA	illustration or reminder of steps
quick_adcp.py	list of pingdata steps	pingdata	illustration or reminder of steps
graphical editing	gautoedit tutorial	all data types	gautoedit tutorial (pronounced “Gee! Autoedit!”)

Older Documentation: detailed, with some relevant pieces

Some documentation (eg. CODAS file format and strategy) contains useful information in spite of being outdated. Links are provided here for interest and completion.

postscript manual	detailed codas database reference	codas database and pingdata details	database construction and operation (eg. profile flags)
original pingdata tutorial	very detailed pindata instructions	old pingdata: details and relevant instructions	look up details of a particular command, (eg. calibration)

Matlab tools to access to ADCP data

Tip

Follow this link for details about Matlab tools to access ADCP data (CODAS and other files).