4.3.3.1.1. read.m: Reading binary ADCP data files¶
4.3.3.1.1.1. Matlab requirements¶
You must have UH Currents group matlab code installed and the matlab path configured. Read the relevant parts of this page to install the CODAS matlab directory and set up the path.
You do not need the entire CODAS installation (executables and/or Python) to use this program, just the matlab directory and the correct MATABPATH.
4.3.3.1.1.2. Reading raw data¶
Programs exist to read raw data from data files collected by various logging programs, sometimes more than one program can log data for a given instrument. The binary data format differs between instruments; the filenames vary between logging programs. The “class” (below) indicates the class of instrument (specifically the kind of binary data stored). There is one read program for each class of instruments: specifying the instrument class is necessary during the read process so matlab will know which “read” program to run. These ‘read’ programs only work with instruments that have 4 beams (regardless of how many are working):
class instrument self-contained underway
name (lowered, (requires a data
moored) acquisition program)
----- ---------- ---------- ------------------------------
nb nb adcp no yes: UHDAS (*.raw)
bb bb adcp yes yes: VmDAS (ENR, ENS, ENX, STA, LTA
yes: Transect (*.r files only)
wh workhorse yes yes: VmDAS (ENR, ENS, ENX, STA, LTA
os ocean surveyor NA yes: VmDAS (ENR, ENS, ENX, STA, LTA
UHDAS (*.raw)
The “read” function returns a data structure with fields present for each beam in a three-dimensional array. Type “help nb/read”, “help bb/read”, (etc) to see the options for the program. In addition to required options (instrument class, file name, and yearbase) you can specify the data variables to output (eg. ‘vel’ is velocity, ‘ts’ is time series (heading, temperature,…)) and the indices to extract. Note that not all fields are available in all data sets.
There are two types of single-ping data:
from the instrument: (eg. moored, lowered) or for shipboard ADCP, the ENR files (from VmDAS) or “raw” files (from UHDAS). These data have no positions, the instrument clock sets the time stamp, and if heading is present it probably came from an internal flux gate compass and/or tilt sensors.
“fully navigated”: These have passed through a data acqusition system and additional information has been added (position, attitude, and corrected time stamp)
NOTE that the data collected may be in earth or beam coordinates, depending on the way the instrument was set up. Fully navigated shiboard data are in earth coordinatees.
Data present in the table above have the following fields:
"RAW" (from the
instrument) lon coord time pings per heading
lat system base ensemble (pitch,roll)
------------------+-----+---------+-----------+-----------+-------------
moored or lowered no note(1) instrument note(1) internal sensor
shipboard:
'raw'
VmDAS ENR no beam instrument 1 see note(2)
UHDAS raw no beam instrument 1 see note(2)
shipboard:
partially
navigated
VmDAS ENS yes beam UTC 1 see note(2)
shipboard:
fully
navigated
VmDAS ENX yes earth UTC 1 as specified
VmDAS STA yes earth UTC note(2) as specified
VmDAS LTA yes earth UTC note(2) as specified
UHDAS raw yes earth UTC 1 as specified
(see note(3))
NOTES
Moored or lowered are usually collected in earthcoordinates, but the transformation matrices are available in this suite of programs to switch from beam to earth coordinates given heading.
Raw RDI (un-navigated) shipboard ADCP data will only have a heading if it is put into the deck unit using the synchro input. You cannot assume this is being used: heading usually comes in a serial input and is not available until the “fully navigated” stage.
UHDAS raw data files are un-navigated, and are stored along with the raw serial inputs (gzipped files with GGA messages, etc). Two steps are needed to create ancillary navigation files, at which point a simple tool (in this sofware suite) can be used to attach the navigation UTC time, and attitude to the UHDAS files.
4.3.3.1.1.3. Working with binary ADCP data files¶
You must run “adcppath” and then “radcppath” to have access to the programs that read raw data from RDI instruments
DO NOT add any subdirectories under ‘rawadcp” other than those provided by this mechanism, or you will be running obsolete code.
NEVER name a variable ‘nb’, ‘bb’, or ‘os’, since this will override the class definition and you will be unable to use that read function until you clear the variable.
Hint: start by reading a subsample of the data so you know how much memory you will be using, and how long it will take to read the file.
Hint: see also ‘cutadcp’ and ‘catadcp’ for crude tools to subsample and concatenate data structures. These work best with data structures obtained using ‘read’ before using restruct. See below for details and use matlab command-line ‘help’ for these commands
Examples:
UHDAS Ocean surveyor
Read all variables, first 300 values
pingdata type = ‘first’
data are RAW (straight from the instrument)
-------------------------
>> osfile = 'lmg0501/raw/os38/lg2004_365_14400.raw';
>> osdata=read(os, osfile,'vars','all','ilist',[1:300])
osdata =
vel: [80x4x300 double] beam velocities
cor: [80x4x300 double] correlation
amp: [80x4x300 double] signal/noise
pg: [80x4x300 double] 0 or 1 flag
heading: [1x300 double] NA, present for consistency
pitch: [1x300 double] NA, present for consistency
roll: [1x300 double] NA, present for consistency
soundspeed: [1x300 double] sound speed from instrument
temperature: [1x300 double] temperature from instrument
tr_pressure: [1x300 double] NA, present for consistency
bt: [1x1 struct] bottom track data
nav: [1x1 struct] NA, present for consistency
dday: [1x300 double] instrument time
ens_num: [1x300 double] profile number
num_pings: [1x300 double] number of pings averaged (1)
config: [1x1 struct] full of useful info
depth: [80x1 double] nominal depth
info: [1x1 struct] (added by 'read'; machinery)
error: [] (added by 'read'; machinery)
-------------------------
Read all variables, every 40th point
FULLY NAVIGATED data (requires presence of gbin files)
-------------------------
>> osfile = 'lmg0501/raw/os38/lg2004_365_14400.raw';
>> osdata = get_xfraw(os,filelist(40).name,'step',40);
-------------------------
get the data in earth coordinates
if you know the transducer alignment, you can get the data in earth coordinates.
---------------------------
>> os_halign = 42.3; %set this to what you found in #2
>> data=get_xfraw(os, filelist(12).name,'h_align', os_halign);
>> [osdd, cfg] = restruct(os, data);
---------------------------
Read all variables, every 40th point
FULLY NAVIGATED data (requires the three config files used during data acquisition and processing)
-------------------------
>> cfg = get_uhdascfg('cfgbase','lmg0501', ...
'cfgpath', 'lmg0501/raw/config',...
'instname', 'os38',...
'pingtype', 'bb',...
'logfpath', '.');
>> osdata = getsome_adcp(cfg, 'ddrange', [364 364.1])
-------------------------
VmDAS data
read some data (this scheme works for all VmDAS data files)
-------------------------
>> filename = 'KMADCP093_000000.LTA'; % this is fully navigated, averaged data
>> osdata = read(os, filename, 'vars', 'all');
-------------------------
concatenating files:
-------------------------
>> filename1 = 'KMADCP093_000000.LTA';
>> filename2 = 'KMADCP093_000001.LTA';
>> filename3 = 'KMADCP093_000002.LTA';
>> osdata1 = read(os, filename1, 'vars', 'all');
>> osdata2 = read(os, filename2, 'vars', 'all');
>> osdata = catadcp(osdata1, osdata2);
>> osdata = catadcp(osdata, read(os, filename3, 'vars', 'all');
>> dd = restruct(os, osdata); % make the variables 2-dimensional; add some
-------------------------
4.3.3.1.1.4. transforming the data into earth coordinates¶
To get raw data with U, V, W, and error velocity you need to transform the velocities from beam coordinates to earth coordinates. For this, you must know the angle of the transducer relative to the hull. This usually comes as a rough guess of the angle, which is used during aquisition, and a calibration value, which is determined after processing. In this example, the heading offset used during aquisition was 45deg, and the “phase” from earlier codas calibrations was 0.55deg. The ‘geometry’ varies with the instrument, but is usually ‘convex’ for the OS and ‘concave’ for NB and BB instruments.
-------------------------
OH = 45.0 + .55; %Mounting Angle + phase
hdg = data.heading;
[data.enu data.xyze] = beam_enu(data.vel, OH + hdg, 0*hdg, 0*hdg', ...
'geometry','convex');
-------------------------
4.3.3.1.1.5. Using raw data¶
Sometimes if is nice to have all four beams available without having to type ‘squeeze’. The function “restruct” returns variables starting as [NBINS x NBEAMS x NPROFS] in four separate variables, each [NBINS x NPROFS]. Restruct requires the instrument class as the first argument. (The optional output structure ‘config’ holds title information used for plotting)
-------------------------
[dd, config] = restruct(bb, data); % "config" is used for plotting; see below
eg: data.vel: [50x4x50 double] (from original read)
turns into
dd.vel1: [50x50 double] (after restruct)
dd.vel2: [50x50 double]
dd.vel3: [50x50 double]
dd.vel4: [50x50 double]
-------------------------
“restruct” is most useful with fully navigated data.
From Example (1b) above:
------------------------------------------
dd= (same as original "data", plus some)
dday: [1x1200 double] original instrument decimal day
ens_num: [1x1200 double] profile number
cor1: [80x1200 double] \ correlation, beam 1 (for a NB, the
cor2: [80x1200 double] \ correlation, beam 2 field would be
cor3: [80x1200 double] / correlation, beam 3 "spectral width"
cor4: [80x1200 double] / correlation, beam 4 not correlation
amp1: [80x1200 double] \ signal/noise, beam 1
amp2: [80x1200 double] \ signal/noise, beam 2
amp3: [80x1200 double] / signal/noise, beam 3
amp4: [80x1200 double] / signal/noise, beam 4
pg1: [80x1200 double] \ flag: 0 or 1, beam 1
pg2: [80x1200 double] \ flag: 0 or 1, beam 2
pg3: [80x1200 double] / flag: 0 or 1, beam 3
pg4: [80x1200 double] / flag: 0 or 1, beam 4
vel1: [80x1200 double] \ if ORIGINALLY beam coords, these are
vel2: [80x1200 double] measured velocities for the 4 beams
vel3: [80x1200 double] if ORIGINALLY in earth coords, (eg. LTA)
vel4: [80x1200 double] / meas vel in [east, north , up, errvel]
xyze1: [80x1200 double] \ measured velocities in ship coords:
xyze2: [80x1200 double] \ starboard, forward, up, errvel
xyze3: [80x1200 double] /
xyze4: [80x1200 double] /
enu1: [80x1200 double] \ measured velocoties in earth coordinates
enu2: [80x1200 double] \ east,north,up,errvel
enu3: [80x1200 double] /
enu4: [80x1200 double] /
uship: [1x1200 double] ship's speed from navigation
vship: [1x1200 double] uses 'll2mps'
umeas: [80x1200 double] \ same as enu1: m/s
vmeas: [80x1200 double] \ same as enu2: m/s
wmeas: [80x1200 double] / same as enu3: m/s
emeas: [80x1200 double] / same as enu4: m/s
fmeas: [80x1200 double] measured velocity aligned with ship heading
pmeas: [80x1200 double] measured velocity aligned with port dir
corr_dday: [1x1200 double] --> UTC time
heading: [1x1200 double] heading from some source (depends on )
pitch: [1x1200 double] pitch from some source (data type and)
roll: [1x1200 double] roll from some source (acquisition )
temperature: [1x1200 double] temperature from thermistor
soundspeed: [1x1200 double] soundspeed used (instrument-dependent)
nav: [1x1 struct] navigation structure, UTC time, lon, lat
mps: [1x1200 double] ship speed (from ll2mps)
cog: [1x1200 double] course over ground
lon: [1x1200 double] lon from source
lat: [1x1200 double] lat from source
uabs: [80x1200 double] ocean u: umeas + uship (no corrections!!)
vabs: [80x1200 double] ocean v: vmeas + vship (no corrections!!)
fabs: [80x1200 double] ocean vel in direction of ship's heading
bt: [1x1 struct] bottom track data
bins: [80x1 double] bin count (positive away from instrument)
depth: [80x1 double] nominal depth (meters) away from instr
config: [1x1 struct] FULL OF USEFUL INFO, eg: bin, pulse...
------------------------------------------