.. ...................................... .. UHDAS/CODAS restructured text document .. ...................................... Reading binary ADCP data files =============================== Matlab requirements ------------------- You must have UH Currents group matlab code installed and the matlab path configured. Read the relevant parts of `this page <../../codas_setup/index.html>`_ to find the and download the matlab code ("matlab.zip"), and set up the path. Reading raw data ---------------- Programs exist to read raw data from data files collected by various logging programs, sometimes more than one program can log data for a given instrument. The binary data format differs between instruments; the filenames vary between logging programs. The "class" (below) indicates the class of instrument (specifically the kind of binary data stored). There is one read program for each class of instruments: specifying the instrument class is necessary during the read process so matlab will know which "read" program to run. These 'read' programs only work with instruments that have 4 beams (regardless of how many are working): :: class instrument self-contained underway name (lowered, (requires a data moored) acquisition program) ----- ---------- ---------- ------------------------------ bb bb adcp yes yes: VmDAS (ENR, ENS, ENX, STA, LTA yes: Transect (*.r files only) bb workhorse yes yes: VmDAS (ENR, ENS, ENX, STA, LTA os ocean surveyor NA yes: VmDAS (ENR, ENS, ENX, STA, LTA UHDAS (*.raw) The "read" function returns a data structure with fields present for each beam in a three-dimensional array. Type "help nb/read", "help bb/read", (etc) to see the options for the program. In addition to required options (instrument class, file name, and yearbase) you can specify the data variables to output (eg. 'vel' is velocity, 'ts' is time series (heading, temperature,...)) and the indices to extract. Note that not all fields are available in all data sets. There are two types of single-ping data: - from the instrument: (eg. moored, lowered) or for shipboard ADCP, the ENR files (from VmDAS) or "raw" files (from UHDAS). These data have no positions, the instrument clock sets the time stamp, and if heading is present it probably came from an internal flux gate compass and/or tilt sensors. - "fully navigated": These have passed through a data acqusition system and additional information has been added (position, attitude, and corrected time stamp) NOTE that the data collected may be in earth or beam coordinates, depending on the way the instrument was set up. Fully navigated shiboard data are in earth coordinatees. Data present in the table above have the following fields: :: "RAW" (from the instrument) lon coord time pings per heading lat system base ensemble (pitch,roll) ------------------+-----+---------+-----------+-----------+------------- moored or lowered no note(1) instrument note(1) internal sensor shipboard: 'raw' VmDAS ENR no beam instrument 1 see note(2) UHDAS raw no beam instrument 1 see note(2) shipboard: partially navigated VmDAS ENS yes beam UTC 1 see note(2) shipboard: fully navigated VmDAS ENX yes earth UTC 1 as specified VmDAS STA yes earth UTC note(2) as specified VmDAS LTA yes earth UTC note(2) as specified UHDAS raw yes earth UTC 1 as specified (see note(3)) **NOTES** (1) Moored or lowered are usually collected in earthcoordinates, but the transformation matrices are available in this suite of programs to switch from beam to earth coordinates given heading. (2) Raw RDI (un-navigated) shipboard ADCP data will only have a heading if it is put into the deck unit using the synchro input. You cannot assume this is being used: heading usually comes in a serial input and is not available until the "fully navigated" stage. (3) UHDAS raw data files are un-navigated, and are stored along with the raw serial inputs (gzipped files with GGA messages, etc). Two steps are needed to create ancillary navigation files, at which point a simple tool (in this sofware suite) can be used to attach the navigation UTC time, and attitude to the UHDAS files. Working with binary ADCP data files: ==================================== You must run "adcppath" and then "radcppath" to have access to the programs that read raw data from RDI instruments - DO NOT add any subdirectories under 'rawadcp" other than those provided by this mechanism, or you will be running obsolete code. - NEVER name a variable 'nb', 'bb', or 'os', since this will override the class definition and you will be unable to use that read function until you clear the variable. - Hint: start by reading a subsample of the data so you know how much memory you will be using, and how long it will take to read the file. - Hint: see also 'cutadcp' and 'catadcp' for crude tools to subsample and concatenate data structures. These work best with data structures obtained using 'read' *before* using restruct. See below for details and use matlab command-line 'help' for these commands Examples: (1) UHDAS Ocean surveyor (a) Read all variables, first 300 values - pingdata type = 'first' - data are RAW (straight from the instrument) :: ------------------------- >> osfile = 'lmg0501/raw/os38/lg2004_365_14400.raw'; >> osdata=read(os, osfile,'vars','all','ilist',[1:300]) osdata = vel: [80x4x300 double] beam velocities cor: [80x4x300 double] correlation amp: [80x4x300 double] signal/noise pg: [80x4x300 double] 0 or 1 flag heading: [1x300 double] NA, present for consistency pitch: [1x300 double] NA, present for consistency roll: [1x300 double] NA, present for consistency soundspeed: [1x300 double] sound speed from instrument temperature: [1x300 double] temperature from instrument tr_pressure: [1x300 double] NA, present for consistency bt: [1x1 struct] bottom track data nav: [1x1 struct] NA, present for consistency dday: [1x300 double] instrument time ens_num: [1x300 double] profile number num_pings: [1x300 double] number of pings averaged (1) config: [1x1 struct] full of useful info depth: [80x1 double] nominal depth info: [1x1 struct] (added by 'read'; machinery) error: [] (added by 'read'; machinery) ------------------------- (b) Read all variables, every 40th point - FULLY NAVIGATED data (requires presence of gbin files) :: ------------------------- >> osfile = 'lmg0501/raw/os38/lg2004_365_14400.raw'; >> osdata = get_xfraw(os,filelist(40).name,'step',40); ------------------------- (c) get the data in earth coordinates - if you know the transducer alignment, you can get the data in earth coordinates. :: --------------------------- >> os_halign = 42.3; %set this to what you found in #2 >> data=get_xfraw(os, filelist(12).name,'h_align', os_halign); >> [osdd, cfg] = restruct(os, data); --------------------------- (d) Read all variables, every 40th point - FULLY NAVIGATED data (requires the three config files used during data acquisition and processing) :: ------------------------- >> cfg = get_uhdascfg('cfgbase','lmg0501', ... 'cfgpath', 'lmg0501/raw/config',... 'instname', 'os38',... 'pingtype', 'bb',... 'logfpath', '.'); >> osdata = getsome_adcp(cfg, 'ddrange', [364 364.1]) ------------------------- (2) VmDAS data (a) read some data (this scheme works for all VmDAS data files) :: ------------------------- >> filename = 'KMADCP093_000000.LTA'; % this is fully navigated, averaged data >> osdata = read(os, filename, 'vars', 'all'); ------------------------- (b) concatenating files: :: ------------------------- >> filename1 = 'KMADCP093_000000.LTA'; >> filename2 = 'KMADCP093_000001.LTA'; >> filename3 = 'KMADCP093_000002.LTA'; >> osdata1 = read(os, filename1, 'vars', 'all'); >> osdata2 = read(os, filename2, 'vars', 'all'); >> osdata = catadcp(osdata1, osdata2); >> osdata = catadcp(osdata, read(os, filename3, 'vars', 'all'); >> dd = restruct(os, osdata); % make the variables 2-dimensional; add some ------------------------- transforming the data into earth coordinates ============================================ To get raw data with U, V, W, and error velocity you need to transform the velocities from beam coordinates to earth coordinates. For this, you must know the angle of the transducer relative to the hull. This usually comes as a rough guess of the angle, which is used during aquisition, and a calibration value, which is determined after processing. In this example, the heading offset used during aquisition was 45deg, and the "phase" from earlier codas calibrations was 0.55deg. The 'geometry' varies with the instrument, but is usually 'convex' for the OS and 'concave' for NB and BB instruments. :: ------------------------- OH = 45.0 + .55; %Mounting Angle + phase hdg = data.heading; [data.enu data.xyze] = beam_enu(data.vel, OH + hdg, 0*hdg, 0*hdg', ... 'geometry','convex'); ------------------------- Using raw data ============== Sometimes if is nice to have all four beams available without having to type 'squeeze'. The function "restruct" returns variables starting as [NBINS x NBEAMS x NPROFS] in four separate variables, each [NBINS x NPROFS]. Restruct requires the instrument class as the first argument. (The optional output structure 'config' holds title information used for plotting) :: ------------------------- [dd, config] = restruct(bb, data); % "config" is used for plotting; see below eg: data.vel: [50x4x50 double] (from original read) turns into dd.vel1: [50x50 double] (after restruct) dd.vel2: [50x50 double] dd.vel3: [50x50 double] dd.vel4: [50x50 double] ------------------------- "restruct" is most useful with fully navigated data. From Example (1b) above: :: ------------------------------------------ dd= (same as original "data", plus some) dday: [1x1200 double] original instrument decimal day ens_num: [1x1200 double] profile number cor1: [80x1200 double] \ correlation, beam 1 (for a NB, the cor2: [80x1200 double] \ correlation, beam 2 field would be cor3: [80x1200 double] / correlation, beam 3 "spectral width" cor4: [80x1200 double] / correlation, beam 4 not correlation amp1: [80x1200 double] \ signal/noise, beam 1 amp2: [80x1200 double] \ signal/noise, beam 2 amp3: [80x1200 double] / signal/noise, beam 3 amp4: [80x1200 double] / signal/noise, beam 4 pg1: [80x1200 double] \ flag: 0 or 1, beam 1 pg2: [80x1200 double] \ flag: 0 or 1, beam 2 pg3: [80x1200 double] / flag: 0 or 1, beam 3 pg4: [80x1200 double] / flag: 0 or 1, beam 4 vel1: [80x1200 double] \ if ORIGINALLY beam coords, these are vel2: [80x1200 double] measured velocities for the 4 beams vel3: [80x1200 double] if ORIGINALLY in earth coords, (eg. LTA) vel4: [80x1200 double] / meas vel in [east, north , up, errvel] xyze1: [80x1200 double] \ measured velocities in ship coords: xyze2: [80x1200 double] \ starboard, forward, up, errvel xyze3: [80x1200 double] / xyze4: [80x1200 double] / enu1: [80x1200 double] \ measured velocoties in earth coordinates enu2: [80x1200 double] \ east,north,up,errvel enu3: [80x1200 double] / enu4: [80x1200 double] / uship: [1x1200 double] ship's speed from navigation vship: [1x1200 double] uses 'll2mps' umeas: [80x1200 double] \ same as enu1: m/s vmeas: [80x1200 double] \ same as enu2: m/s wmeas: [80x1200 double] / same as enu3: m/s emeas: [80x1200 double] / same as enu4: m/s fmeas: [80x1200 double] measured velocity aligned with ship heading pmeas: [80x1200 double] measured velocity aligned with port dir corr_dday: [1x1200 double] --> UTC time heading: [1x1200 double] heading from some source (depends on ) pitch: [1x1200 double] pitch from some source (data type and) roll: [1x1200 double] roll from some source (acquisition ) temperature: [1x1200 double] temperature from thermistor soundspeed: [1x1200 double] soundspeed used (instrument-dependent) nav: [1x1 struct] navigation structure, UTC time, lon, lat mps: [1x1200 double] ship speed (from ll2mps) cog: [1x1200 double] course over ground lon: [1x1200 double] lon from source lat: [1x1200 double] lat from source uabs: [80x1200 double] ocean u: umeas + uship (no corrections!!) vabs: [80x1200 double] ocean v: vmeas + vship (no corrections!!) fabs: [80x1200 double] ocean vel in direction of ship's heading bt: [1x1 struct] bottom track data bins: [80x1 double] bin count (positive away from instrument) depth: [80x1 double] nominal depth (meters) away from instr config: [1x1 struct] FULL OF USEFUL INFO, eg: bin, pulse... ------------------------------------------ Plotting raw data ================= After you load the data and restructure it, you can plot any 2-dimensional parameter with "aplotit", for example: :: ---------------------- aplotit(data,config,'fn','amp1'); or aplotit(data,config,'fn','amp1','cur_ax', subplot(211)); aplotit(data,config,'fn','cor1','cur_ax', subplot(212)); ----------------------