LAST CHANGED : 2023/12/31 17:24:44 CRUISE NAME(S) : km2322, hot348 CRUISE DATES : 2023/12/26 20:34:12 to 2023/12/31 17:24:18 SHIP NAME : Kilo Moana PORTS : Honolulu, HI to Honolulu, HI CHIEF SCIENTIST : Fernando Carvalho Pacheco DATABASE NAME : a_km DATA FILES : km2023_359_73752.raw to km2023_364_57600.raw STATUS : to do done ------ ----------- averaged [ x ] loaded [ x ] NOTE: heading correction instrument exists NOTE: time-dependent heading corrections applied IN the ensembles (see cal/rotate/ens_hcorr.ang) check heading correction [ x ] calibration [ x ] edited [ x ] re-check heading correction [ x ] check editing [ x ] figures [ x ] INSTRUMENT : os38 ACQUISITION : PROGRAM : uhdas PROCESSING: : python LOGGING : PARAMETERS : BT : bottom track mode (on or off) SI : sampling interval or averaging period for ensemble (sec) NB : number of bins BL : bin length (m) TD : transducer depth (m) BK : blanking length (m) HO : heading offset applied by DAS (deg) HB : heading bias (deg) CRPH : compensation for roll-pitch-heading, 1:on, 0:off) yy/mm/dd hh:mm:ss BT SI NB BL TD BK HO HB CRPH 2023/12/26 20:34:12 off 300 75 24 7 16 44.55 0.00 0001 HEADING : PRIMARY : heading from gyro CORRECTION : heading correction from posmv NOTE: time-dependent heading corrections applied IN the ensembles (see cal/rotate/ens_hcorr.ang) POSITIONS : gps positions from posmv CALIBRATION : (check original processing parameters) (1) transducer alignment original transducer alignment: 44.55 additional rotation 0.0576 final transducer angle is: (original transducer angle) - (rotate_angle) 44.55 - 0.0576 = 44.4924 (2) scale factor original scale factor 1 additional scale factor (1.004) (3) ADCP (dx=starboard, dy=fwd) meters from GPS original: xducer_dx xducer_dy correction 16 6 final offset 16 6 final = original + corrections COMMENTS : gaps in heading correction? no scattering layers? no bubbles? lots - missing data chunks from julian day 361.5-362 underway bias? minimal PROCESSOR : T. Rohrer --- final processing parameters from dbinfo.txt ---------- <...> NOTE: Replace the text below with the final dbinfo.txt content <...> after you are done with all postprocessing. Change the <...> comment to say "final processing parameters" ## (determined from "sonar"): model = os ## (determined from "sonar"): frequency = 38 ## (determined from "sonar"): instname = os38 ## (determined from "sonar"): pingtype = nb beamangle 30 cruisename km2322 datatype uhdas dbname a_km ens_len 300 fixfile a_km.gps frequency 38 hcorr_inst posmv instname os38 model os pingtype nb proc_engine python ref_method refsm refuv_smoothwin 3 refuv_source nav sonar os38nb txy_file a_km.agt xducer_dx 16 xducer_dy 6 yearbase 2023 <...> Please consider submitting your final processed data to JASADCP <...> (Joint Carchive for SHipblard ADCP) data so other people can benefit <...> from all the work that went into this. GoogleSearch for <...> "JASADCP shipboard ADCP" in your browser. <...> <...> NOTE: Markup description: <...> - Narration is preceeded by these characters: <...> <...> - Anything with those characters at the beginning of a line will be <...> stripped out by JASADCP before archiving. <...> ================================================== POST PROCESSING STEPS: ================================================== <...> Post processing commands start in the os38nb directory. <...> If the cruise dates are wrong in the header, adjust manually. <...> The time range of processed data can be found in os38nb/adcpdb/a_km.tr <...> This template covers most common situations when processing UHDAS data in CODAS. <...> For more detail, the latest version of the CODAS manual can be found at: <...> https://currents.soest.hawaii.edu/docs/adcp_doc/codas_doc/index.html -------------------------- ### 1. Check visual oddities -------------------------- <...> Perform a visual first pass of the data looking for the following problems: <...> (a) good heading correction for all of the ADCP data (any gaps?) <...> (b) sharp transitions in velocities <...> (i) on station vs underway (transducer angle or scale factor) <...> (ii) vertical stripes when transitioning between underway and <...> stopped (an offset between ADCP and GPS is not accounted for) <...> (iii) biases when underway with poor PG (will need to apply uvship) <...> (c) gaps due to "free inertial" flag in Seapath (heading exists, <...> but positions are missing, so no ship speed) <...> (d) false bottoms near steep topography: symptom strong scattering <...> layer is mis-identified as the bottom, resulting in shorter <...> profiles that stop at the scattering layer, but data on both <...> sides is much deeper <...> <...> When to post-process vs reprocess (eg. with adcp_database_maker.py) <...> You should reprocess if <...> (a) heading correction: if there are gaps and there is another accurate <...> heading device, you could reprocess using that other device <...> (b) sharp transitions in velocities <...> (ii) vertical stripes when transitioning between underway and stopped <...> (likely an offset between ADCP and GPS is not accounted for) <...> (iii) biases when underway with poor PG (will need to apply uvship) <...> If the offset is greater than 2m in x or y AND if there are periods <...> when the ship is underway in heavy weather, then you should reprocess <...> ensuring that you use the final xducer_dx and xducer_dy. Then the <...> 'uvship' algorithm can be attempted. Otherwise do not use it <...> (c) there are gaps due to "free inertial" flag in Seapath. In this case, <...> pick another position device, and make sure you get its xducer_dx and <...> xducer_dy correct. You might have to process twice, once to estimate <...> the values, and a second time to use them. <...> (d) false bottoms near steep topography: If the Percent Good is BLUE below <...> the scattering layers, you can only get those parts back by reprocessing <...> and NEVER looking for the bottom. Then you have to edit out the bottom <...> using the threshold tool for Bottom Selector. If the PG is red or orange, <...> you can resurrect those profiles by using "reset editing". <...> <...> Otherwise you may postprocess the cruise. <...> <...> Heading correction: <...> Check that we have a good heading correction for all of the ADCP data. <...> <...> There should be no holes in the heading correction graphs <...> (cal/rotate/ens_hcorr*.png), where "valid corrections" are green <...> circles and "bad corrections" are red crosses. Gaps in the plots where <...> there are no symbols are okay, no data was collected there. <...> <...> If there are red crosses on the heading correction graphs <...> (cal/rotate/ens_hcorr_*.png) then they need to be patched in the cal/rotate dir <...> using patch_hcorr.py. <...> -------------------------- ### Run this: (to look for gaps in the cruise track) plot_nav.py nav/a*.gps ### Run this: (to look for gaps in the heading correction, examine the character of <...> the watertrack and bottom track calibration, etc figview.py ### Run this: (looking for missing heading correction values or missing positions) dataviewer.py <...> NOTE: IF there are holes in the heading correction plots, you need to <...> run "patch_hcorr.py" to interpolate across the gaps. Do this BEFOE doing any <...> further post-processing ### Run this: (to interpolate for missing heading correction fixes) cd cal/rotate patch_hcorr.py cd ../.. ### Run this: (to check watertrack calibration after patching hcorr) catwt.py ### Run this: (to check watertrack calibration after patching hcorr) catbt.py ### Run this: (to check dxdy statistics calibration after patching hcorr) catxy.py --------------------- ### 2. ADCP calibration --------------------- <...> Calibrate the dataset in sum by checking the calibration values <...> and applying a fix to the necessary categories if necessary. <...> <...> When looking at a water track or bottom track calibration, we want to see <...> statistics inside the following values: <...> <...> median_tolerance <...> <...> - amplitude range .997 to 1.003 # 0.3% <...> - phase/angle range -0.05 to 0.05 # 1/2 degree <...> - offset (dx,dy) rounded residual should be between -2 and 2 meters <...> <...> When looking at an xy calibration to adjust the relative location of the GPS <...> to the sonar we try to get the dx and dy values close to zero, bearing <...> in mind that this is a guess. The "signal" should be between 1000 and 5000. <...> Less than 1000 means very little data, more than 5000 means too much change. <...> <...> If values are out of range we make a bulk correction to the entire dataset <...> which should reduce the number of outliers to be edited out. <...> <...> If we make any changes to x or y they should be integers. <...> <...> The command to be run to edit the calibration is listed below, where * <...> is replaced with desired values for the options to be changed and <...> other options removed: <...> <...> quick_adcp.py --steps2rerun rotate:navsteps:calib --rotate_amplitude *.** --rotate_angle *.** --xducer_dx * --xducer_dy * --auto --------------------- ### Run this: (look at the watertrack calibration residuals remaining) catwt.py ### Run this: (look at the bottom track calibration residuals remaining) catbt.py ### Run this: (look at the remaining offsets required) catxy.py --- ### If a large dxdy correction is needed (values greater than 5), do this first: ### Run this: (replace '*' with your values (round to nearest integer)) quick_adcp.py --steps2rerun rotate:apply_edit:navsteps:calib --xducer_dx * --xducer_dy * --auto ### Run this: (look at the watertrack calibration again) catwt.py ### Run this: (look at the offset again) catxy.py ### If a scale factor or phase need to be applied, do that now (replace '*' with your values) ### Run this: quick_adcp.py --steps2rerun rotate:navsteps:calib --rotate_amplitude * --rotate_angle * --auto ### Run this: (look at the watertrack calibration again) catwt.py ### Run this: (look at the offset again) catxy.py <...> How do the calibrations look? See above for median tolerances and <...> repeat steps if the calibrations are still out of median tolerances. ------------------- ### 3. Editing points ------------------- <...> Edit out biased data or artifacts deeper than the range of the sonar <...> in the dataset, or other problems. We are looking for 1 cm/s accuracy <...> in ocean velocities, and any transitions larger than that could be suspect. <...> <...> Use `dataviewer.py -e` to look for problems with the data and <...> flag the bad data as bad. After editing out data we rerun the <...> calibration to see if the changed dataset statistics have <...> changed enough that a new calibration is needed. ------------------- ### Run this: (to go through the dataset and edit out bad values) dataviewer.py -e ### Run this: (to recompute the calibration residuals) quick_adcp.py --steps2rerun navsteps:calib --auto ### Run this: catwt.py ### Run this: catxy.py <...> The calibrations should still be good after editing. -------------------------------------------------- ### 4. Re-check heading correction and other figures -------------------------------------------------- <...> Check all figures from section 1 again to make sure that any problems <...> were addressed and no new problems have appeared after making changes <...> to the dataset. -------------------------------------------------- ### Run this: (Is there anything strange with any of the figures?) figview.py -------------------------------------------------------------- ### 5. Check edited, calibrated dataset against original dataset -------------------------------------------------------------- <...> Compare the edited, calibrated dataset against <...> the original dataset to make sure all problems have been dealt <...> with, and no new problems have appeared.There is likely to be strong <...> colors in the difference plot if big changes were made. <...> Compare sonars against the next nearest frequencies available <...> (Ex. os150 with wh300 and os75, not wh1200 and os38). <...> Compared datasets across different sonars should show a uniform <...> bias, Ex. all faint red or blue, equivalent to roughly 1 cm/s. <...> Look at the compared datasets on the default scale (0.8 days) and a <...> medium term scale (2 to 3 days) to look for longer scale bias. <...> When running dataviewer.py -c, put the finer resolution/smaller bin <...> sized data first to show the diff at the finer resolution. <...> NOTE that changes in bin size or number of bins will look like <...> a major problem. Just choose a different time range so the transition <...> is not in view and it will look better. -------------------------------------------------------------- ### Run this: (Has this corrected the problems in the original?) dataviewer.py -c . ../os38nb.orig ### Run this to compare this sonar with another sonar (after both are finished) dataviewer.py -c ../ANOTHER_SONAR . <...> Use the comparison to determine if anything else needs to be done. ------------------------- ### 6. Make plots and files ------------------------- <...> Create the figures and data files needed to finish processing <...> and make public/submit to a repository (JASADCP, if nowhere else). <...> Make the plots needed for web viewing, matlab files (legacy), <...> and netCDF files, then check that the netCDF files are readable. <...> Plots should be in 3 to 5 day chunks for a cruise longer than one week, <...> or divided by geographic features where it makes sense. <...> <...> If quick_web.py --interactive has been done for a different sonar, do: <...> Run this: (edit for your particular case) <...> mkdir webpy <...> cp ../ANOTHER_SONAR /webpy/sectinfo.txt webpy <...> quick_web.py --redo ------------------------- <...> If it's the first time: ### Run this: quick_web.py --interactive ### Run this to extract matlab files quick_adcp.py --steps2rerun matfiles --auto ### Run this to extract a netCDF file adcp_nc.py adcpdb contour/os38nb CRUISENAME os38nb --ship_name SHIPNAME ### Run this to look at the headers ncdump contour/os38nb.nc -h (unless otherwise stated, comments are relative to root processing directory) codaspy:(os38nb)$ =================================== (0) remake all the plots quick_mplplots.py --plots2run all --yearbase 2023 (1) look at heading correction: are there gaps? figview.py cal/rotate/*png [X] There were no gaps! [] There WERE some gaps! (2) see what calibration looks like: amp in [0.995, 1.005] : good enough to proceed to the next step. tweak later phase in (-.5, .5) : good enough to proceed to the next step. tweak later rule of thumb: errors when underway: scale factor of 1.02 (i.e. 2%) gives 10cm/s alongtrack error phase error of 1deg gives 10cm crosstrack error (goal is 1cm/s) tail -20 cal/watertrk/adcpcal.out Number of edited points: 40 out of 42 amp = 1.0038 + -0.0009 (t - 362.3) phase = 0.06 + 0.0647 (t - 362.3) median mean std amplitude 1.0040 1.0037 0.0064 phase 0.0760 0.0582 0.4702 nav - pc 2.0000 0.9500 3.3813 var 0.0010 0.0016 0.0016 min var 0.0010 0.0013 0.0015 delta-u -0.2000 -0.1125 3.3592 delta-v -1.3150 -0.7037 4.7748 No tweaks necessary. (3) gautoedit.py Minor edits.Lots of missing data julian day 361.5-362. Probably rough seas. (4) apply editing quick_adcp.py --steps2rerun apply_edit:navsteps:calib --proc_engine python --datatype uhdas --auto (5) rechecking calibration tail -20 cal/watertrk/adcpcal.out ** watertrack ** Number of edited points: 40 out of 41 amp = 1.0040 + -0.0010 (t - 362.3) phase = 0.06 + 0.0653 (t - 362.3) median mean std amplitude 1.0040 1.0040 0.0063 phase 0.0740 0.0576 0.4698 nav - pc 2.0000 0.9500 3.3355 var 0.0010 0.0016 0.0016 min var 0.0010 0.0013 0.0015 delta-u -0.2000 -0.1125 3.3598 delta-v -1.3150 -0.7030 4.7746 quick_adcp.py --steps2rerun rotate:apply_edit:navsteps:calib --rotate_amplitude 1.004 --rotate_angle 0.0576 --auto Number of edited points: 40 out of 41 amp = 1.0000 + -0.0010 (t - 362.3) phase = 0.00 + 0.0649 (t - 362.3) median mean std amplitude 1.0000 1.0000 0.0064 phase 0.0355 0.0031 0.4701 nav - pc 1.0000 0.7250 2.6697 var 0.0010 0.0015 0.0018 min var 0.0010 0.0013 0.0017 delta-u -0.2000 -0.1125 3.3598 delta-v -1.3150 -0.7030 4.7746 Very nice. (6) web plots (make sure to save this file now): Making web plots. mkdir webpy cp ../wh300/webpy/sectinfo.txt webpy/ quick_web.py --redo cd ../ cat os38nb/cruise_info.txt dm_notes_os38nb.txt > os38nb.txt ALL done