LAST CHANGED : 2024/04/05 18:19:51 CRUISE NAME(S) : km2402, hot349 CRUISE DATES : 2024/04/01 18:04:11 to 2024/04/05 18:19:13 SHIP NAME : Kilo Moana PORTS : Honolulu, HI to Honolulu, HI CHIEF SCIENTIST : Tully Rohrer DATABASE NAME : a_km DATA FILES : km2024_091_64751.raw to km2024_095_64800.raw STATUS : to do done ------ ----------- averaged [ X ] loaded [ X ] NOTE: heading correction instrument exists NOTE: time-dependent heading corrections applied IN the ensembles (see cal/rotate/ens_hcorr.ang) check heading correction [ X ] calibration [ X ] edited [ X ] re-check heading correction [ X ] check editing [ X ] figures [ X ] INSTRUMENT : os38 ACQUISITION : PROGRAM : uhdas PROCESSING: : python LOGGING : PARAMETERS : BT : bottom track mode (on or off) SI : sampling interval or averaging period for ensemble (sec) NB : number of bins BL : bin length (m) TD : transducer depth (m) BK : blanking length (m) HO : heading offset applied by DAS (deg) HB : heading bias (deg) CRPH : compensation for roll-pitch-heading, 1:on, 0:off) yy/mm/dd hh:mm:ss BT SI NB BL TD BK HO HB CRPH 2024/04/01 18:04:11 off 300 115 12 7 16 44.55 0.00 0001 2024/04/02 19:04:13 off 300 115 12 7 16 44.55 0.00 0001 2024/04/03 20:04:15 off 300 115 12 7 16 44.55 0.00 0001 2024/04/04 21:04:11 off 300 115 12 7 16 44.55 0.00 0001 HEADING : PRIMARY : heading from gyro CORRECTION : heading correction from posmv NOTE: time-dependent heading corrections applied IN the ensembles (see cal/rotate/ens_hcorr.ang) POSITIONS : gps positions from posmv CALIBRATION : (check original processing parameters) (1) transducer alignment original transducer alignment: 44.55 additional rotation -0.1818, 0.1805 final transducer angle is: 44.55 - -0.1818 - 0.1805 = 44.5513 (original transducer angle) - (rotate_angle) (2) scale factor original scale factor 1 additional scale factor (1.0028) (3) ADCP (dx=starboard, dy=fwd) meters from GPS original: xducer_dx xducer_dy correction 16 6 final offset 16 6 final = original + corrections COMMENTS : gaps in heading correction? yes, jd 93.7 and 95.5-95.7. Data removed manually as gaps were too big for accurate correction scattering layers? no bubbles? yes, lots underway bias? Not really, compared to WH300 PROCESSOR : T. Rohrer --- final processing parameters from dbinfo.txt ---------- <...> NOTE: Replace the text below with the final dbinfo.txt content <...> after you are done with all postprocessing. Change the <...> comment to say "final processing parameters" ## (determined from "sonar"): model = os ## (determined from "sonar"): frequency = 38 ## (determined from "sonar"): instname = os38 ## (determined from "sonar"): pingtype = bb beamangle 30 cruisename km2402 datatype uhdas dbname a_km ens_len 300 fixfile a_km.gps frequency 38 hcorr_inst posmv instname os38 model os pingtype bb proc_engine python ref_method refsm refuv_smoothwin 3 refuv_source nav sonar os38bb txy_file a_km.agt xducer_dx 16 xducer_dy 6 yearbase 2024 <...> Please consider submitting your final processed data to JASADCP <...> (Joint Carchive for SHipblard ADCP) data so other people can benefit <...> from all the work that went into this. GoogleSearch for <...> "JASADCP shipboard ADCP" in your browser. <...> <...> NOTE: Markup description: <...> - Narration is preceeded by these characters: <...> <...> - Anything with those characters at the beginning of a line will be <...> stripped out by JASADCP before archiving. <...> ================================================== POST PROCESSING STEPS: ================================================== <...> Post processing commands start in the os38bb directory. <...> If the cruise dates are wrong in the header, adjust manually. <...> The time range of processed data can be found in os38bb/adcpdb/a_km.tr <...> This template covers most common situations when processing UHDAS data in CODAS. <...> For more detail, the latest version of the CODAS manual can be found at: <...> https://currents.soest.hawaii.edu/docs/adcp_doc/codas_doc/index.html -------------------------- ### 1. Check visual oddities -------------------------- <...> Perform a visual first pass of the data looking for the following problems: <...> (a) good heading correction for all of the ADCP data (any gaps?) <...> (b) sharp transitions in velocities <...> (i) on station vs underway (transducer angle or scale factor) <...> (ii) vertical stripes when transitioning between underway and <...> stopped (an offset between ADCP and GPS is not accounted for) <...> (iii) biases when underway with poor PG (will need to apply uvship) <...> (c) gaps due to "free inertial" flag in Seapath (heading exists, <...> but positions are missing, so no ship speed) <...> (d) false bottoms near steep topography: symptom strong scattering <...> layer is mis-identified as the bottom, resulting in shorter <...> profiles that stop at the scattering layer, but data on both <...> sides is much deeper <...> <...> When to post-process vs reprocess (eg. with adcp_database_maker.py) <...> You should reprocess if <...> (a) heading correction: if there are gaps and there is another accurate <...> heading device, you could reprocess using that other device <...> (b) sharp transitions in velocities <...> (ii) vertical stripes when transitioning between underway and stopped <...> (likely an offset between ADCP and GPS is not accounted for) <...> (iii) biases when underway with poor PG (will need to apply uvship) <...> If the offset is greater than 2m in x or y AND if there are periods <...> when the ship is underway in heavy weather, then you should reprocess <...> ensuring that you use the final xducer_dx and xducer_dy. Then the <...> 'uvship' algorithm can be attempted. Otherwise do not use it <...> (c) there are gaps due to "free inertial" flag in Seapath. In this case, <...> pick another position device, and make sure you get its xducer_dx and <...> xducer_dy correct. You might have to process twice, once to estimate <...> the values, and a second time to use them. <...> (d) false bottoms near steep topography: If the Percent Good is BLUE below <...> the scattering layers, you can only get those parts back by reprocessing <...> and NEVER looking for the bottom. Then you have to edit out the bottom <...> using the threshold tool for Bottom Selector. If the PG is red or orange, <...> you can resurrect those profiles by using "reset editing". <...> <...> Otherwise you may postprocess the cruise. <...> <...> Heading correction: <...> Check that we have a good heading correction for all of the ADCP data. <...> <...> There should be no holes in the heading correction graphs <...> (cal/rotate/ens_hcorr*.png), where "valid corrections" are green <...> circles and "bad corrections" are red crosses. Gaps in the plots where <...> there are no symbols are okay, no data was collected there. <...> <...> If there are red crosses on the heading correction graphs <...> (cal/rotate/ens_hcorr_*.png) then they need to be patched in the cal/rotate dir <...> using patch_hcorr.py. <...> -------------------------- ### Run this: (to look for gaps in the cruise track) plot_nav.py nav/a*.gps ### Run this: (to look for gaps in the heading correction, examine the character of <...> the watertrack and bottom track calibration, etc figview.py ### Run this: (looking for missing heading correction values or missing positions) dataviewer.py <...> NOTE: IF there are holes in the heading correction plots, you need to <...> run "patch_hcorr.py" to interpolate across the gaps. Do this BEFOE doing any <...> further post-processing ### Run this: (to interpolate for missing heading correction fixes) cd cal/rotate patch_hcorr.py cd ../.. ### Run this: (to check watertrack calibration after patching hcorr) catwt.py ### Run this: (to check watertrack calibration after patching hcorr) catbt.py ### Run this: (to check dxdy statistics calibration after patching hcorr) catxy.py --------------------- ### 2. ADCP calibration --------------------- <...> Calibrate the dataset in sum by checking the calibration values <...> and applying a fix to the necessary categories if necessary. <...> <...> When looking at a water track or bottom track calibration, we want to see <...> statistics inside the following values: <...> <...> median_tolerance <...> <...> - amplitude range .997 to 1.003 # 0.3% <...> - phase/angle range -0.05 to 0.05 # 1/2 degree <...> - offset (dx,dy) rounded residual should be between -2 and 2 meters <...> <...> When looking at an xy calibration to adjust the relative location of the GPS <...> to the sonar we try to get the dx and dy values close to zero, bearing <...> in mind that this is a guess. The "signal" should be between 1000 and 5000. <...> Less than 1000 means very little data, more than 5000 means too much change. <...> <...> If values are out of range we make a bulk correction to the entire dataset <...> which should reduce the number of outliers to be edited out. <...> <...> If we make any changes to x or y they should be integers. <...> <...> The command to be run to edit the calibration is listed below, where * <...> is replaced with desired values for the options to be changed and <...> other options removed: <...> <...> quick_adcp.py --steps2rerun rotate:navsteps:calib --rotate_amplitude *.** --rotate_angle *.** --xducer_dx * --xducer_dy * --auto --------------------- ### Run this: (look at the watertrack calibration residuals remaining) catwt.py ### Run this: (look at the bottom track calibration residuals remaining) catbt.py ### Run this: (look at the remaining offsets required) catxy.py --- ### If a large dxdy correction is needed (values greater than 5), do this first: ### Run this: (replace '*' with your values (round to nearest integer)) quick_adcp.py --steps2rerun rotate:apply_edit:navsteps:calib --xducer_dx * --xducer_dy * --auto ### Run this: (look at the watertrack calibration again) catwt.py ### Run this: (look at the offset again) catxy.py ### If a scale factor or phase need to be applied, do that now (replace '*' with your values) ### Run this: quick_adcp.py --steps2rerun rotate:navsteps:calib --rotate_amplitude * --rotate_angle * --auto ### Run this: (look at the watertrack calibration again) catwt.py ### Run this: (look at the offset again) catxy.py <...> How do the calibrations look? See above for median tolerances and <...> repeat steps if the calibrations are still out of median tolerances. ------------------- ### 3. Editing points ------------------- <...> Edit out biased data or artifacts deeper than the range of the sonar <...> in the dataset, or other problems. We are looking for 1 cm/s accuracy <...> in ocean velocities, and any transitions larger than that could be suspect. <...> <...> Use `dataviewer.py -e` to look for problems with the data and <...> flag the bad data as bad. After editing out data we rerun the <...> calibration to see if the changed dataset statistics have <...> changed enough that a new calibration is needed. ------------------- ### Run this: (to go through the dataset and edit out bad values) dataviewer.py -e ### Run this: (to recompute the calibration residuals) quick_adcp.py --steps2rerun navsteps:calib --auto ### Run this: catwt.py ### Run this: catxy.py <...> The calibrations should still be good after editing. -------------------------------------------------- ### 4. Re-check heading correction and other figures -------------------------------------------------- <...> Check all figures from section 1 again to make sure that any problems <...> were addressed and no new problems have appeared after making changes <...> to the dataset. -------------------------------------------------- ### Run this: (Is there anything strange with any of the figures?) figview.py -------------------------------------------------------------- ### 5. Check edited, calibrated dataset against original dataset -------------------------------------------------------------- <...> Compare the edited, calibrated dataset against <...> the original dataset to make sure all problems have been dealt <...> with, and no new problems have appeared.There is likely to be strong <...> colors in the difference plot if big changes were made. <...> Compare sonars against the next nearest frequencies available <...> (Ex. os150 with wh300 and os75, not wh1200 and os38). <...> Compared datasets across different sonars should show a uniform <...> bias, Ex. all faint red or blue, equivalent to roughly 1 cm/s. <...> Look at the compared datasets on the default scale (0.8 days) and a <...> medium term scale (2 to 3 days) to look for longer scale bias. <...> When running dataviewer.py -c, put the finer resolution/smaller bin <...> sized data first to show the diff at the finer resolution. <...> NOTE that changes in bin size or number of bins will look like <...> a major problem. Just choose a different time range so the transition <...> is not in view and it will look better. -------------------------------------------------------------- ### Run this: (Has this corrected the problems in the original?) dataviewer.py -c . ../os38bb.orig ### Run this to compare this sonar with another sonar (after both are finished) dataviewer.py -c ../ANOTHER_SONAR . <...> Use the comparison to determine if anything else needs to be done. ------------------------- ### 6. Make plots and files ------------------------- <...> Create the figures and data files needed to finish processing <...> and make public/submit to a repository (JASADCP, if nowhere else). <...> Make the plots needed for web viewing, matlab files (legacy), <...> and netCDF files, then check that the netCDF files are readable. <...> Plots should be in 3 to 5 day chunks for a cruise longer than one week, <...> or divided by geographic features where it makes sense. <...> <...> If quick_web.py --interactive has been done for a different sonar, do: <...> Run this: (edit for your particular case) <...> mkdir webpy <...> cp ../ANOTHER_SONAR /webpy/sectinfo.txt webpy <...> quick_web.py --redo ------------------------- <...> If it's the first time: ### Run this: quick_web.py --interactive ### Run this to extract matlab files quick_adcp.py --steps2rerun matfiles --auto ### Run this to extract a netCDF file adcp_nc.py adcpdb contour/os38bb CRUISENAME os38bb --ship_name SHIPNAME ### Run this to look at the headers ncdump contour/os38bb.nc -h (unless otherwise stated, comments are relative to root processing directory) codaspy:(os38bb)$ =================================== (0) remake all the plots quick_mplplots.py --plots2run all --yearbase 2024 (1) look at heading correction: are there gaps? figview.py cal/rotate/*png [] There were no gaps! [X] There WERE some gaps! Big gaps at 93.7, 95.3-95.5. Ran patch_hcorr.py. ---Original Calibration--- - Water Track Calibration: median mean std amplitude 1.0035 1.0036 0.0060 phase -0.2095 -0.0457 0.7023 ---New Calibration--- - Water Track Calibration: median mean std amplitude 1.0030 1.0034 0.0060 phase -0.2110 0.0870 0.8973 Correction not great on bigger gaps, will probably have to edit out data. (2) see what calibration looks like: amp in [0.995, 1.005] : good enough to proceed to the next step. tweak later phase in (-.5, .5) : good enough to proceed to the next step. tweak later rule of thumb: errors when underway: scale factor of 1.02 (i.e. 2%) gives 10cm/s alongtrack error phase error of 1deg gives 10cm crosstrack error (goal is 1cm/s) tail -20 cal/watertrk/adcpcal.out Number of edited points: 19 out of 23 amp = 1.0034 + -0.0005 (t - 93.8) phase = 0.09 + -0.0880 (t - 93.8) median mean std amplitude 1.0030 1.0034 0.0060 phase -0.2110 0.0870 0.8973 nav - pc 1.0000 2.9474 6.9400 var 0.0030 0.0057 0.0082 min var 0.0020 0.0044 0.0058 delta-u -1.6600 0.0158 4.7843 delta-v 0.1900 0.5142 3.6381 Continuing with editing. (3) gautoedit.py Removed the portions with the bad heading correction. Removed some below sea floor points. Lots of holes in the data, likely from the bad seas. Yucky cruise. (4) apply editing quick_adcp.py --steps2rerun apply_edit:navsteps:calib --proc_engine python --datatype uhdas --auto (5) rechecking calibration tail -20 cal/watertrk/adcpcal.out Number of edited points: 12 out of 14 amp = 1.0028 + 0.0002 (t - 93.6) phase = -0.18 + -0.1413 (t - 93.6) median mean std amplitude 1.0035 1.0028 0.0048 phase -0.2200 -0.1818 0.5178 nav - pc 1.0000 3.5833 7.0641 var 0.0030 0.0063 0.0099 min var 0.0025 0.0047 0.0068 delta-u -1.5500 0.1808 5.3355 delta-v -0.7250 -0.1175 3.8878 quick_adcp.py --steps2rerun rotate:apply_edit:navsteps:calib --rotate_amplitude 1.0028 --rotate_angle -0.1818 --auto Number of edited points: 12 out of 14 amp = 0.9969 + 0.0001 (t - 93.6) phase = 0.18 + -0.1428 (t - 93.6) median mean std amplitude 0.9980 0.9969 0.0047 phase 0.1450 0.1805 0.5169 nav - pc 1.0000 3.0000 6.9019 var 0.0040 0.0070 0.0102 min var 0.0035 0.0054 0.0069 delta-u -1.5500 0.1808 5.3355 delta-v -0.7250 -0.1175 3.8878 Overcorrected for some reason. Trying one more tweak. quick_adcp.py --steps2rerun rotate:apply_edit:navsteps:calib --rotate_amplitude 1 --rotate_angle 0.1805 --auto Number of edited points: 12 out of 14 amp = 0.9970 + 0.0002 (t - 93.6) phase = -0.00 + -0.1452 (t - 93.6) median mean std amplitude 0.9980 0.9970 0.0049 phase -0.0335 -0.0023 0.5258 nav - pc 1.0000 3.0833 6.6395 var 0.0035 0.0065 0.0097 min var 0.0035 0.0053 0.0067 delta-u -1.5500 0.1808 5.3355 delta-v -0.7250 -0.1175 3.8878 Good enough. (6) web plots (make sure to save this file now): Making web plots. mkdir webpy cp ../wh300/webpy/sectinfo.txt webpy/ quick_web.py --redo cd ../ cat os38bb/cruise_info.txt dm_notes_os38bb.txt > os38bb.txt ALL done