3.3.1. Troubleshooting Serial feeds: check ZMQ first

3.3.1.1. Overview of UHDAS and ZMQ

UHDAS was designed to record data when a cruise is active and “start recording” has been clicked (Monitor Tab shows green bars). When there is no active cruise (or recording is off) nothing is read from the serial ports, so there is no way to monitor any of the serial streams.

For ships that often have UHDAS logging secured, we have implemented a way to access ship’s position regardless of the UHDAS status. This is done using a protocol called “ZMQ”.

When properly configured, ZMQ listens to a specified serial port and “publishes” the information on an internal network (“tcp”) port. UHDAS then uses that tcp port to get position when it is logging data. We can check that the ZMQ process is running and can see the incoming data.

The penalty is that if the zmq publisher fails, it makes troubleshooting more difficult. This section describes:

1. How to tell whether UHDAS is configured to expect zmq_publisher.py to be running

2. How to troubleshoot serial feeds If zmq_publisher.py is not expected

3. How to troubleshoot serial feeds If zmq_publisher.py is expected

4. How to find clues in the daily email

Note

This page does not work with DAS_autopilot.py (autopilot is another level of difficulty)

3.3.1.2. (1) UHDAS configuration: does it expect ZMQ or not?

The critical configuration file for ZMQ is the same file that controls the serial logging information: ports, baud rates, which NMEA strings are being recorded, the use of checksums, which messages are being translated into “rbin” files, etc. It also controls whether ZMQ is expected, and how the speedlog output is set up. This file is /home/adcp/config/sensor_cfg.py.

This configuration file is outlined in the UHDAS+CODAS configuration files layout.

For zmq_publisher.py to be used, the following must be true:

• sensor_cfg.py must have the following line saying True:

use_publishers = True

If it says use_publishers = False, proceed to section (2) (troubleshooting when zmq_publisher.py is not supposed to be running)

If it says use_publishers = True, proceed to section (3) (troubleshooting when zmq_publisher.py is supposed to be running)

3.3.1.3. (2) Troubeshooting when ZMQ is not expected

Note

The USB-serial device probably labels the serial ports as [1,2,…8], expecting COM1, COM2, … COM8. Linux starts the serial ports with letter 0, so ttyUSB0, ttyUSB1,…

The file sensor_cfg.py uses the 0,1,…7 numbering scheme.

Be sure to refer to something like “linux port ttyUSB7” instead of “port 7” (which is ambiguous)

Before proceeding, - stop recording - kill the UHDAS GUI

Note

The UHDAS GUI MUST NOT be running when attempting to use tk_terminal.py or other serial diagnostics.

More about troubleshooting serial feeds in UHDAS is in the section about tk_terminal.py.

Look in /home/adcp/config/sensor_cfg.py for the serial settings.

If a serial feed turns red, the problem is likely to be in the feed, not UHDAS or the UHDAS computer. Before rebooting the UHDAS computer or otherwise messing around (ports, baud rates etc) connect another computer with the capability for logging serial data and verify that the - baud rate is what UHDAS expects - feed is what you think it is (eg. GPGGA or HEHDT)

Then work with the UHDAS Team to determine what might be the problem. The following is not a comprehensive list, but all the problems (or solutions) are on the UHDAS side of the cable: - baud rate needs to be changed - port name needs to be changed - serial port failed (move to another one; reconfigure) - serial port temporarily failed when the USB-serial unit lost power. Reboot. - serial message no longer has a checksum - serial message changed (now $INGGA not $GPGGA)

3.3.1.4. (3) Troubeshooting when zmq_publisher.py is expected

3.3.1.4.1. ZMQ publishers setup

At present, we genrally only read one feed with ZMQ. The ZMQ feed is the one specified in the publishers section of sensor_cfg.py. Look for this block:

publishers = [
               {
               'subdir'      :  'seapath',
               'pub_addr'    :  'tcp://127.0.0.1:38000', # uses this port
               'sample_opts' :  '-tc -s60',
               },
             ]

The subdir refers to a unique serial block earlier in the file.

The usual feed for zmq_publisher.py would be one with position.

Note

zmq_publisher.py is started at boot (50 seconds after boot) if the relevant line in sensor_cfg.py is set (use_publishers = True) when the computer is rebooted

Note

If the USB cable from the serial-USB device becomes disconnected from the UHDAS computer, zmq_publisher.py will halt. Other bad things will happen as well. Reboot the computer and proceed.

If use_publishers = True is set, run this command to see whether there are any ZMQ publishers running:

zmq_publisher.py --query

It will either return with no information (nothing running) or it will return with a complicated message that includes the tcp port being used. It will include a string like this, possibly with a different number: tcp://127.0.0.1:38000. You can verify that the feed is coming in by running this command:

zmq_monitor.py  tcp://127.0.0.1:38000

You have to type <control-c> to stop the scrolling (hold control-key down and then push c)

3.3.1.4.2. Troubleshooting

If you are troubleshooting some feed that is not related to ZMQ, go back to (2), and follow the normal serial troubleshooting procedure.

If you are troubleshooting the feed that is controlled by zmq_publisher.py, it is easiest to figure out the problem by:

• turn off zmq:

  • stop logging

  • end cruise

  • kill the UHDAS GUI

  • edit /home/adcp/config/sensor_cfg.py and change use_publishers = True to use_publishers = False

  • reboot

    • use the blue icon in the top-left corner with a white mouse outline

    • choose “Logout”

    • choose “Reboot”

• troubleshoot the serial port in question, i.e. go back to (2)

• when success is achieved, put the sensor_cfg.py entry back to the way it was before you started, with use_publishers = True or use_publishers = False

• If you changed to use_publishers = True, then reboot and wait an additional 60 seconds after the desktop lets you manipulate windows (waiting for ZMQ to start).

• then check zmq_monitor.py as above

When you start UHDAS and start logging, the left column of the UHDAS GUI will indicate which serial port is being used. The ZMQ feed will try to squeeze in the name tcp:127.0.0.1:38000 and the bar should be green.

3.3.1.5. (4) Clues in the daily email

The daily email should tell you whether zmq is enabled or not, and whether it is working. (newer emails are better at this than earlier emails)

If enabled, it says:

approximate lat, lon, depth:   31 42.91500 N   79 39.72510 W   depth=59
position from zmq

If not enabled, it should say

position from serial

If there is no position and no indication of where position is coming from, either the email is an older generation, or there is a zmq problem. - Look farther down for the zmq message status

If position from serial then you are done.

If ZMQ is enabled:

1. zmq is working fine:

========= zmq summary ==========
zmq is enabled
zmq publisher (ser_asc_zmq) is running

2. zmq is not working:

========= zmq summary ==========
zmq is enabled
zmq publisher (ser_asc_zmq)  is not  running

Serial ports for data logging appear to exist,
but zmq is not running
If this is the primary computer,
   ===> zmq_publisher.py should be restarted <===

You can always force an email and look at the contents and the directory. To force an email to go off the ship, run this command:

daily.py --use_defaults

To force an email to be staged, but not sent, run this command:

daily.py

The email content is staged in /home/adcp/daily_report and the contents of the daily email text are in /home/adcp/daily_report/status_str.txt.

3.3.1.6. Appendix

Random notes for experts:

• You can stop zmq_publisher.py by typing

  zmq_publisher.py --stop

• Even with the UHDAS GUI running, you can still use ports that are not mentioned in sensor_cfg.py, so if ttyUSB4 is not being used by UHDAS, you can plug something in and use tk_terminal.py to look at the feed

• You can start zmq_publisher.py by typing

  zmq_publisher.py --start --quiet

  … but it’s kind of pointless unless sensor_cfg.py says “use_publisher=True”

• If Autopilot is running (special configuration) and you need to troubleshoot a serial port, then

  DAS_autopilot.py --stop zmq_publisher.py --stop

.

…and when you are ready to go back to Autopilot, do this in a window you will not kill, or do it in a “screen” session:

zmq_publisher.py --start --quiet DAS_autopilot.py