Troubleshooting Serial feeds with ZMQ

Overview of UHDAS and ZMQ

UHDAS was designed to record data when a cruise is active and “start recording” has been clicked (Monitor Tab shows green bars). When there is no active cruise (or recording is off) nothing is read from the serial ports, so there is no way to monitor any of the serial streams.

For ships that often have UHDAS logging secured, we have implemented a way to access ship’s position regardless of the UHDAS status. This is done using a protocol called “ZMQ”.

When properly configured, ZMQ listens to a specified serial port and “publishes” the information on an internal network (“tcp”) port. UHDAS then uses that tcp port to get position when it is logging data. We can check that the ZMQ process is running and can see the incoming data.

The penalty is that if the zmq publisher fails, it makes troubleshooting more difficult. This section describes:

  1. How to tell whether UHDAS is configured to expect zmq
  2. How to troubleshoot serial feeds If zmq is not expected
  3. How to troubleshoot serial feeds If zmq is expected
  4. How to find clues in the daily email

(1) UHDAS configuration: does it expect ZMQ or not?

The critical configuration file for ZMQ is the same file that controls the serial logging information: ports, baud rates, which NMEA strings are being recorded, the use of checksums, which messages are being translated into “rbin” files, etc. It also controls whether ZMQ is expected, and how the speedlog output is set up. This file is /home/adcp/config/sensor_cfg.py.

This configuration file is outlined in the UHDAS+CODAS configuration files layout.

For ZMQ to be used, the following must be true:

  • sensor_cfg.py must have the following line saying True:
use_publishers = True

If it says use_publishers = False, proceed to section (2) (troubleshooting when ZMQ is not supposed to be running)

If it says use_publishers = True, proceed to section (3) (troubleshooting when ZMQ is supposed to be running)

(2) Troubeshooting when ZMQ is not expected

Note

The USB-serial device probably labels the serial ports as [1,2,...8], expecting COM1, COM2, ... COM8. Linux starts the serial ports with letter 0, so ttyUSB0, ttyUSB1,...

The file sensor_cfg.py uses the 0,1,...7 numbering scheme.

Be sure to refer to something like “linux port ttyUSB7” instead of “port 7” (which is ambiguous)

More about troubleshooting serial feeds in UHDAS is in the section about tk_terminal.py.

(3) Troubeshooting when ZMQ is expected

ZMQ setup

At present, we are only reading one feed with ZMQ. The ZMQ feed is the one specified in the publishers section of sensor_cfg.py. Look for this block:

publishers = [
               {
               'subdir'      :  'seapath',
               'pub_addr'    :  'tcp://127.0.0.1:38000', # uses this port
               'sample_opts' :  '-tc -s60',
               },
             ]

The subdir refers to a unique serial block earlier in the file.

The usual feed for ZMQ would be one with position. For ZMQ to work, the program zmq_publisher.py must be running.

Note

zmq_publisher.py is started at boot (50 seconds after boot) if the relevant line in sensor_cfg.py is set (use_publishers = True) when the computer is rebooted

Note

If the USB cable from the serial-USB device becomes disconnected from the UHDAS computer, zmq_publisher.py will halt. Other bad things will happen as well. Reboot the computer and proceed.

If use_publishers = True is set, run this command to see whether there are any ZMQ publishers running:

zmq_publisher.py --query

It will either return with no information (nothing running) or it will return with a complicated message that includes the tcp port being used. It will include a string like this, possibly with a different number: tcp://127.0.0.1:38000. You can verify that the feed is coming in by running this command:

zmq_monitor.py  tcp://127.0.0.1:38000

You have to type <control-c> to stop the scrolling (hold control-key down and then push c)

Troubleshooting

If you are troubleshooting some feed that is not related to ZMQ, go back to (2), and follow the normal serial troubleshooting procedure.

If you are troubleshooting the feed that is controlled by ZMQ, it is easiest to figure out the problem by:

  • turn off zmq:

    • stop logging

    • end cruise

    • kill the UHDAS GUI

    • edit /home/adcp/config/sensor_cfg.py and change use_publishers = True to use_publishers = False

    • reboot

      • use the blue icon in the top-left corner with a white mouse outline
      • choose “Logout”
      • choose “Reboot”
  • troubleshoot the serial port in question, i.e. go back to (2)

  • when success is achieved, put the sensor_cfg.py entry back to the way it was before you started, with use_publishers = True or use_publishers = False

  • If you changed to use_publishers = True, then reboot and wait an additional 60 seconds after the desktop lets you manipulate windows (waiting for ZMQ to start).

  • then check zmq_monitor.py as above

When you start UHDAS and start logging, the left column of the UHDAS GUI will indicate which serial port is being used. The ZMQ feed will try to squeeze in the name tcp:127.0.0.1:38000 and the bar should be green.

(4) Clues in the daily email

The daily email should tell you whether zmq is enabled or not, and whether it is working. (newer emails are better at this than earlier emails)

If enabled, it says:

approximate lat, lon, depth:   31 42.91500 N   79 39.72510 W   depth=59
position from zmq

If not enabled, it should say

position from serial

If there is no position and no indication of where position is coming from, either the email is an older generation, or there is a zmq problem. - Look farther down for the zmq message status

If position from serial then you are done.

If ZMQ is enabled:

  1. zmq is working fine:
========= zmq summary ==========
zmq is enabled
zmq publisher (ser_asc_zmq) is running
  1. zmq is not working:
========= zmq summary ==========
zmq is enabled
zmq publisher (ser_asc_zmq)  is not  running

Serial ports for data logging appear to exist,
but zmq is not running
If this is the primary computer,
   ===> zmq_publisher.py should be restarted <===

You can always force an email and look at the contents and the directory. To force an email to go off the ship, run this command:

daily.py --use_defaults

To force an email to be staged, but not sent, run this command:

daily.py

The email info is staged in /home/adcp/daily_report and the contents of the daily email text are in /home/adcp/daily_report/status_str.txt.