Raspberry Pi to monitor serial output of a node or TNN Gateway, and alert on Slack or Telegram

While trying to create log files for non-frequent network/firmware problems for the TTN Gateway, I ended up with my laptop connected to the gateway 24/7 since February 14th. So, I was kind of happy it finally failed and needed a reboot last Friday. :wink:

Rather than hooking up my laptop again, I’m now using a Raspberry Pi 3 to do the monitoring, prefix proper timestamps to the output, and as a bonus send notifications to a (private) Slack workspace and/or Telegram chat to know when to act.

The full thing (using Node.js) with instructions is on GitHub, but the main points regarding the UART:

  • One needs to connect the gateway’s RX to the Pi’s RXD. Earlier, with an FTDI cable I also noticed that an ESP8266 uses different labeling, so yes, I guess the TX and RX pins on the gateway are labeled wrong.

  • Raspberry Pis that use Bluetooth will have their preferred UART hooked up to that Bluetooth. You might want to disable Bluetooth which will automatically make that best UART be mapped to the Pi’s RXD and TXD pins; see the README.

  • An UART is not line-based but will receive series of bytes from which, for the TTN Gateway, one can assume that each message ends with a newline. So, to get a message, the JavaScript needs to await such newline(s):

    const raspi = require('raspi');
    const Serial = require('raspi-serial').Serial;
    
    let buffer = '';
    function log(data) {
      // To debug the raw stream, use: process.stdout.write(data);
      buffer += data.toString();
      if (buffer.indexOf('\n') > -1) {
        const lines = buffer.split('\n');
        // Log whatever complete lines from the buffer, up to the last newline
        for (let i = 0; i < lines.length - 1; i++) {
          console.log(`[${ (new Date()).toISOString() }] ${ lines[i] }`);
        }
        // The message following the last newline might not be complete yet; 
        // handle later
        buffer = lines[lines.length - 1];
      }
    };
    
    raspi.init(() => {
      const serial = new Serial();
      serial.open(() => {
        serial.on('data', data => log(data));
      });
    });
    

And yes, it’s a bit weird to (semi-)permanently connect a Pi to the TTN Gateway… But this also allows me to remotely access the logs, which are stored on the Pi. And I could even grant the TTN crew access if they want to experiment; I just need a relay then to remotely power cycle the gateway whenever needed. :wink: One could also easily parse the incoming messages to, say, create statistics. And when building one’s own firmware one could also log more details about the packets that are not yours, and then get even more details using https://github.com/anthonykirby/lora-packet

image

5 Likes

To see how often I get MQTT errors, and above all to be alerted when my LoRa card simply stops working and needs manual intervention (bug #29), I’ve meanwhile extended the monitor with support for specific watchdogs, and simple reports.

Like the following shows that the gateway was working just fine, until it rebooted for a firmware upgrade check, after which the LoRa card failed to receive any LoRa packets. A watchdog that expects at least one LoRa packet every 15 minutes (even one with a bad CRC would satisfy that watchdog), and another expecting one accepted uplink (forwarded using MQTT) at least once every 30 minutes, nicely detect that.

When configuring Slack to get both warnings and errors:

image

And when configuring Telegram to only see the errors (where the firmware upgrade has been configured as an error too, just to know Telegram is getting its messages…):

image

Alerts can be configured to be repeated at some interval until finally fixed, and will report that:

image

2 Likes

Thanks for sharing the alternative solution! Because I don’t use PlatformIO and wasn’t sure about the 3,3 Volt limit on the UART of a RaspberryPi, I modified your solution slightly to use a CP2102 USB to UART Bridge dongle connected to one of the USB ports

#!/bin/bash

LOG="serial-`date +'%Y%m%d-%H%M%S'`.log"
USB2UART=/dev/ttyUSB0

stty -F $USB2UART raw 115200 
cat $USB2UART | while read l; do echo "[`date +'%F %T'`] $l" | tee -a $LOG; done