Mosquitto_sub piping not working with TTN v2

Application Development
1

I used to be able to get my payload decoded using this:

mosquitto_sub -h eu.thethings.network -t ‘+/devices/+/up’ -u $TTN_UID -P $TTN_AKEY -t ‘+/devices/larue_pro-mini-barometric_temp_001/up’ | jq -r ‘.payload’ | base64 --decode | od -t x1

after the update to TTN V2 I know the data segment is now payload_raw but for what ever reason, I can’t chain these pipes and get it working.

mosquitto_sub -h eu.thethings.network -t ‘+/devices/+/up’ -u $TTN_UID -P $TTN_AKEY -t ‘+/devices/larue_pro-mini-barometric_temp_001/up’ | jq -r ‘.payload_raw’ | base64 --decode | od -t x1

What I now get is the raw payload from jq but it does not pass on the pipe to base64. And this includes if I direct the mosquitto_sub output to a file using tee and then in another process tail( with follow ) that file to the rest of the pipe.

it’s a mystery I’d like to solve so any insight would be helpful.

UPDATE:
A work-around is to send the JSON message to a file:
mosquitto_sub -h eu.thethings.network -t ‘+/devices/+/up’ -u $TTN_UID -P $TTN_AKEY | tee /tmp/outfile.ttn

In another process I follow the file with tail -f and when a new line shows up, process just that line and wait again for another line. I put that in a script:

#!/bin/bash

tail -f $1 | while IFS=’’ read line; do
echo $line | jq -r ‘.payload_raw’ | base64 --decode | od -t x1
done

While not as clean/easy as a pipe chain it works.

Doug

2

Wow, thank you so much! This was sending me insane. Of course, Google is full of matches but they’re all about jq needing an explicit filter, which is a different issue.

I have a simple pipe chain like so:

mosquitto_sub <params> | jq <filter> | my_script.sh

Which silently fails. After a million attempts to narrow down the issue I’ve come to the same conclusion as you - jq does not pass its output through a pipe or redirect (but works fine to stdout) when called on the command line.

There’s two interacting weirdnesses going on here, which I’ll try to illustrate:

Works
mosquitto_sub <params> | jq <filter>

Fails
mosquitto_sub <params> | jq <filter> | tee out.txt

Works
mosquitto_sub <params> | tee in.txt
tail -f in.txt | jq <filter>

Fails
mosquitto_sub <params> | tee in.txt
tail -f in.txt | jq <filter> | tee out.txt

Fails
mosquitto_sub <params> | tee in.txt
tail.sh in.txt | jq <filter> | tee out.txt

Works
mosquitto_sub <params> | tee in.txt
tail_and_jq.sh | tee out.txt

Works
mosquitto_sub <params> | jq.sh

Fails
mosquitto_sub <params> | jq.sh | tee out.txt

Fails
mosquitto_sub <params> | tee in.txt
tail -f in.txt | jq.sh | tee out.txt

Summary: mosquitto passes output to jq fine, but then output is not available. mosquitto and jq in isolation are fine. But to get output from jq when fed by mosquitto, there needs to be a file inbetween AND the reading of that file and parsing by jq must be in the same script.

I don’t understand at all why this is the case, but at least we have a workaround! Pinning down the root cause is proving extremely elusive.

2 Likes
3

The jq command has an --unbuffered option, which might help with piping.

4

@tlu is does indeed. This was answered for me over on Stack Overflow. In summary, jq buffers its output when piped. To request that it not do that, use --unbuffered.

2 Likes