Only Reading the Payload Fields in MQTT

Gateways TTKG - The Things Kickstarter Gateway
1

Is it possible to receive just the “Payload Fields” via MQTT, filtering out the metadata, app_id, etc.? The SCADA application we’re trying to get this data to only supports JSON Schema A format, which just requires a timestamp, the tag name, the tag value, and nothing else. Like this:

{
“time” : 1584630099093,
“value” : 1234
}

But the current MQTT payload looks like this:

tns_home_sensors/devices/1939a0113/up
{
“app_id” : “tns_home_sensors”,
“dev_id” : “1939a0113”,
“hardware_serial” : “647FDA00000043CF”,
“port” : 20,
“counter” : 5610,
“payload_raw” : “gAEDAgTSOtk=”,
“payload_fields” : {
“time” : 1584630159094,
“value” : 1234
},
“metadata” : {
“time” : “2020-03-19T15:02:38.885002565Z”,
“frequency” : 904.6,
“modulation” : “LORA”,
“data_rate” : “SF8BW500”,
“airtime” : 25728000,
“coding_rate” : “4/5”,
“gateways” : [ {
“gtw_id” : “eui-647fdafffe00547b”,
“timestamp” : 1636785216,
“time” : “”,
“channel” : 8,
“rssi” : -73,
“snr” : 11,
“rf_chain” : 0
} ]
}
}

Subscribing to the individual tag Topic only returns the tag value which is also unsupported:

tns_home_sensors/devices/1939a0113/up/value
1234

2

I was never able to accomplish that on terminal, but this python code may work for you:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
#-----------------------------
#<-------- Libraries -------->
#-----------------------------
import paho.mqtt.client as mqtt
import json
import time
import datetime
import requests

#-----------------------------
#<------ Configuration ------>
#-----------------------------
#TTN Configuration
TTN_appeui = "<the application eui>"
TTN_appid  = "<the application id>"
TTN_password = "<ttn-account-XXXXXXXXXXXXXXX>"

#-----------------------------
#<-------- Functions -------->
#-----------------------------

def on_connect(mqttc, mosq, obj, rc):
    # subscribe to specific device in a specific app
    mqttc.subscribe('<theapplication_ID>/devices/<thedevice_ID>/up')

def on_subscribe(mosq, obj, mid, granted_qos):
    print("Subscribed")

def on_message(mqttc, obj, msg):
	
	try:
		x = json.loads(msg.payload.decode('utf-8'))
		alt_data = str("%s" % x['payload_fields'])		
		print(alt_data)
		print("------------------------")	
		print("")		
			
	except Exception as e:
		print(e)
		pass

#-----------------------------
#<---------- Main ----------->
#-----------------------------
mqttc= mqtt.Client()

# Assign event callbacks
mqttc.on_connect=on_connect
mqttc.on_message=on_message

mqttc.username_pw_set(TTN_appid, TTN_password)
mqttc.connect("us-west.thethings.network", 1883, 10)

mqttc.loop_forever()
1 Like
3

Hi @geordie_fox, you could use mosquitto_sub to subscribe to the TTN MQTT/JSON uplinks then pipe the output to jq to do whatever data reduction and conversion that you want and then pipe the jq JSON output into your system.

This is also very easy to automate on a Linux type system. I use this - very reliable.

1 Like
4

Aside: when using Paho in Python, see some notes about proper error logging and TLS.

1 Like
5

Thank you all for your responses. Sounds like it won’t be a quick and easy solution (for my skillset).

6

Is MQTT your only option? If, e.g., a HTTP POST would do as well, then you could clean up the TTN-posted payload in some intermediate server (or script on your computer, for testing) and then push that result to your application.

7

It is the main protocol we are looking to utilize right now, as the SCADA system has an integrated MQTT driver for data polling. I don’t think a driver exists for HTTP POST commands.

8

Then I’m afraid you’ll need to somehow run your own MQTT broker, to which some client (which connects to the TTN MQTT API, or which accepts an HTTP POST from the HTTP integration) can publish the exact messages you expect. Instead, I’d rather invest time and money into making the SCADA system handle any message format though. In fact, given its name, I’d expect some means of parsing/formatting to exist. But I don’t know SCADA.

Beware that this has been disabled in the EU region, and might be disabled in other regions too. I’d not rely on that. From the documentation:

Uplink Fields

Warning: not every cluster publishes uplink fields to individual topics. See status.thethings.network for details.

1 Like