Accessing decrypted data in MQTT in node-red

1

I am setting up my node-red again after all the issues with V3 of TTN. I bought a new gateway as the Things Network gateway did not support upgrading to V3 (as far as I know and believe me I tried).
Anyway I then found out node-red support has gone so swapped to MQTT. I followed the instructions on this site and :

  • created a gateway
  • created an application
  • added a device (my Pycom FiPy)
  • added an MQTT integration and generated an API key
  • setup MQTT in node-red with
    ** the server name as au1.cloud.thethings.network, port 8883,
    ** MQTT v3.1.1 and
    ** security as my gatewayID#ttn,
    ** topic, QoS=0,
  • MQTT connects and I am getitng the messages from my device in both live data under the TTN console and in node-red

The problem is that the payload is encrypted, e.g. for a payload of “hello world” I am seeing

{“f_port”:2,“f_cnt”:5,“frm_payload”:“AAAABUCKacdCLbLJQiVAAECLxddHxbVgQi4JKkLG4lRDkI7gvYXAALx4AAA/fJAA”,

I am using ABP and have double checked my NWSkey and AppSKey in my device and the application.
What is confusing me is that the message appears twice in “live data” as
Link to image showing live data

Now I know that I am seeing ascii on the ‘mac payload’ and it is indeed “Hello World” but clicking on this link never shows the ascii data in hex nor ascii. Where is it and why is it not in the packet?

Then on my node-red I am seeing all of the meta-data, but the payload seems weird

"data": {
    "@type": "type.googleapis.com/ttn.lorawan.v3.UplinkMessage",
    "raw_payload": "QFgsACYAAwACoAxxwhK5K0YvEkdlZ5X7",
    "payload": {
      "m_hdr": {
        "m_type": "UNCONFIRMED_UP"
      },
      "mic": "ZWeV+w==",
      "mac_payload": {
        "f_hdr": {
          "dev_addr": "26002C58",
          "f_ctrl": {},
          "f_cnt": 3
        },
        "f_port": 2,
        "frm_payload": "oAxxwhK5K0YvEkc=",
        "full_f_cnt": 3
      }

Isn’t the “frm_payload” supposed to be unencrypted?

2

I did a bit more digging and created a payload formatter that is just:

function Decoder(bytes, port) {
  // Decode plain text; for testing only 
  return {
      myTestValue: String.fromCharCode.apply(null, bytes)
  };
}

Now I have new meta data in the node-red packet that says:

decoded_payload: object
    mytestValue: "hello world"

But I really want to pass the hex buffer of 48 values unchanged as they arrive from the FiPy as I jave encoded all my data into byte pairs and already have Node-red functions to unpack the array and convert back to floating point values.
Do I need a payload formatter when I dont want to format the payload?

3

Yes it does! At the risk of causing issues given your attitude to others who try to help in other post (Access Key) wrt being pointed in that direction can I ask… have you read and followed the documentation?! If so what did you do, what did you see, what problem did you find? Details, details… your assertion clearly incorrect as a generalisation given the hundreds (thousands?) of TheThings Gateway users who have their units running under V3. - so let’s investigate what is specific to your case? Hardware? Software? Firmware? Meatware? Documentation?..

1 Like
4

It isn’t encrypted, it’s encoded as Base64 - see https://www.thethingsnetwork.org/docs/devices/bytes/ - I know you’ll bypass the Red banner :wink: - it’s a general article so it’s OK to do so in this instance.

You can then write your own payload formatter / processor in Node-RED - decode the frm_payload and you have your byte array to do as you wish with.

5

Thanks Nick, I got it all going after reading that link

6

Hi Jeff, I spent literally weeks trying to get my TTN gateway to V3. I followed the documentation to the letter (I am a software engineer) but ran into issues as the documentation was for EU bands and I run on Au bands. I posted questions on what settings were appropriate for AU (after searching hundreds of pages of on-line information) but did not get any response. At that point, given my TTN gateway was very unreliable at the best of times (I auto reboot it every 30 minutes and it rarely stayed up that long), I ditched it.
Recently I bought a new gateway (MikroTik wAP Lora9) and am pleased that it has stayed running from the moment I powered it up.

So sorry about my ‘attitude’ but it is a result of a long period of frustration. Now that I have a reliable gateway I am keen to see if I can re-establish the free public domain network that was using my gateway in the community. I am already seeing quite a few devices attach via it.

I also realise that you experts deal with hundreds of questions every day that could easily be found in the documentation. I run my own wiki in my specialist field and deal with this also. So thanks and sorry for my attitude, I will work on improving it :slight_smile:

7

:thinking: Would likely have ditched it too or done a tear-down and rebuild as something very wrong in that case :wink:
I managed with mine and I am not a software engineer (as Nick would attest!), but it sounds like you were hindered by unreliable kit.

Good choice - I also use a wAP Lora (8) - assume similar - only downside I find is a bit over complex in setting up as a simple GW, but good for someone into networking config or software (and I’m not… as above!)
Given stability and your ‘progress’ message to Nick above it sounds like you are heading the right way… good luck! :slight_smile:

8

Hi Nick, that link you sent me to does not cover base64. Its mainly an intro to bits and bytes.
I use a byte array in micropython to pack my data

    data = bytearray(48)
    data[0:4] = bytearray(struct.pack(">i", count))
    data[4:8] = bytearray(struct.pack(">f", vt))
    data[8:12] = bytearray(struct.pack(">f", dew))
    data[12:16] = bytearray(struct.pack(">f", temp1))
    data[16:20] = bytearray(struct.pack(">f", roll1))
    data[20:24] = bytearray(struct.pack(">f", press1))
    data[24:28] = bytearray(struct.pack(">f", temp2))
    data[28:32] = bytearray(struct.pack(">f", hum1))
    data[32:36] = bytearray(struct.pack(">f", relhum))
    data[36:40] = bytearray(struct.pack(">f", acc1))
    data[40:44] = bytearray(struct.pack(">f", acc2))
    data[44:48] = bytearray(struct.pack(">f", acc3))

I was used to seeing this arrive in Hex, which I can pretty much decode in my head as I am VERY OLD so have been working in hex for decades.

I used the free base64 decoder at https://www.base64decode.org/ and put in “oAxxwhK5K0YvEkc=” and it did not show up as “hello world”. Am I missing something?

9

It’s a Russian dolls problem - the payload you’ve quoted is the MAC payload which is encrypted but the frm_payload that is delivered to you by MQTT / Webhook / ANO integration is decrypted.

You’ve got the JSON from the line that says “Successfully processed data message” which is the Network Server saying “Ta Da!”:

"name": "ns.up.data.process",
  "data": {
    "@type": "type.googleapis.com/ttn.lorawan.v3.UplinkMessage",
    "raw_payload": "QKAcASYAt6MBG9bNwcLGp7fV9JZIyQ==",
    "payload": {
      "m_hdr": {
        "m_type": "UNCONFIRMED_UP"
      },
      "mic": "9JZIyQ==",
      "mac_payload": {
        "f_hdr": {
          "dev_addr": "26011CA0",
          "f_ctrl": {},
          "f_cnt": 41911
        },
        "f_port": 1,
        "frm_payload": "G9bNwcLGp7fV",
        "full_f_cnt": 172983
      }
    },

Whereas the very next line (which will be above the NS line on the console) labeled “Forward uplink data message” is the Application Server saying “Here you go”:

  "name": "as.up.data.forward",
  "data": {
    "@type": "type.googleapis.com/ttn.lorawan.v3.ApplicationUp",
    "end_device_ids": {
      "device_id": "descartes-groffice",
      "application_ids": {
        "application_id": "descartes-office-sensors"
      },
      "dev_eui": "DE5CA7FFFFFFFFF9",
      "join_eui": "70B3D57ED0025585",
      "dev_addr": "26011CA0"
    },
    "received_at": "2022-01-24T10:23:37.089503926Z",
    "uplink_message": {
      "f_port": 1,
      "f_cnt": 172983,
      "frm_payload": "AAP/AsIBcwYq",

The outgoing JSON from any integration will be this second one.

My apologies, I appear to have confused you with my earlier assertion that it’s not encrypted - the bit you quoted, in retrospect was, but the general principle that the data we get out of TTS via normal routes isn’t encrypted, just encoded. I’ll re-read my TTS JSON formats book, it’s a real potboiler.

Additional, for anyone else finding this:

  • As the frm_payload is decrypted, best practise is to transfer via an encrypted channel like HTTPS.
  • You can actually elect to turn off decryption, but you need the devices keys to do the decryption yourself.
  • The truly paranoid encrypt the payload on the device using their own scheme before giving it to the LoRaMAC to encrypt - so then the frm_payload is a Base64 encoded hex array that is still encrypted using your own scheme.
  • I’m so old that I can still read punch tape
  • Screenshot 2022-01-24 at 10.38.05
  • But not so old that I’ve heard the pulses in a mercury delay tank. I think Alan was right, Gin would have been better.
10

That’s clear now. I have my node-red processing just fine, and my two heltec LoRa Wifi 32 (V1) units plus FiPy all working well. I note that only Heltec V2 units are available as ‘drop down’ devices but had no issue with V1 set up manually. I use ABP as I never got OTAA to work with my Things Gateway, maybe I should give this a go now with with a real gateway?
Every so often I get a node-red error message that says the data packet does not contain ‘object’ data so i assume this is some sort of status message maybe? My dim memory tells me that there was a recommendation to have a node-red function that ditched these messages and did not pass them onto the data decoder. Do you know what I am referring to?

Also why am I seeing “schedule data download for transmission” with MAC in my application console? Is the application trying to send back to my ABP device? If I used OTAA would I see these messages arrive and if so, what is in them?