Best practices to limit application payloads

Application Development
1

Application payloads should really be limited, if only to comply with the TTN Fair Access Policy of 30 seconds per day for uplinks, and 10 messages per day for downlinks. Let’s compile a checklist of best practices to achieve that:

  • Always, always, always use binary encoding to send your data. See The Things Uno MQTT Workshop for an introduction to TTN’s payload functions to help you decode it.

    Why? From The Things Network 2016 Update, where “SF7” is actually not bad at all:

    Don’t waste your time!

    • Simple:
      • { “Count”: 1234, "Temperature": 20.635 }
      • 40 bytes: 292 messages per day (SF7)
    • Remove counter (is already included in header†), spaces, and compress names:
      • {“t”:20.63}
      • 11 bytes: 486 messages per day
    • No JSON:
      • 20.63
      • 5 bytes: 582 messages per day
    • Signed 16 bit integer
      • 0x080F
      • 2 bytes: 648 messages per day

    And from Slack:

    We’re definitely moving away from JSON (and even plaintext) data. The MQTT or API would just give you the (base64 encoded) binary data that your device sent. And then you can run it through ConCaVa. Or just convert the binary data to whatever value you want with two lines of code.

  • Do not send a precision that is higher than your use case needs, or your sensors offer. Like a GPS unit might be less accurate than you think; see Best practices when sending GPS location data.

  • Combine sensors in a single device, or when collecting data that is not needed realtime then consider sending in small batches. As the LoRaWAN protocol always adds at least 13 bytes of overhead to the application payload, and also needs some air time to send some more housekeeping data, a 55 bytes application payload might only need 2 to 2.5 times more air time compared to 8 bytes. See Spreadsheet for LoRa airtime calculation, and compare different payload sizes with different spread factors.

  • When a single node has multiple sensors for which to expect similar readings, maybe a single (multi-byte) value of one sensor plus (smaller) offsets for the other sensors can decrease the data size.

  • Only send (in short intervals) when the data has changed more than some threshold. (But consider sending more often whenever the data changes, if only as that makes it easier to detect faulty sensors that emit readings that change randomly.)

  • If it’s about finding lost things or reading data on command (rather than tracking continuously), assuming a LoRaWAN Class A device (where an end-device can receive some data right after sending some), and when not expecting the device to move out of range: make the device send a small “alive” packet quite often, and only if the application responds in some specific way, send a measurement. (This might also preserve battery life.) (it seems even “alive” packets need quite some air time and would quickly hit the maximum daily Fair Access Policy limit; to be continued…) But note that the application should then only reply in a specific way when data is needed, and keep silent if not:

    Fair Access Policy: Practice

    • […]
    • Downlink bandwidth is even more restricted
      • you can’t send all messages as ‘confirmed uplink’

    …which might actually be limited to only 10 downlink messages per day!

  • When sending battery level, there is no need to include that in every single packet.

  • If your node can send different message types: use the LoRaWAN “port” to define what payload is sent.

  • If you have old and new nodes that use different encodings of their data: as one can always determine the node’s ID from the LoRaWAN header, one does not need to send any software version if one keeps a proper administration of node IDs and their installed software.

† Beware that FCntUp might also be increased for non-application uplinks, like for responses to MAC commands.

14 Likes
2

I like to share one thought on the effect of encoding payload in a complex way so the only one that can use the data is the specific application.

One of the advantage of IoT technology is the possibility to share data coming from a node among many different application.

To allow this a “talking paylod”.is needed.

3

By design, given LoRaWAN security, a single specific application is the only one that can decrypt the data. You need an application to process the data from the network server; you cannot just make applications connect to some node or gateway.

First, TTN’s dashboard and command line tools make security easy by handling the security keys for you. Next, TTN also allows you to define decoding functions:

function Decoder (bytes, port) {
  // Sign-extend to 32 bits to support negative values, by shifting 24 bits
  // (too far) to the left, followed by a sign-propagating right shift: 
  var data = bytes[2]<<24>>16 | bytes[1];
  return {
    temperature: data / 100
  };
}

And finally, you can grant others access to the data. And presto: MQTT serves an additional readable fields to whomever you grant access, even though the node sends it in a binary format:

{
  "temperature": 21.5
}

(Full example and full MQTT response in Best practices when sending GPS location data.)

5

Very nice library by @joscha, as mentioned elsewhere, with good documentation:

6

For information, see Low Power Payload (LPP) that allows the device to send multiple sensor data at one time.

7

LPP needs two additional bytes for each value that is being sent. Though TTN supports this as an integration as well, I’d not qualify that as a best practice. Nice for prototyping, of course.

8

Of course :slight_smile:
Only useful for “universal” nodes.

Also one could be inspired by the LPP binary format (i.e. 9 bytes for the GPS location)

We coud imagine that the node send “once” the description of his payload in LPP (with a special LPP/channel of FF) and for the others packets, just the meaningful data ?

Payload (Hex) 03 67 01 10 05 67 00 FF

Data Channel,Type,Value
03 ⇒ 3, 67 ⇒ Temperature, 0110 = 272 ⇒ 27.2°C
05 ⇒ 5, 67 ⇒ Temperature, 00FF = 255 ⇒ 25.5°C

Could be described as i.e. FF 03 67 05 67 in the first packet, and 01 10 00 FF in next packets.

9

Unless you’re using confirmed uplinks for the first uplink, I guess you’d need to send the LPP details a few times, to be (quite) sure they arrive. But the idea would be the same.

However, I’ve never understood the need to do the administration in the node. Like I wrote elsewhere:

And as an aside:

Don’t forget negative temperatures :wink: