Correlation IDs

Application Development
1

Hello,

I am developing an application for a class C TTN device and have a couple of newbie questions, if someone would be kind enough to point me in the right direction. I’ve looked at many docs, posts.

What would be the recommended way to tie uplink responses to specific downlink commands?

I have seen in the docs and posts that custom correlation ids might be used for this purpose. (MQTT Server | The Things Stack for LoRaWAN) says that you can push a downlink with your own correlation id and then once the downlink gets acknowledged, a message is published to the topic ending /down/ack containing your custom id.

And I also saw in a post by @htdvisser “It is possible for a user to schedule a downlink with correlation ID as:up:trololololo . If that downlink is acknowledged by their end device, you may receive that correlation ID in your uplink.”

But how is this done? I assume that it it the NS that generates the /down/ack containing your own correlation id and it doesn’t reference subsequent uplinks. So how is it possible to receive a copy of your downlink custom correlation id in subsequent uplinks or to generate your own correlation ids in uplinks?

One last question, just checking … if I see a down/ack with my correlation id does that mean definitively that the end device has received my downlink payload uncorrupted?

Thank you for any assistance.

Chris

2

Include a (small!) id in your downlink and uplinks…

Nope. It means the downlinks was scheduled for transmission. There is no way the NS knows if the device received it or it was lost in noise.
To know you could include your unique (small!) id in the next uplink.

Keep in mind TTN (community network) allows just 10 downlinks a day and that is already pushing limits. LoRaWAN does not scale well with downlinks due to the half duplex nature of the radio and the airtime limitations gateways are subject to in large parts of the world. One or two downlinks a week is what works for LoRaWAN or adding sufficient gateways near the devices if you need more downlinks.

3

Thank you very much for the response Jac.

I’m solely using the TTN Community Edition to verify if LoRaWAN meets our requirements, we only need a few uplinks a day and a tiny payload. I’ve added gateways near our test devices as I found that even though theoretically in coverage I could not get end devices to reliably exchange information even with near-ish gateways. I assume the commercial TTI network will not be any better in that regard ¯_(ツ)_/¯

I must be missing something as I thought that the downlink sequence was /queued (on NS) /sent (from NS) and that /ack was generated by the NS in response to an uplink, is the /ack generated in response to communication from the forwarding gateway then?

I assumed that I’d be able to send a correlation id from the end device but couldn’t see how in the TTU documentation on that. Thank you for confirming the device itself sends these - I’ll look at the code examples more.

I wasn’t sure if @htdvisser meant that if you add an as:up:‘my_cid’ it would be automatically included in all uplinks as part of the lower level protocol rather that built into the frame by the application layer.

Best wishes,
Chris

4

How far are your gateway and node from each other?

And what is the RSSI, SNR of your uplinks?

5

And what are your application?

The more info we have the better we can guide you.

6

TTN does not modify your payload on downlinks or uplinks (apart from your own supplied encoder/decoder). Them modifying payloads would make writing firmware for end devices hard as you basically receive an array of bytes when a downlink arrives at the device. You define that byte array and TTN can’t modify it.
(Please don’t be tempted to use json or another ascii based format to transfer data, airtime is a precious commodity and ascii formatted messages waste a lot of airtime)

7

Hi and thank you again @kersing,

No, my payloads are binary; 1 or 5 bytes for downlinks and 5 bytes for uplinks.

Ah, so when you say send a “small” correlation id you mean I should add my own as part of the payload!

So, in a typical uplink MQTT msg, I might see these correlation ids:

"correlation_ids": [
  "as:up:01GNATSVGD3FHJF0W9M1RHR5NX", 
   ...
  "rpc:/ttn.lorawan.v3.NsAs/HandleUplink:01GNATSVGCH0YZVYA4FRMEJNHR"
]

I take it the as:up: is generated by the application server and is for internal TTx use? I don’t really see the point of sending custom correlation ids in downlinks only then, as referenced in MQTT Server | The Things Stack for LoRaWAN

The Things Stack supports some cool features, such as confirmed downlink with your own correlation IDs. For example, you can push this:

```
{

  • “downlinks”: [{*
  • “f_port”: 15,*
  • “frm_payload”: “vu8=”,*
  • “priority”: “HIGH”,*
  • “confirmed”: true,*
  • “correlation_ids”: [“my-correlation-id”]*
  • }]*
    }
    ```

But do either of you know what @htdvisser meant by his comment:
It is possible for a user to schedule a downlink with correlation ID as:up:trololololo . If that downlink is acknowledged by their end device, you may receive that correlation ID in your uplink.?

It seems to imply you will see your as:uptrololololo in an uplink which I have seen neither for confirmed nor unconfirmed uplinks?

@Johan_Scheepers - my application is a metered electricity dispenser. And there haven’t been any uplinks for me to read off the RSSI and SNR for a while on the unreliable nodes, the console has refreshed itself. I am not sure of the distance for any particular node. What RSSI, SNR might you look for to get reliable messaging? But please don’t spend your time on this, I can research that in the forums.

I was trying to understand the best way to associate pairs of downlinks and uplinks before looking at reception. And for that it seems I might be able to use fcnts for this purpose rather than a) using default correlation ids, or b) embedding custom cids as part of the end device payloads.

Thanks,
Chris

8

The confirmation of a downlink is part another uplink (depending on the device stack that may be the next uplink you schedule with application data or an immediate uplink with no application data, in the latter case I don’t expect you will get an uplink in your application and as a result no correlation id)

However, I would suggest you search the forum regarding confirmed downlinks. The generic consensus is you should avoid it. It is a mechanism that can burn airtime (and battery) quickly if not used carefully. Using an application mechanism is generally advised and if well implemented will cause far less issues (and retransmissions).

9

Whilst the notes make for the potential for adding in your own id’s to tie up the downlink confirmations, there won’t be a large body of expertise here, correlation ids are a thing for linking up all the different processes in the stack but they are a deep dive detail that aren’t required for day to day operations.

That and the fact that we (the community) never start from a place where downlinks are in the mix as a key element. And definitely never confirmed downlinks.

I’d include the downlink frame count in all the subsequent uplinks - that way if any uplinks get lost, the next time you hear from the device you will get confirmation of what it last heard.

But overall, for LoRaWAN, the majority of the logic needs to be autonomous - you adjust the settings that the device uses to make decisions - you can’t rely on the timeframe for delivery.

A forum search will reveal the details of what is mean’t by “the half duplex nature of the radio” - a gateway can’t hear uplinks whilst downlinks are being sent - resulting in potential packet loss.

10

Great - thank you very much for all your time Jac!
Best regards,
Chris

11

Thanks Nick,

That’s really interesting to hear it put like that, and I possibly need to rethink what we were planning. So would you go as far as to say TTx is not really suitable for an application where you need to control end device behaviour remotely using downlinks, even if only a few times a day and timing is not particularly important? Or are you solely talking about TTN and not TTI?

Regards,
Chris

12

The console is a live view, so ether you need to sit and watch it or wright the RSSI, SNR and SF to a DB for later analysis.

13

Ahh, gotcha, okay. Thanks, Johan!

14

No, not at all. But yes.

I put the above for emphasis.

It’s not TTN or TTI, they are one of the leaders in the LoRaWAN space, I’m referring to the suitability of LoRaWAN which is mostly about low power sensor devices with good radio range/penetration (rural/in-building).

TTN as a free shared community has a FUP policy of 30 seconds air time & a maximum of 10 downlinks a day. We quite like 1 a fortnight. TTI allows you freedom up to the legal limits. TTN has a soft limit on commercial deployment and isn’t routinely used for Class C, so you should look at the TTI offerings, not least because it would give you access to the support engineers at TTI to answer such details as the use of correlation id’s. The forum is volunteer run, some are end-users & some are solution providers & a mix in between.

LoRaWAN is very asymmetrical: downlinks are the work of the devil, they are hugely detrimental to the local network because [insert plea to read the existing posts on this commonly raised issue - forum search is your friend] the gateway can’t hear on any of its 8 receivers whilst it’s sending. And as a single transmitter serving a number of devices, it can all too easily run in to legal duty cycle issues where it can’t/won’t transmit as it has been too active for a period and needs a refectory period.

In a “chatty” radio environment the typical metric is 10% packet loss. So building some forward error recovery in to your uplinks is a good idea if there are going to be downlinks occurring as routine. There will be some mitigation due to the scheduled nature of Class C. But even the scheduling doesn’t bring certainty in reception.

One of the other common recurring themes on here is sharing enough that we can be more specific - like what it is precisely that the device needs telling that it can’t figure it out itself and what the consequences are for a device to not hear a downlink - which could be a couple of days or more whilst someone figures out, as an example, who put the builders skip in front of the gateway.

I’ve some very good uses of LoRaWAN for command & control but it is to C&C local autonomy with no critical impact if the device doesn’t get its new working parameters for a while. Like a deployment of water tanks across a field - if one of them doesn’t know it’s meant to be watering a little more often, the others effectively compensate. A single irrigation system could end up with dead crops.

15

I have a couple of nodes (Class A) that I need to send once a month a downlink to, these I want to ensure that the downlink reaches the node.

Instead of using confirmed downlinks, I have a watch dog byte in my payload. Normally the byte is 0, when I send the downlink (once at the most a month) in the window just after the uplink. In the next uplink if the node did received a downlink the watch dog byte is 1, the back end checks the watch dog. If the watch dog is not what the back end expected, it will resend the downlink in the next window after the uplink again. Until it gets conformation from the watch dog byte that the downlink were successful.
In this way most of the time I don’t need to send an unnecessary uplinks as conformation of successful downlinks.

I also made sure my nodes have good RSSI, SNR and SF and they have redundant gateways.

PS – My uplink is just a keep alive message every 2 hours, 2 bytes (1 byte for battery and 1 byte for watch dog), so no unnecessary airtime or battery usage.

16

How many attempts before back end gives up and simply alarms user to say not getting confirmed messgage getting though? Or does it keep trying ever after…

So potentially 12 attempts per day every day…. Unless curtailed :wink:

17

Keep on trying, I could change it, but I need the downlink successful. Very seldom saw a second attempted, this were even under development when I sent 2 or 3 downlinks per day per node for testing and doing this to 3 nodes at the same time. (maybe I were lucky, so don’t count on it)

Yes 12 per day, 84 per week, 360 per month. BUT

So very unlikely the builders skip parked in front of 2 or more gateways.

Responsible use is not only from 30sec per day uplink and 10 downlinks, also to try ensure using SF7 instead of SF8 (not always possible I know, sometimes you need use SF10) and redundant gateways

I would love to have a few nodes deployed in very busy gateway area, somewhere in the 1000’s

The point here for me is that there are ways of implementing ‘fairly reliable’ (not mission critical) ways to make your nodes to work. You can’t think that very low RSSI and bad SNR you are going to get reliability, doesn’t matter what RF technologies you are using, you are going to fail.

And one more point, also try to use same gain antenna on the gateway as on the node for applications where downlinks are used.

18

Is there a benefit of using ‘gain’ antennas ?

19

I think that whether 0dbi, 2.15 dbi, 3 dbi or whatever Johan means keep it symetric :wink: Not go for increased gain…

20

Spot on :slight_smile: - Which is partly why I chuckle a little when people talk of an SLA in context of a Radio/RF based system… even where the SLA relates to the harder wired bit of the system - principal losses of messages often at the RF stage - and often folk think they have hard backhaul when in fact some legs/hops there may also be RF based…and not necessarily known… :man_shrugging: