Unique Uplink IDs in TTS v3.70

[johnwiles]

In the uplink payload for v3 there are fields: correlation_ids and uplink_token.
I’m understanding that these are unique ids, my question is: to what scope are these unique?

Are they globally unique, unique to my application, or unique to my network server, etc.

I’d like to know how hard I can rely on these fields being truely unique so I can handle de-duplication of messages in my processing queue for my client application.

Here is an uplink example for reference:

{
   "end_device_ids":{
      "device_id":"202653aa",
      "application_ids":{
         "application_id":"test-application"
      },
      "dev_eui":"947EB911000F04EA",
      "join_eui":"70B3D07ED00136DE",
      "dev_addr":"00F3FE3D"
   },
   "correlation_ids":[
      "as:up:01E6BYY2Y7FAKCWQQ754PX6B9F",
      "gs:conn:01E6BPJRHW64WYZBEVQZ2BCFF",
      "gs:uplink:01E6BYY2QFSTFQEN1EQFCAPRWBM",
      "ns:uplink:01E6BYY2QH24VAR9ZAS89KB8GX",
      "rpc:/ttn.lorawan.v3.GsNs/HandleUplink:01E6BYY2QHCTZ0YRC0V4QQ5TQR"
   ],
   "received_at":"2020-04-20T14:12:37.961145931Z",
   "uplink_message":{
      "session_key_id":"ASPK<EWB&3i3mqk5aNon4y7+tg==",
      "f_port":3,
      "f_cnt":105,
      "frm_payload":"CQAACgAACwAAHQFFBBEEeABBCCeAAEABBBA",
      "decoded_payload":{
      },
      "rx_metadata":[
         {
            "gateway_ids":{
               "gateway_id":"eui-397c5daabfe0071ef",
               "eui":"397C1D5AABFE0561A1"
            },
            "timestamp":1241518731,
            "rssi":-90,
            "channel_rssi":-90,
            "snr":8,
            "uplink_token":"QiIKIBoUZXVpLTY0N2ZkeFdmZmUwMDgxZjESCGR/3v/+XILxEIulgNAE",
            "channel_index":4
         }
      ],
      "settings":{
         "data_rate":{
            "lora":{
               "bandwidth":125000,
               "spreading_factor":7
            }
         },
         "data_rate_index":5,
         "coding_rate":"4/5",
         "frequency":"922800000",
         "timestamp":1241518731
      },
      "received_at":"2020-04-20T14:12:37.745813218Z"
   }
}

[htdvisser (Hylke Visser)]

I edited your post to improve the formatting of the uplink example.

If transmissions are received by multiple antennas or gateways, those will already be de-duplicated and merged by The Things Stack. For each of those duplicates, you’ll see an additional entry in rx_metadata, as well as correlation IDs starting with gs:conn: and gs:uplink. Unless configure your devices to skip the frame counter check, you should not receive the same message more than one.

If you have control over the application(s), you can use the correlation ID starting with as:up:. The token after as:up: is a ULID, consisting of a 48 bit timestamp and 80 bits of randomness.

If you don’t control the application(s), it’s important to keep in mind that there may be user-provided correlation IDs in the list in addition to the ones generated by The Things Stack. It is possible for a user to schedule a downlink with correlation ID as:up:trololololo. If that downlink is acknowledged by their end device, you may receive that correlation ID in your uplink.

[johnwiles]

Thanks for your reply, it’s very helpful.

Sorry I should have explained better; I’m placing incoming data into a queue, and processing items off that queue could potentially cause the duplications (I wasn’t worried about dupe’s from multiple active gateways, I know this is covered by the NS).

So having a ULID per uplink is fantastic to verify items in the queue only get processed once.
I think this is what I was looking for.

But I do understand the issue of a user generated correlation ID ending up in an uplink response and potentially confusing things. I think this may not be a problem for my application…

Does the uplink_token help get around this very slight complication?
I can’t find any information on how the uplink_token is generated?
And based on my testing with different gateways, this is being generated by TTS somewhere.

[htdvisser (Hylke Visser)]

The uplink token is indeed generated by The Things Stack. For gateways directly connected to The Things Stack, it’s a combination of the gateway ID/EUI, antenna index and a timestamp. For gateways connected through (Packet Broker) peering it could be anything.

I think you should just generate or compose your own unique ID when receiving an uplink. That way you have full control over the properties of that ID. Maybe you want to easily sort uplinks by time, in which case a ULID is great. Maybe you want to partition by network, tenant, cluster, application, etc. Then you could for example combine the fields in end_device_ids with the received_at timestamp.

[johnwiles]

Great to know how uplink tokens are generated!
Thanks for your help, I might just generate my own.