Missing fields when retrieving uplinks from storage

TTN V2 to V3 Upgrade V3 Integrations
1

Hi,

I wish to retrieve uplinked messages from my devices using Storage Integration. However, the uplink_message seems to be missing some fields, compared to what I receive using MQTT. In particular, I don’t get correlation_ids, uplink_message.session_key_id and uplink_message.uplink_token. Is that expected?

Sample message from MQTT:

v3/robryk-seemon@ttn/devices/gattiker-weiher/up
{
  "end_device_ids": {
    "device_id": "gattiker-weiher",
    "application_ids": {
      "application_id": "robryk-seemon"
    },
    "dev_eui": "A8404117F182664D",
    "join_eui": "7C4353C1232C196E",
    "dev_addr": "260B2290"
  },
  "correlation_ids": [
    "as:up:01F44VQPFH5ZCMZWA7GDEPQ2SM",
    "ns:uplink:01F44VQP909A5RX4AVQXFKVBXS",
    "pba:conn:up:01F3AVQXAJ9QS9FKWGFMQP1Q40",
    "pba:uplink:01F44VQP88AAZWWGYMC1E8YRE3",
    "rpc:/ttn.lorawan.v3.GsNs/HandleUplink:01F44VQP8ZY73FFX73TXKE6SF0",
    "rpc:/ttn.lorawan.v3.NsAs/HandleUplink:01F44VQPFH71ZV6WMEDRS18ECT"
  ],
  "received_at": "2021-04-25T15:52:56.562085679Z",
  "uplink_message": {
    "session_key_id": "AXi8P3j99a7MI+3xkqfsWw==",
    "f_port": 2,
    "f_cnt": 4333,
    "frm_payload": "DTUAtAEEAP////8=",
    "decoded_payload": {
      "ADC_CH0V": 0.26,
      "BatV": 3.381,
      "Digital_IStatus": "L",
      "Door_status": "OPEN",
      "EXTI_Trigger": "FALSE",
      "Hum_SHT": 6553.5,
      "TempC1": 18,
      "TempC_SHT": -0.1,
      "Work_mode": "IIC"
    },
    "rx_metadata": [
      {
        "gateway_ids": {
          "gateway_id": "packetbroker"
        },
        "packet_broker": {
          "message_id": "01F44VQP88AAZWWGYMC1E8YRE3",
          "forwarder_net_id": "000013",
          "forwarder_tenant_id": "ttn",
          "forwarder_cluster_id": "ttn-v2-eu-4",
          "home_network_net_id": "000013",
          "home_network_tenant_id": "ttn",
          "home_network_cluster_id": "ttn-eu1",
          "hops": [
            {
              "received_at": "2021-04-25T15:52:56.328227101Z",
              "sender_address": "52.169.150.138",
              "receiver_name": "router-dataplane-f8764784f-5hp49",
              "receiver_agent": "pbdataplane/1.5.2 go/1.16.2 linux/amd64"
            },
            {
              "received_at": "2021-04-25T15:52:56.328838736Z",
              "sender_name": "router-dataplane-f8764784f-5hp49",
              "sender_address": "forwarder_uplink",
              "receiver_name": "router-7665c7b677-tz5nb",
              "receiver_agent": "pbrouter/1.5.2 go/1.16.2 linux/amd64"
            },
            {
              "received_at": "2021-04-25T15:52:56.329885120Z",
              "sender_name": "router-7665c7b677-tz5nb",
              "sender_address": "deliver.000013_ttn_ttn-eu1.uplink",
              "receiver_name": "router-dataplane-f8764784f-n76ql",
              "receiver_agent": "pbdataplane/1.5.2 go/1.16.2 linux/amd64"
            }
          ]
        },
        "time": "2021-04-25T15:52:56.237019Z",
        "rssi": -117,
        "channel_rssi": -117,
        "snr": -6.8,
        "uplink_token": "eyJnIjoiWlhsS2FHSkhZMmxQYVVwQ1RWUkpORkl3VGs1VE1XTnBURU5LYkdKdFRXbFBhVXBDVFZSSk5GSXdUazVKYVhkcFlWaFphVTlwU2pOYVZFSnFUVmQzZEZwSE9ERmpWazVKVjBoc1VrbHBkMmxrUjBadVNXcHZhV1J0T0RGak1rNXhUVmhvV1dOVlRubE9hMVpFWkZob1MxSkljRzFSVTBvNUxqVjFORWswTkVGZlNXbDZTM2hNZG14aU9HZDViMmN1TFVWUVQyeDBiVE0xTkRBeFgxWTNhUzVXWmxoSFdFWnJhMFZpVVMxVE1rUnVWbFI1Wmt0WmIzWnNla0ZMUkd0SVVWbEtUWFUyVWw5SWFEVktZVzE0VERadlVFc3dVMGwwTjJWU2QydDJibGxIYUdNMGFFSlBibGwzU2xwbFFVMW9SR2t6T0cxZlExSmFNMlJzUWxrMU1HSkJPV3hmU0c5T056TlJYMDl6UVVzMWNYVjJiV05XVUV4S1RFVm9RamRRZFdwMWVFeHNRVlpaVEhKWk5ISjFla2xSTkdaeFduUkZPR3RoVURaV2FrTTBVRWxIVGxKaVVTNWFRa1pCWlhBMFgwZGpaVWxOZWsxTmRHVnBiRVZCIiwiYSI6eyJmbmlkIjoiMDAwMDEzIiwiZnRpZCI6InR0biIsImZjaWQiOiJ0dG4tdjItZXUtNCJ9fQ=="
      }
    ],
    "settings": {
      "data_rate": {
        "lora": {
          "bandwidth": 125000,
          "spreading_factor": 7
        }
      },
      "data_rate_index": 5,
      "coding_rate": "4/5",
      "frequency": "867900000"
    },
    "received_at": "2021-04-25T15:52:56.352075568Z",
    "consumed_airtime": "0.061696s",
    "locations": {
      "user": {
        "latitude": 47.28372706145917,
        "longitude": 8.552110791206362,
        "source": "SOURCE_REGISTRY"
      }
    }
  }
}

and the corresponding entry from https://eu1.cloud.thethings.network/api/v3/as/applications:/robryk-seemon/packages/storage/:

{
  "result": {
    "end_device_ids": {
      "device_id": "gattiker-weiher",
      "application_ids": {
        "application_id": "robryk-seemon"
      },
      "dev_eui": "A8404117F182664D",
      "dev_addr": "260B2290"
    },
    "received_at": "2021-04-25T15:52:56.562085679Z",
    "uplink_message": {
      "f_port": 2,
      "f_cnt": 4333,
      "frm_payload": "DTUAtAEEAP////8=",
      "decoded_payload": {
        "ADC_CH0V": 0.26,
        "BatV": 3.381,
        "Digital_IStatus": "L",
        "Door_status": "OPEN",
        "EXTI_Trigger": "FALSE",
        "Hum_SHT": 6553.5,
        "TempC1": 18,
        "TempC_SHT": -0.1,
        "Work_mode": "IIC"
      },
      "rx_metadata": [
        {
          "gateway_ids": {
            "gateway_id": "packetbroker"
          },
          "packet_broker": {
            "message_id": "01F44VQP88AAZWWGYMC1E8YRE3",
            "forwarder_net_id": "000013",
            "forwarder_tenant_id": "ttn",
            "forwarder_cluster_id": "ttn-v2-eu-4",
            "home_network_net_id": "000013",
            "home_network_tenant_id": "ttn",
            "home_network_cluster_id": "ttn-eu1",
            "hops": [
              {
                "received_at": "2021-04-25T15:52:56.328227101Z",
                "sender_address": "52.169.150.138",
                "receiver_name": "router-dataplane-f8764784f-5hp49",
                "receiver_agent": "pbdataplane/1.5.2 go/1.16.2 linux/amd64"
              },
              {
                "received_at": "2021-04-25T15:52:56.328838736Z",
                "sender_name": "router-dataplane-f8764784f-5hp49",
                "sender_address": "forwarder_uplink",
                "receiver_name": "router-7665c7b677-tz5nb",
                "receiver_agent": "pbrouter/1.5.2 go/1.16.2 linux/amd64"
              },
              {
                "received_at": "2021-04-25T15:52:56.329885120Z",
                "sender_name": "router-7665c7b677-tz5nb",
                "sender_address": "deliver.000013_ttn_ttn-eu1.uplink",
                "receiver_name": "router-dataplane-f8764784f-n76ql",
                "receiver_agent": "pbdataplane/1.5.2 go/1.16.2 linux/amd64"
              }
            ]
          },
          "time": "2021-04-25T15:52:56.237019Z",
          "rssi": -117,
          "channel_rssi": -117,
          "snr": -6.8
        }
      ],
      "settings": {
        "data_rate": {
          "lora": {
            "bandwidth": 125000,
            "spreading_factor": 7
          }
        },
        "data_rate_index": 5,
        "coding_rate": "4/5",
        "frequency": "867900000"
      },
      "received_at": "2021-04-25T15:52:56.352075568Z",
      "consumed_airtime": "0.061696s",
      "locations": {
        "user": {
          "latitude": 47.28372706145917,
          "longitude": 8.552110791206362,
          "source": "SOURCE_REGISTRY"
        }
      }
    }
  }
}
2

Yes. The storage integration only stores a subset of the data available in MQTT. In V2 only the uplink data (user data) was being stored. In V3 there is meta data available as well but data that is only useful at reception time and expires fast is not stored.

3

Would it make sense to make this clear in documentation? Currently Storage Integration API | The Things Stack for LoRaWAN claims that correlation_ids should be present (and the link to ApplicationUplink there is broken).

I would expect that session_key_id is useful indefinitely: the session_key_id and f_cnt pair is the simplest way to uniquely identify a message. It also allows me to group messages by sessions, which is a grouping that the sensor is aware of and might rely on (e.g. it seems sensible to have a sensor that has some parameters that can be set via downlink and that resets them to default values on rejoin).

I don’t see how to actually use retrieval from Storage Integration in this case. I would like to once in a while fetch any recent messages. I’m not sure how I can reliably identify if a message I’ve just retrieved is one I’ve already retrieved earlier:

  • I could try indexing messages by the received timestamp, but documentation about any timestamps present is very sparse and doesn’t guarantee their uniqueness,
  • I could try indexing messages by the hash of the complete uplink_message, but that contains information about all the gateways that received the message, so I expect that it might be amended if one more gateway (with very high latency) reports hearing the message after it has already been retrieved.

How would you suggest that I proceed, if I want to maintain a local database with a copy of all uplinks from my devices, and want it to be complete (i.e. contain all messages, and no duplicates) even if the process that fetches messages into it is down for something like a few hours?