The Things Network V3 - API: Bad JSON Format

tomascharad · August 31, 2021, 12:57am

Hi!

When hitting: https://nam1.cloud.thethings.network/api/v3/as/applications/app-id/packages/storage/uplink_message?limit=2

I receive the following body:

{"result":{"end_device_ids":{"device_id":"t-beam-11-weather-station","application_ids":{"application_id":"apartment-weather-station"},"dev_eui":"000479C57080D2EE","dev_addr":"260C5681"},"received_at":"2021-08-29T00:01:04.680953136Z","uplink_message":{"f_port":1,"f_cnt":429,"frm_payload":"/3//fw==","decoded_payload":{"message":{"degreesC":99.951171875,"humidity":99.951171875}},"rx_metadata":[{"gateway_ids":{"gateway_id":"a84041ffff1ede24tomas","eui":"A84041FFFF1EDE24"},"timestamp":1356767240,"rssi":-50,"channel_rssi":-50,"snr":12.8,"channel_index":4}],"settings":{"data_rate":{"lora":{"bandwidth":125000,"spreading_factor":10}},"coding_rate":"4/5","frequency":"904700000","timestamp":1356767240},"received_at":"2021-08-29T00:01:04.465791633Z","consumed_airtime":"0.329728s","network_ids":{"net_id":"000013","tenant_id":"ttn","cluster_id":"ttn-nam1"}}}}
{"result":{"end_device_ids":{"device_id":"t-beam-11-weather-station","application_ids":{"application_id":"apartment-weather-station"},"dev_eui":"000479C57080D2EE","dev_addr":"260C5681"},"received_at":"2021-08-29T00:02:23.959239182Z","uplink_message":{"f_port":1,"f_cnt":432,"frm_payload":"/3//fw==","decoded_payload":{"message":{"degreesC":99.951171875,"humidity":99.951171875}},"rx_metadata":[{"gateway_ids":{"gateway_id":"a84041ffff1ede24tomas","eui":"A84041FFFF1EDE24"},"timestamp":1436054746,"rssi":-107,"channel_rssi":-107,"snr":-8,"channel_index":6}],"settings":{"data_rate":{"lora":{"bandwidth":125000,"spreading_factor":10}},"coding_rate":"4/5","frequency":"905100000","timestamp":1436054746},"received_at":"2021-08-29T00:02:23.744378167Z","consumed_airtime":"0.329728s","network_ids":{"net_id":"000013","tenant_id":"ttn","cluster_id":"ttn-nam1"}}}}

As you can see between every {"result": [...]} there are no commas and no brackets surrounding it, so it fails to comply with JSON standard and it fails when trying to parse it as is.

descartes · August 31, 2021, 8:17am

It’s an event stream, each line is a separate entity that is targeted towards Server Side Events like the console log etc.

It is a PitA for static downloads but I solve it like this:

github.com

descartes/TheThingsStack-Integration-Starters/blob/d438fcefdcf02614932013647329f69446f8ab8d/DataStorage-to-Tab-Python3/TTS.DataStorage.Tab.py#L52


print("URL: " + r.url)
print("Status: " + str(r.status_code))
print()




# The text returned is one block of JSON per uplink with a blank line between.
# Event Stream (see headers above) is a connection type that sends a message when it 
# becomes available. This script is about downloading a bunch of records in one go
# So we have to turn the response in to an array and remove the blank lines.


theJSON = "{\"data\": [" + r.text.replace("\n\n", ",")[:-1] + "]}";


someJSON = json.loads(theJSON)
#print(json.dumps(someJSON, indent=4))
someUplinks = someJSON["data"]


# Output to timestamped file
now = datetime.now()
pathNFile = "DataStorage-" + theApplication + "-" + now.strftime("%Y%m%d%H%M%S") + ".txt"
print(pathNFile)
if (not os.path.isfile(pathNFile)):

tomascharad · September 3, 2021, 8:55pm

Yeap! I did exactly the same. Thanks!