Data Storage cURL interface for v3

TTN Network and Services The Things Stack v3
1

Hi,
I have been watching videos on you tube regarding how to download data collected on the server from a device. The video showed (see below) how to collect data with curl, and also had a button to test it. But I see its for version 2, and cant see the same for version 3. Is there a similar thing for V3 ?

Best Regards
Scott

2

No. But the documentation shows the required cURL commands so you can do it that way.

Iā€™ve altered the most generic & uninformative topic title Iā€™ve ever seen on this forum - please consider that you will get better responses if the volunteers answering can hone in on their areas of expertise.

3

Hi
Looking at my title itā€™s not great , having Dyslexia makes my job a bit harder. I have been a software and electronic engineer for over 20 years , and not been easy , but have designed many large systems.

I have looked at the documentation of storage integration and donā€™t find it as good as earlier versions . I think if I started using it in earlier versions then it would be easier to learn, as there are good articles , and easy to follow. But jumping in at v3 doesnā€™t make it easier.

Scott

4

Hi,
I have been trying for a while to read data collected in my account using Storage Integration V3, but not having luck. Just wondering if someone can point me to where I am going wrong?

Below you can see where I activated the storage.
Storage Int1

Below you can see where I setup the Key.
Storage Int2

I have written c code to exectute curl commands as follows (I have used the curl in may applications in the past so know it works), Im not sure about the uplink_message I added below:

 hnd = curl_easy_init();
  if(hnd) {
  struct curl_slist *slist1=NULL;
  slist1 = curl_slist_append(slist1, "Authorization: NSXS.PFEOGHX2SCCZC3Y6VHIOXJELCxxxxxxxxxx");
  slist1 = curl_slist_append(slist1, "Accept: text/event-stream");

  curl_easy_setopt(hnd, CURLOPT_URL, "https://eu1.cloud.thethings.network/api/v3/as/applications/xxxxx/devices/xxxxx/packages/storage/uplink_message:9");

  curl_easy_setopt(hnd, CURLOPT_POSTFIELDS, "limit=10&after=2022-03-29T10:45:49.435171464Z");

  curl_easy_setopt(hnd, CURLOPT_HTTPHEADER, slist1);
  curl_easy_setopt(hnd, CURLOPT_WRITEFUNCTION, got_data);
    ret = curl_easy_perform(hnd);
   
    if(ret != CURLE_OK)
      TRACE("Error");

    curl_easy_cleanup(hnd);

When I run the code it returns:

{ā€œcodeā€:12,ā€œmessageā€:ā€œNot Implementedā€}

Any help would be great

Best Regards
Scott

5

What is the :9 at the end of the URL for?

Have you tried it on the command line as suggested above? See Retrieve Messages | The Things Stack for LoRaWAN

Always best to do the simplest thing first, so leave out the limit & after clauses until youā€™ve got it working.

If you want something to compare & contrast with, hereā€™s a version in Python: TheThingsStack-Integration-Starters/DataStorage-to-Tab-Python3 at main Ā· descartes/TheThingsStack-Integration-Starters Ā· GitHub

Please do not open a new topic on the same subject. Please do use the tools on the toolbar to format the code so itā€™s readable - the </> for code - see How do I format my forum post? [HowTo]

6

Hi
The :9 from the end of the code was a typo.
I removed the ā€œlimit=10&after=2022-03-29T10:45:49.435171464Zā€ from my code , and works fine. I have been trying for a while getting to get the ā€˜limitā€™ and ā€˜afterā€™ to work, but no luck. I have tried using the command line , and checking the documentation, but no luck. I havent done Python, so cant try this. Can anyone let me know how they get these 2 commands to work, with an example?

Best Regards
Scott

7

Does limit work on its own?

Does after work on its own?

Even if you havenā€™t ā€œdoneā€ Python, you could look at it even if you donā€™t run it - mostly because that provides a working example ā€¦

8

i did try both on their own as you suggested before, and didnt make a difference. But after looking at the Python, I found the limit and after on the same line as the address , and that worked. I now have:

curl_easy_setopt(hnd, CURLOPT_URL, ā€œhttps://eu1.cloud.thethings.network/api/v3/as/applications/demoboard/devices/xxx/packages/storage/uplink_message?order=received_at&type=uplink_message&limit=10&after=2022-03-30T12:45:00.000000000Zā€);

I see each reading is JSON , and has two characters 10 after each one. Can I assume this always happens? This way I can easily decode each JSON entry between the two characters.

Scott

9

Curl by default does a post rather than a get - with a -G on the line as per the TTI docs it takes the -d (datas) and appends it to the URL and does a get.

As Iā€™ve not tried curl via a C app this is a subtly that I hadnā€™t considered. PHP, Python and Javascript for applications, C & C++ (and a bit of Ada) for firmware is more than I can keep track of.

The docs and the various implementations Iā€™ve done donā€™t have any decimals for seconds. Seems unlikely to cause an issue but just be aware of it. I normally request a small overlap in time and deduplicate as I process because the transfer to the database is buffered - so several minutes can go by without any uplinks being available - I go back 5 minutes. Alternatively use a nice rounded number like on the hour and retrieve about 5 past the hour.

Yes, itā€™s a minor PitA but itā€™s due to it being an event stream - which means you can, in theory, open a connection and receive data as it happens. So line 52 of the Python theJSON = "{\"data\": [" + r.text.replace("\n\n", ",")[:-1] + "]}"; removes the two carriage returns.

1 Like