Data Storage Integration Issue

[dieguitto]

When I get https url, response is a invalid json:

{"result":{"end_device_ids":{"device_id":"***","application_ids":{"application_id":...}
{"result":{"end_device_ids":{"device_id":"***","application_ids":{"application_id":...}
{"result":{"end_device_ids":{"device_id":***","application_ids":{"application_id":...}
{"result":{"end_device_ids":{"device_id":"***","application_ids":{"application_id":...}
{"result":{"end_device_ids":{"device_id":"***","application_ids":{"application_id":...}

when the correct is:

     [{"result":{"end_device_ids":{"device_id":"****","application_ids":{"application_id":...},
    {"result":{"end_device_ids":{"device_id":"***","application_ids":{"application_id":...},
    {"result":{"end_device_ids":{"device_id":"****","application_ids":{"application_id":...},
    {"result":{"end_device_ids":{"device_id":"****","application_ids":{"application_id":...},
    {"result":{"end_device_ids":{"device_id":"****","application_ids":{"application_id":...}]

[descartes (Nick McCloud)]

It’s because it’s an event stream and each line is an event.

There’s this dude on the forum that knows about these things, if you do a search this comes up:

Please help with SyntaxError: JSON.parse in storage integration

@neoaggelos, to actually answer the question for @Petr: Once you get the response, in Javascript you need: response = '{"data": [' + response.replace(/\n\n/gm, ',').slice(0, -1) + ']}'; If you do Python for an application or service it is: theJSON = "{\"data\": [" + r.text.replace("\n\n", ",")[:-1] + "]}"; The docs are good if you are really old (like me) & can see “The Matrix”. I get the whole Event Stream idea but as it does not have any CORS options in the browser it’s a b1tch to use and…

[dieguitto]

@descartes

It’s because it’s an event stream and each line is an event.

There’s this dude on the forum that knows about these things, if you do a search this comes up:

Ok, thanks for the explanation. The point is that the response will not always be treated with javascript as indicated in the post. I still think it is wrong, and correcting the answer, you can continue dealing with javascript as indicated in the post or not, as is my case, since I consume the api directly by a datasource that expects a JSON.

[descartes (Nick McCloud)]

The API is described thus: “Returns a stream of application messages that have been stored in the database”. The documented response for that API is event-stream so it is not wrong, it is as designed. If you want application/json you need to use another API, albeit one that doesn’t currently exist.

There is nothing about what TTI have done or I wrote that says it will always be processed by JS - indeed I gave a solution for Python as well, I can provide one for PHP, Perl, Java, Go, C#, C/C++ as required - as a way to use the API call for a static download. This works well for my code base and, equally pertinently, I also use the event-stream.

Data Storage is a bonus for us, it’s not included in the Open Source edition but has been enabled for our community edition, it puts an additional load on the servers, doesn’t hold all the meta data that can be useful and is designed as a backup to other integrations, not a primary data source.

I’d recommend using a vanilla WebHook that receives the data, parses it and puts it in to a database, which you can then create your own API as you require. I have a generic version acting as a global backup running on some webspace where I pay £6 a year for the domain name and get 10Gb of space plus 1Gb of database - that’s a huge amount of uplinks.

We get TTN for free, the least we can do is grab our data and look after it ourselves.

[dieguitto]

@descartes

albeit one that doesn’t currently exist.

Thank you very much for your detailed reply. Seeing that it is not an error and that the API response is event-stream, I will look for alternatives as you say to treat the response.

I totally agree that it is a fantastic tool for the community and that the minimum is to save our data ourselves.

[descartes (Nick McCloud)]

@dieguitto

I will look for alternatives as you say to treat the response.

Well now I’ve waved the flag for TTI, I would say that there isn’t an alternative and frankly, I’m not sure why the Data Storage ended up with event-stream except I suspect it works for the CLI & cURL. But it is early days for deployments.

What is the API / toolset you have that makes the request without allowing you to separate out the request and the first parsing?

I think in the medium term we may be able to petition for a DS API request mode that returns normal JSON excepting at present TTI are asking us to respect the rate limit header - and the DS is a prime target for someone to try to create a web app that constantly pulls data from the API without any regard to server load. Adding any form of local caching would help, but overall, running our own database is best.

[dieguitto]

Hello, thanks for answering. I am integrating it into Grafana. In the end I have done it with Telegraf to register it in InfluxDB and thus consume it in Grafana.

[Mat_Lynx]

Hello descartes
Would it be possible to get a solution for HPH please?
I would like to json_decode the data, but I did understand that it is an event stream and not a valid json file. As I am not able to convert the solution you gave in javascript and python to PHP, I would be very happy if you could provide one.

Thanks a lot
br Mat

P.S.: maybe I found the solution in the meantime…
Is this correct? (PHP)
$json = “[” . substr(str_replace("\n\n",",",$EventStream), 0, -1) . “]”;

[descartes (Nick McCloud)]

@Mat_Lynx

$json = “[” . substr(str_replace(“\n\n”,“,”,$EventStream), 0, -1) . “]”;

Looks right - the acid test being you trying it!

[Mat_Lynx]

Thanks for your reply! I really appreciate your help. The code is working fine so far (also an online jason format validator gave “green light”)