Hi everybody,
I wounder if anybody else is experiencing delays in the storage integration?
I fetch data from storage integration via a bash-script in regular intervals and store it in a txt-file. Since approximately one week or so, data seems to be coming in in batches instead of being countinously added. Is there a specific reason for this behavior? Am I doing somthing wrong here?
I noted chunks of data comming in at arround midnight (Berlin Timezone) and the earlier evening.
Firstly an observation wrt SI’s purpose and roll… as I understand it SI is intended as a short term back-up source of data in the event that your own real-time capture processes fail or fall over; e.g. if you discover a missed message (perhaps based on fcount sequence or skipped timestamp) and need to pull a copy from this short term back up. What process do you have in place (Webhook, MQTT feed, or what ever) to capture and store the principal feed in your own storage instance?
As such SI is intended for occasional vs regular access. If all users and all devices were taking regular SI feeds I believe we would quickly saturate and exhaust this valuable community resource and loads would be unsustainable (much as with the internal payload decoder function this is a convenience and not a guaranteed service and is load dependent - if I’m wrong in this description no doubt a fellow mod, contributor or member of TTi core will step in and correct! )
That said
How regular? 1/hr, 1/day…1/per when needed?
I would not be suprised to hear of a change in behaviour if your access is being rate limited or as a general change to avoid:
Yes, you are using it as a primary data store - it’s only there as a backup if your primary integration - like HTTP - has any down time at your end and you need to fill in some gaps.
DS is always delayed and committed to the database as & when, says so in the docs. And if everyone is doing what you do, it will increase delays on the free to use community system. There is a limit to the number of servers that TTI can fund for free.
You also may be rate limited.
How many devices are you doing this with?
Why are you fetching 100 - why not fetch since the last uplink time - if there’s only 5 that’s a lot less load.
Despite not being a softie, you are completely on the button here. I’m surprised we even get DS in TTN and having heard of some spectacular loads due to DS abuse, wouldn’t oppose its removal.
Its a valuable resource for occational use and server load/storage burden was reduced somewhat when nominal storage time reduced from days to hours a while back…but even so we are dependent of users like @nico11 behaving reasonably, reading the supporting docs and not abusing.
Anyone treating as a (near) live stream and (primary) feed should indeed be ‘policed’ and corrected as needed… can’t be sure but think that in >6 years on TTN my devices and demoes have used it maybe 6 or 7 times… did a fantastic job when needed
Sorry for the long absence,
I got the point. I’ll use MQTT instead querying data from the Si. Sorry for implementing this in the wrong way. I never thought this would do any harm. I think I’ll switch then to a backup use of SI only.
Thanks for the hints.
I hope you don’t mind, that not everybody is able to consider all these effects instantaneously. So it was not obvious (at least to me) that the SI is not intended as a regular interface for retrieving data.
)