Connection with the console – The Things Network V3 – not stable

Seems you figured out how to do that 13th July as that’s when my data backup server started saving your uplinks but it appears that stopped at yesterday at f_cnt 4862.

The whole point of me setting it up & providing you access for free was so we could audit this independently.

I’m not going to get in to this as I’ve not got the information I wanted to carry on with this. If you turn it back on and then find an instance, feel free to post the details again.

+1 for using the Hide details but -1 for not using the </> for code/logs.

Thank you again for your support.

-1 again for me as I stopped your bakup saving my uplinks (I assume I did that by setting up my own capture)
I restored the webhook pointing again to http://datacache.co.uk/pietrodelamancha/UplinkToTab.php.

I will follow up further for missing packages, but when you only get the same information as I saw coming in in the tab files, I do not see what we might learn extra.

You can have several, so you can have one too.

Proof.

Last 24 hours : missing 17 packets (out of 288 as my sending frequency is every 5 minutes = 6%);
In the log of my LPS8 gateway : all of them were found as uploaded, but for 4 of them I did not find a PUSH_ACK; these 4 packets were also missing in the TTS gateway console - so I assumed lost ‘on the internet’ (4 out of 288 = 1,4%)
Besides that a few out of these 17 packets appeared twice in the TTS gateway console (resends ?)
None of these 17 packets can be found in the TAB files created by my webhook integration (your php script).
Hereby a few of the (17) missing FCnt : 5359, 5380, 5514, 5634
I restored your webhook link last Satuday. If I did well, you probably can check if these FCnt are indeed missing.
Desperate, I am looking to adapt my application to take into account the loss of packets, but 6% is too high in my opinion.
Thanks again for your support and patience

I assume your LPS8 is set to use Semtech legacy packet forwarder? UDP based… an inherently lossy and somewhat insecure Internet Protocol so depending on back haul connectivity and routing to TTN back end some loss to be expected… but 6% does seem a bit high…

Depending on your actual payload size and content and given you don’t seem to loose sequential data you might consider sending a moving window with not just current data but possibly last 2 or even 3 readings… which you then compare and extract at application level. Careful use of bytes (search working with bytes) may even allow some level of data compression. The nature of LoRaWAN is on air time doesn’t extend in direct relationship to payload length but rather follows a staircase effect… whose step tread length varies with SF. So it may be you can adapt without actually (significantly) impacting your on air time whilst then adding resilience to your data stream…

I can confirm that they are missing from my data cache.

The documentation from TTI says 10% is to be expected: https://www.thethingsindustries.com/docs/devices/concepts/best-practices/

Picking out a quick to get to dataset that’s running within the confines of my Bat Cave with a v2 & a v3 gateway on TTN/TTS, I’ve got 5% loss but I don’t have the stats from the gateways to correlate with nor does it have a complete download of Data Storage to cross reference.

I do have a gateway packet forwarder modification that would make it easier to gather stats. And I have a TTS OS instance that I could run a gateway & device against to see what the results are. And I could start gathering Data Storage but only on TTS.

Transmitting windowed data sets as @Jeff-UK would be a simple solution. Storing uplinks on device and expiring them after a few hours (giving time for the server to request resends) is another.

But fundamentally, given your packet loss is less than TTI’s expectation and similar to mine I’m not sure there is much to be concerned about.

What is the data you are sending and why do you aspire to 100%?

Well, I am monitoring an open/close situation with a distance sensor. I have put my interval on 5 minutes, as more frequent measuring would have too much impact on my ‘fair use policy’. Is 5 minutes really important ? Maybe not, I probably can adapt my application. Having more than 30 years of experience in administrative computing, physical computing was/is rather new to me and maybe my expectations were too high. But reading about Iot and its applications, I am still wondering how you manage it all with an accepted loss of 10 %. When one uses a water sensor and loses just that packet that would tell me that there is water in the basement, one need a different approach than an application relying on one alarm message. Thanks again for your support. As far as I understand, my configuration should be fine and a different approach is needed.

Because we don’t expect the use of the ISM unlicensed band to be for our own exclusive use so we expect a level of packet loss due to anything from a dodgy microwave being used for 20 minutes at lunch time or a big bag of water with meat wrapper having a fag whilst leaning up against the sensor housing five times a day. It’s transmitted in the blind, anything else could be in the airwaves at the time and there’s no handshaking or acknowledgement.

If the door is within 2 or 300m, then the NRF24L01 is pretty good for a ‘reliable’ transmission. If you want something shockingly bad, try ASK 433MHz! If you have power, maybe WiFi on a narrow beam antenna. Or a line of sight laser. Or maroon flares.

Before I type the following, if you really really need to know if a door is open, closed or in the process thereof, LoRaWAN is not a good fit but can work.

For the more important uplinks, my devices use a smaller payload, more frequently whilst awaiting a downlink that is initiated by my application server to acknowledge that “the system” is on the case.

For the rarest of circumstances where only LoRaWAN is an appropriate radio system, ie lots of very small battery driven sensors over a distributed area that are mostly ticking over phoning home every few hours until something very bad happens that at least two or three sensors will be triggered on, then they use a confirmed uplink and keep going until they get that confirmation from the gateway, then revert to the a pattern of uplinks with one in X being confirmed, until it gets the back-end acknowledgement.

However, the backend database usually predicts the forthcoming moment of doom based on readings thus far. It can also spot trends and send a downlink to increase the uplink frequency so that data arrives more frequently, thus ironing out some of the dropped packets.

The only time I’ve seen a device go in to full Blue Light Defcon 1 mode was a water trough when on a hot day a pile of horses decided to drink it dry in the morning. They then went on to the next trough, but the device doesn’t have any knowledge of its peers, so whilst no horses went thirsty, it just knew it had gone from enough to empty. As I said, usually the backend system can see the trend and give a prediction of how long before refill time. At some point I may link in weather forecasts to see if that helps plus some mandatory Machine Learning once I have time to play.

We (the boss & I) came to all of this via security systems - security, fire, access control, CCTV etc - so I have a pretty good idea of ways of monitoring a door remotely and it’s all about levels of certitude. I can bypass most security systems or sensors but for 99% of situations, a reed sensor + 120dB siren will do the job, but even then, a can of pepper spray will always persuade you to turn off the alarm for me. So you need to fit the monitoring system to what you are protecting.

I guess the TL;DR version is, send status “all good, I haven’t been stolen, battery level is X” reports every 60 minutes and then if the sensor is triggered, two byte packets on a different port in quick order - for EU on SF7 that’s one every 5 seconds on the 1% duty cycle (bit over the top) or one every two minutes on FUP but that’s spread over an hour so you could do once every 15 seconds. But reality is you just need to send with confirmed and then a normal uplink to get the backend acknowledgement about 10 seconds later and that’s it, alerted.

Now your next problem is, is your phone in service for that all important SMS? For a small fee I can arrange a satellite pager and modify it with a big toe fitted taser module to ensure you wake up.