Are there best practises to minimise missing data?
I understand that one should be aware of about 10% package loss and that sending uplinks is much better than downlinks.
Ultimate a resend downlink command is needed, but I want to avoid those.
I can send data more than one time.
The easy way is to send the data always twice, aka add the (then new) data from the previous datatransmission. For example send every 4 hours the data from the previous 8 hours.
Another way is to make specific “resend” transmissions, for example every 12 hours for data that’s already 24hours old. Aka avoid longer “downtime”, maybe because of bat weather.
First you have to decide what your goals are, and what you can do about failure to meet them.
For example, are you hoping to capture a complete historical record? If so, locally retaining data for a few days then then once or twice a day sending a downlink containing a bitmap of which packets from the preceding few days are still missing could allow resending those. Since it’s tricky to know exactly which packet a downlink refers to (due to infrastructure side queueing) you probably want to make the coverage group implement modulo 8 or something. Eg you could send a 40 bit (5 byte) downlink covering the most recent 40 or so opportunities. 5 bytes is a nice payload size because it likely requires just one more 4/5 coding group than the bare network overhead does (the network overhead is typically a byte short of using an integral number of 4-byte groups).
If you’re concerned about gaps in current data, then TTN (and really the design of LoRaWan itself when applied to a non-trivial number of nodes) isn’t going to let you send downlinks often enough to ack or resend-request those, so you’re just going to have to go with the shortest, most compact packet possible and send it at a moderate interval (consistent with the usage policy) and hope you have current data “often enough”.
First answer: it depends, as cslorabox already said. In other words, you have to be clear about certain parameters/requirements, such as
are telegrams sent on a regular basis, or on demand, or both?
what is the probability of a package loss? Keep in mind that there may be different scenarios, ranging from constant background noise or poor radio conditions to a new transmitter that keeps on interfering, even in a distinct time scheme.
how important is it to have every telegram delivered? Is there a certain amount of packet loss still tolerable?
…and possibly many more.
With the additional constraint of the specific LoRaWAN downlink difficulties, it is clear that a TCP-like protocol cannot be realized.
The bitmap-based approach with a sliding window seems a good start. Simulation may also be helpful regarding the parameters I mentioned. Interesting topic for sure, I am interested in any hints that might come up here .
I think that when using LoRaWAN, your application should simply be prepared to have some missing data, and missing up to 10% of packets is quite realistic in my experience.
If you send a kind of absolute measurement, like a counter, it is not a super big deal if you miss a packet. Each new packet provides a full status.
For example, we have a project with motion sensor nodes (TBMS100). The motion sensor sends a packet immediately upon detecting motion. Sometimes this packet is lost. It has a packet structure that tells you the number of minutes since the last movement, and it contains a counter for the absolute number of times a movement was detected. So even if you miss a packet, you can still see that the counter was increased by some amount since the last packet. And you can reconstruct the time of the last motion event using the minutes-since-motion field.
Another example, where data loss is more-or-less tolerable: I’m using LoRaWAN to send airborne particulate matter measurements. The node sends data every 5 minutes (depending on spreading factor). If I miss a packet, the node will miss the “heartbeat” of the map on maps.sensor.community and will not appear on their map. A pity but not a super big deal.
The dutch national institute for public health and environment uses that data too, but calculates hourly averages, so missing 10% of packets means only a slightly inferior hourly average, we still get enough data.
I can miss individual packets, it’s far more important to get a -almost- complete historical record. I also need the data more than 8 hours later.
Further I know that I can have a very small relative encoding for following measurements.
I probably always send data from the current period and the previous one (thus sending data 2 times) as this doesn’t need that much bytes and use a bit map system to get still missing data.
.