High latency cellular backhaul suspected to cause TOO_EARLY downlink transmission errors

[Jeff-UK]

Notice from your picture that you are using confirmed uplinks - these are really to be avoided if possible - even on a private instance Despite the reasonable RSSI for the one that gets through I note use of SF12 - again to be avoided/moved from under ADR if possible…also join at bottom at SF12? What are nodes? What firmware/how configured? For the recieved uplink FCount =? How all nodes configured =OTAA or ABP - from the Join assume that one is OTAA… is the one with Dev ADDR 26086757 one of yours or a ‘foreign’ device? (Are you seeing that device in your Application/Device Console?)

Actually jus notice its 2 pics…

Clearly a decoder problem also?

[Johan_Scheepers]

Stick the SIM card to a cellular router and test with a PC

[LasseRegin]

@Jeff-UK

confirmed uplinks - these are really to be avoided if possible - even on a private instance

Wasn’t aware of this. Thanks for pointing out!

@Jeff-UK

Despite the reasonable RSSI for the one that gets through I note use of SF12 - again to be avoided/moved from under ADR if possible

The nodes are our own with custom firmware, where we’ve implemented ADR so the SF is increased as long as the node fails sending uplinks, hence it should decrease to SF7 when uplinks flow successfully again

@Jeff-UK

How all nodes configured =OTAA or ABP - from the Join assume that one is OTAA

All are configured for OTAA.

@Jeff-UK

is the one with Dev ADDR 26086757 one of yours or a ‘foreign’ device?

They are all ours. They are listed in Application.

@Jeff-UK

Clearly a decoder problem also?

That’s correct. We have problems with the JS decoder in TTN, so we are manually decoding the packets in our Cloud Solution

[LasseRegin]

@Johan_Scheepers

Stick the SIM card to a cellular router and test with a PC

Thank you for the suggestion, however it’s not possible for us at the moment since we are remote debugging this.

[Jeff-UK]

@LasseRegin

The nodes are our own with custom firmware, where we’ve implemented ADR so the SF is increased as long as the node fails sending uplinks, hence it should decrease to SF7 when uplinks flow successfully again

If they are 5-50m away with reasonable (say <50 → <115) RSSI then you should see joins OK at SF7/8/9… SF12 suggests you have configuration/behavioural problems with these nodes!

[Johan_Scheepers]

@LasseRegin

remote

Where in the world is the gateway?

And unfortunately you can’t fix all problems remotely.

[LasseRegin]

@Jeff-UK

Potentially yes in that basic SMTC PF is UDP based with no QoS/certainty of delivery - packets are just thrown out on a wing & a prayer and whilst most will get there…!, where BasicStation is more secure and being TCP/IP based will have opportunity for replay/redelivery/easier network snooping/diagnostics etc.

Switching to Basics Station mode seems to have fixed the problem. Thank you all very much for your help!

[descartes (Nick McCloud)]

@kersing

There are no such things as attempted uplinks. Either an uplink arrives at the gateway or it doesn’t. If it does and the CRC is correct it will be forwarded to the back-end. As the status messages arrive, the uplinks should arrive as well.

Pretty much everything after this wisdom looks like advanced guessing or suggestions that need time to be answered before more suggestions are made - I’d highly recommend slowing down responses because if a change is made that loses remote access to the gateway, someone has to go on a fly-drive holiday.

As per protocol, we know the OP has mastered the art of the drip feed information - @LasseRegin, please be more expansive with answers - ten minutes giving us everything will probably save the volunteers subbing for your €200 saving on a support contract a whole heap of time. Also, most of everything typed in response is essential detail - ping utilities were provided, you dove straight to the command line. I’m not sure why Johan wants you to move an EU installation to half way round the planet either

Here’s what we seem to know:

• The gateway is sending Acks with some variability in timing & consistency.
• Some of the uplinks arrive on the console.
• The ADR mechanism has been altered.
• Devices that are on top of the gateway (all of them at 5-50m) are SHOUTING which won’t help.
• We have yet to know how many devices there are.
• And how many are heard by the gateway.

The refinement of the devices can come after resolving why the basics of uplinks not being relayed is solved.

[descartes (Nick McCloud)]

@LasseRegin

Switching to Basics Station mode seems to have fixed the problem.

Being a community, we share solutions here - what detailed differences did you see on the gateway log, the gateway console and the app/device console that lead you to believe this is resolved?

[Jeff-UK]

Has it also solved the SF12 join problem or is that still showing (and needs fixing) - if so likely your device firmware needs refinement.

@descartes at -78/79 RSSI atleast one device is in the sweatspot for placement/signal strength … allowing for coding gain likely equivalent to say -85-95 at lower SF’s… though even with ‘thin wood walls’ inbetween would be concered at 5m seperation if ant’s planar aligned!

[Johan_Scheepers]

@descartes

EU installation to half way round the planet eithe

The latency for me to

Ireland eu-west-1 635 ms
Sydney ap-southeast-2 560 ms
N. California us-west-1 647 ms

I have a EU installation.

Please suggest the cluster I should use?

It all depends where you are siting in the world, some times latency is not that related to distance.

[descartes (Nick McCloud)]

@Johan_Scheepers

It all depends where you are siting in the world, some times latency is not that related to distance.

The OP said it was an installation in the EU …

[LasseRegin]

First of all the end device uplinks were being successfully received immediately after the change

In the gateway log the lora_pkt_fwd log was obviously replaced with a basicstation log. Without fully understanding each log statement it definitely didn’t print similar warnings like before related to “out of sync ACK packet” which I assume is due to the Basics Station mode working very differently from the Packet Forwarder mode.

Here’s the latest snippet from the Gateway log.

Tue Oct  4 07:11:08 2022 user.info basicstation[20050]: [S2E:INFO] Beaconing suspended for 10s - insufficient GPS data: time
Tue Oct  4 07:11:08 2022 user.debug basicstation[20050]: [SYN:VERB] Time sync rejected: quality=3055 threshold=159
Tue Oct  4 07:11:18 2022 user.info basicstation[20050]: [S2E:INFO] Beaconing suspended for 10s - insufficient GPS data: time
Tue Oct  4 07:11:19 2022 kern.info quectel-CM[2363]: requestRegistrationState2 MCC: 288, MNC: 1, PS: Attached, DataCap: UMTS
Tue Oct  4 07:11:19 2022 user.info mwan3track: Lost 3 ping(s) on interface wwan (wwan0)
Tue Oct  4 07:11:22 2022 kern.info quectel-CM[2363]: requestRegistrationState2 MCC: 288, MNC: 1, PS: Attached, DataCap: UMTS
Tue Oct  4 07:11:25 2022 user.info basicstation[20050]: [SYN:INFO] MCU/SX130X drift stats: min: +0.5ppm  q50: -2.4ppm  q80: -6.2ppm  max: +7.1ppm - threshold q90: +6.2ppm
Tue Oct  4 07:11:25 2022 user.info basicstation[20050]: [SYN:INFO] Mean MCU drift vs SX130X#0: 1.3ppm
Tue Oct  4 07:11:27 2022 user.debug basicstation[20050]: [SYN:VERB] Time sync rejected: quality=160 threshold=159
Tue Oct  4 07:11:28 2022 user.info basicstation[20050]: [S2E:INFO] Beaconing suspended for 10s - insufficient GPS data: time
Tue Oct  4 07:11:36 2022 kern.info quectel-CM[2363]: requestRegistrationState2 MCC: 288, MNC: 1, PS: Attached, DataCap: UMTS
Tue Oct  4 07:11:36 2022 user.info mwan3track: Lost 3 ping(s) on interface wwan (wwan0)
Tue Oct  4 07:11:38 2022 user.info basicstation[20050]: [S2E:INFO] Beaconing suspended for 10s - insufficient GPS data: time
Tue Oct  4 07:11:39 2022 kern.info quectel-CM[2363]: requestRegistrationState2 MCC: 288, MNC: 1, PS: Attached, DataCap: UMTS
Tue Oct  4 07:11:43 2022 user.info basicstation[20050]: [SYN:INFO] Time sync qualities: min=87 q90=160 max=3055 (previous q90=159)
Tue Oct  4 07:11:46 2022 user.debug basicstation[20050]: [SYS:VERB] Sys state 101
Tue Oct  4 07:11:46 2022 user.info basicstation[20050]: [RAL:INFO] RX mod=LORA f=867700000 bw=4214640 sz=42 dr=8 80A82508268004000111E3785DC63F694953DFD36E3C213ED5630CB4AA473CB1F0C1D3EDE2D8B82AE3D3
Tue Oct  4 07:11:48 2022 user.info basicstation[20050]: [S2E:INFO] Beaconing suspended for 10s - insufficient GPS data: time
Tue Oct  4 07:11:52 2022 kern.info quectel-CM[2363]: requestRegistrationState2 MCC: 288, MNC: 1, PS: Attached, DataCap: UMTS
Tue Oct  4 07:11:53 2022 user.info mwan3track: Lost 3 ping(s) on interface wwan (wwan0)
Tue Oct  4 07:11:53 2022 user.debug basicstation[20050]: [S2E:VERB] ::1 diid=18297 [ant#0] - class A has no more alternate TX time
Tue Oct  4 07:11:56 2022 kern.info quectel-CM[2363]: requestRegistrationState2 MCC: 288, MNC: 1, PS: Attached, DataCap: UMTS
Tue Oct  4 07:11:58 2022 user.info basicstation[20050]: [S2E:INFO] Beaconing suspended for 10s - insufficient GPS data: time
Tue Oct  4 07:12:02 2022 user.debug basicstation[20050]: [SYS:VERB] Sys state 101
Tue Oct  4 07:12:02 2022 user.info basicstation[20050]: [RAL:INFO] RX mod=LORA f=867700000 bw=4214640 sz=42 dr=8 80A82508268004000111E3785DC63F694953DFD36E3C213ED5630CB4AA473CB1F0C1D3EDE2D8B82AE3D3
Tue Oct  4 07:12:04 2022 user.info basicstation[20050]: [SYN:INFO] MCU/SX130X drift stats: min: -0.2ppm  q50: -3.8ppm  q80: +7.1ppm  max: +9.5ppm - threshold q90: +8.9ppm
Tue Oct  4 07:12:04 2022 user.info basicstation[20050]: [SYN:INFO] Mean MCU drift vs SX130X#0: 2.5ppm
Tue Oct  4 07:12:08 2022 user.info basicstation[20050]: [S2E:INFO] Beaconing suspended for 10s - insufficient GPS data: time
Tue Oct  4 07:12:10 2022 kern.info quectel-CM[2363]: requestRegistrationState2 MCC: 288, MNC: 1, PS: Attached, DataCap: UMTS
Tue Oct  4 07:12:10 2022 authpriv.info dropbear[12592]: Child connection from 10.8.0.56:52355
Tue Oct  4 07:12:10 2022 user.info mwan3track: Lost 3 ping(s) on interface wwan (wwan0)
Tue Oct  4 07:12:16 2022 kern.info quectel-CM[2363]: requestRegistrationState2 MCC: 288, MNC: 1, PS: Attached, DataCap: UMTS
Tue Oct  4 07:12:17 2022 user.debug basicstation[20050]: [SYS:VERB] Sys state 101
Tue Oct  4 07:12:17 2022 user.info basicstation[20050]: [RAL:INFO] RX mod=LORA f=867300000 bw=4214640 sz=42 dr=8 80A82508268004000111E3785DC63F694953DFD36E3C213ED5630CB4AA473CB1F0C1D3EDE2D8B82AE3D3
Tue Oct  4 07:12:18 2022 user.info basicstation[20050]: [S2E:INFO] Beaconing suspended for 10s - insufficient GPS data: time
Tue Oct  4 07:12:28 2022 user.info basicstation[20050]: [S2E:INFO] Beaconing suspended for 10s - insufficient GPS data: time
Tue Oct  4 07:12:29 2022 kern.info quectel-CM[2363]: requestRegistrationState2 MCC: 288, MNC: 1, PS: Attached, DataCap: UMT

Please let me know if I am missing additional information. And once again thank you all for helping out, it’s very much appreciated!

[descartes (Nick McCloud)]

@LasseRegin

which I assume is due to the Basics Station mode working very differently from the Packet Forwarder mode.

Not so much a “mode” as like the difference between BSD & Linux - both do the same job but in different ways.

You may want to consider learning what those differences are if your business relies on the use of gateways as part of its product offering as there is more than one way a LoRaWAN installation can bite back.

[descartes (Nick McCloud)]

High latency cellular backhaul is unrelated to TOO_EARLY downlink.

October’s advanced guessing competition is now closed. We all got the prize.

[LasseRegin]

@descartes

High latency cellular backhaul is unrelated to TOO_EARLY downlink.

Why is it unrelated? Most threads on this forum I’ve read related to TOO_EARLY mention high latency backhaul as a probable cause, and the only thing changed from the setting of testing (where everything was fine) to the setting on-site is that the cellular connection is worse.

[descartes (Nick McCloud)]

Because if a downlink arrives late, it can’t generate a too early error, only a too late error.

See the other post.