A farmer has a TTOG with ethernet backhaul. All devices are enabled for ADR, 3 days ago I noticed SF steadily increasing on all devices from SF7 to their present state SF12. JoinRequests are also failing and it’s clear that downlinks are not being received. I suspect network latency on the farmers internet service provider. How can we test this to verify poor latency? The gateway is connecting to EU router. Should we do a traceroute to the router? Ping the router?
It’s a quick win, so why not. Also do a speed test - http://speedof.me - which will give an independent ping / latency figure from which ever server it tests from.
As the routers are behind loadbalancers pinging typically doesnt work though a trace route might get you so far…
the farmer doesnt have kids in lock down or recent visitors on his service hammering Netflix et al by any chance?! 
1 Like
Yes, we can’t ping the EU router and traceRoute only gets us so far. I love the netflix theory though 
.
Its not so daft as may seem! One rural deployment I know of started seeing big packet losses 1-2 months back and SF changes for some nodes…turns out as we came out of lockdown a local farmer had started renting out a field he used each year for campers and caravaners once he had cut his silage/winter feed with a jury rigged set of WiFi networks to service the campers needs…all the extra traffic was then congesting the local rural exchange link and kirb side green box/dslam…adding a seperate Mobile Wifi Dongle with 4G/LTE backhaul (right on the limit of decent reception mind) as failover for the 2 affected GW’s solved the problem 
I’ve never worked with a TTOG, but my understanding is it runs a fairly ordinary packet forwarder.
If you can get at the logs, I’d expect there would be an indication of when a downlink packet arrives too late; typically it is actually logged as too early because it is long in advance of the next time the local microsecond counter would reach the commanded value after rolling over.
I’d also look at any messages about the latency and success percentage of push/pull acks, which are directly part of the protocol used to pass downlinks (assuming the TTOG uses UDP; the setup info suggests so but I’m not 100% certain)
While this is certainly a possibility, absent some evidence I think there’s at least as much chance that the cause is something else, as there is that it is this.
I can believe you! My similar experience (albeit for a totally different reason); the undersea West Africa Cable System (WACS) broke in March, severely degrading our access to the WWW. Internet connectivity was there but with a huge latency, also resulting in devices not being able to receive downlinks, failing join requests, ADR forcing SF12 etc etc 
I agree there is something more sinister than just netflix going on. I’m seeing pings of around 200ms to European servers (the farmer is in southern africa) so I’m happy with the internet connection. I have access to the logs but they are beyond my paygrade to interpret, I do see statements like this:
# TX rejected (too early): 7.45% (req:416, rej:31)
# TX rejected (too late): 0.72% (req:416, rej:3)
# TX rejected (collision beacon): 0.00% (req:416, rej:0)
# TX rejected (collision packet): 0.00% (req:416, rej:0)
# TX errors: 0
For lack of ideas I’ve bumped up the Gain a smidge to see if that might help.
That probably does indicate excessive latency; they’re really “too late”
For lack of ideas I’ve bumped up the Gain a smidge to see if that might help.
Sorry, but not a chance