Using TTN trafic monitor to troubleshoot a TTIG-based setup

[codename5281]

@descartes

The console log is very useful. A quick scan of it reveals a number of issues. Can someone transcribe it

Yes sir, just finished it.
I’ll answer to the other messages in the morning tomorrow ! Sleep tight.

[codename5281]

@Jeff-UK

Yes deploy a Canary node

Ok will do. We may even try the Dragino LSE01 soil moisture sensor for a change.

@Jeff-UK

4G ‘should be ok’ for backhaul unless your network is way out on a limb and hopping through many repeaters or long fibre or dsl back hauls to RAN and Cellco central office.

Our locations are pretty remote ; is there a way to check what might be the impact of the backhaul ?

@Jeff-UK

are they LA certified devices?

At least they say so on the last version of the technical data sheet, however i cannot find any trace of them on the LoRa Alliance certified products showcase page. Also, i just noticed on the last TDS the “GEN2” label, we’ll have to check which one Sensoterra provided as this is not specified in our invoice, and also what are the main differences between the two generations.

@descartes

Shame we don’t know what each sensors DevEUI is so we can place it on the map.

We don’t have an explicit info about that, Sensoterra’s using an intermediate in-house serial number in between both in their monitoring interface and raw measurement files.

Not much useful on the probes either, the QR code is S/N as well.

@descartes

Jeff Mckeown was around at the time they launched LoRa. He’s deployed dozens of community gateways around the UK. If anyone was going to know about gateway’s, it’s Jeff. Or Jac Kersing. Or LoRaTracker. All do RF & antennas, me, not so much, I’m more firmware, devices and business intelligence (data processing).

I am not in place of putting @Jeff-UK knowledge and experience into question, i was just pointing out these contradictions

@descartes

I wonder why the Sensoterra back end is not going berserk with alerts about all the re-joins - I assume you don’t get any emails saying your deployed sensors aren’t happy??

Nope, we hear nothing from them.

@descartes

Is there anyway of getting hold of one - anyone out in Morocco that can pull one out the ground and send it north? Positioning the sensor & a TTIG for experimentation would be good.

Well, it’s gotten a bit difficult now…

@descartes

Do you have pictures of the TTIG in it’s weather proof box - how the antenna was extended out of it would be useful knowledge to help with elimination of any other spurious objections from Sensoterra.

The antenna extension was very similar to what has been done by Tnkerman and Disk91, i.e. disconnecting the PCB antenna and connecting a 868 MHz-tuned one via a 5cm UFL to SMA cable.
.

@Jeff-UK

when txing a downlink (or repeated join accepts for that matter if node doesnt catch and action forcing repeats) the GW is unable to listen for uplinks - it goes deaf! …and that might result in gaps in Rx data…just saying

…but this makes sense mainly in the case where all probes send data at the exact same time, right ?

@descartes

Sensoterra have closed firmware that the management do not appear to be able to describe the general scheme of operation

Well, the discussions have been mostly handled by the commercial director, but i can try to be in touch and send specific questions to the proper tech guys.

[descartes (Nick McCloud)]

@codename5281

@Jeff-UK

4G ‘should be ok’ for backhaul unless your network is way out on a limb and hopping through many repeaters or long fibre or dsl back hauls to RAN and Cellco central office.

Our locations are pretty remote ; is there a way to check what might be the impact of the backhaul ?

That’s what the canary will help do for us - if it uplinks every 10 minutes you get to see if there are any gaps in the connectivity.

In this situation, with a 2 byte payload and bypassing ADR to keep it at DR5 and go for 2.5 minutes to get the best resolution possible. It can be remote commanded to return to a less pro-active rate.

The log (thanks for that) shows up some points:

• The sensor is using a Multitech DevEUI - which is good as it means we can be reasonably sure the actual LoRaWAN stack isn’t borked, just something about the firmware.

• I think DevEUI “00 80 00 00 04 01 3A 0F” and “00 80 00 00 04 00 9F B8” may be a little too close to the gateway as it looks to me that they are overloading the input and the gateway thinks it’s heard a JR on several channels when it was just one.

• DevAddr “26 01 22 11” is clearly getting on with things but there are gaps in it’s uplinks and its count is only on 15.

• It’s pretty clear that an uplink frequently gets a downlink. This may be MAC requests from the network but as uplink 15 gets downlink 6, either the node isn’t doing what’s being asked of it (unlikely) or they do get a response more often than what could be considered ‘normal’

• The pattern of uplinks isn’t very discernible so it’s not sure if the “26 01 22 11” is working as expected or if something is happening on site that means connectivity to the backend isn’t available (bring in the canary).

I suspect the only way to match DevEUI to a serial number to a location is by the usual “turn them all off, turn one back on at a time” method. I realise this can’t be done at present.

Re @Jeff-UK, I know you weren’t querying his credentials, that’s my job - it was more an observation that there are others here that are the subject matter experts on gateways, antennas and RF in general. I’ll leave others to comment on the position of the gateway, but I suspect Sensoterra (hi to you guys if you are reading along) will query it’s deployment. Personally, I’m a great fan of DuckTape but ideally have it on a standalone pole. I’ve not been in a position to try to cook a gateway as I live in the north of England and Westminster hasn’t authorised the use of sunshine up here, but I will add it to the summer projects.

@codename5281

but this makes sense mainly in the case where all probes send data at the exact same time, right

Yes & No - with all those repeated JRs with corresponding JAs we won’t know if there was other traffic going on but it would be a huge co-incidence for such a small number of sensors to overlap. We do get lots of people trying to use downlinks as a matter of course in their firmware (potentially like this sensor) and then the problem escalates. Five sensors with a downlink once every three hours is one thing. But then you get 100 sensors deployed … And as the some developers command JR’s without any backoff (delay), you can end up with a cascading effect. So we tend to mention the whole ‘gateway is deaf’ during a downlink as it comes up often enough - a sensor should be able to cope most of the time without any downlinks. For this sort of sensor, as it’s not tracking nuclear assets or my wallet, I’d leave it to uplink and employ my usual schemes for checking connectivity. If I miss a few uplinks I could extrapolate what the readings might be. If I really really need a data point to fill in and I’d allowed for it in firmware, I could send a request for a resend of one or more readings, or an average for some time ranges or similar.

[codename5281]

@descartes

That’s what the canary will help do for us - if it uplinks every 10 minutes you get to see if there are any gaps in the connectivity.

As the TTIG gateway is not the only device enjoying some wifi backhaul, we have a way to check for big gaps in 4G connectivity with other sensors that ping every 30 minutes - as they successfully do so most of the time despite not having a light payload, we know there is no major trouble in sending the data.

@descartes

I’ll leave others to comment on the position of the gateway, but I suspect Sensoterra (hi to you guys if you are reading along) will query it’s deployment.

This picture purpose was illustrative, as the box is in practice positioned at ~2m as previously stated. Still, yes, there is duck tape (as well as polystyrene foam-based insulation attempt) involved in the baking !

[codename5281]

Hello ! A quick update on this topic. I don’t know if anything has changed on TTN or Sensoterra side, but it’s now been three days since our whole setup is working without any issue, i.e. all probes are reporting as they should, and no join requests/accepts, unly uplinks/downlinks in the console during the last 24 hours.

That’s funny because today was the day where we had planned to setup the RAK gateway on the field

Edit : i wonder if this can somehow be related to an OTA firmware update of the TTIG, but i wasn’t able to probe the TTN noc server to check the firmware version

[descartes (Nick McCloud)]

Good to hear they are working - have Sensoterra communicated a plan for the transition from v2 to v3?

[codename5281]

So far, we didn’t hear from them, no. We’re going to reach them and ask, in the meantime do you know if there has been any major changes on TTN side that could explain this sudden change ?

[descartes (Nick McCloud)]

No, but my recently acquired one is working fine, which is sort of good and sort of frustrating. I am going to park a metal box on top of it to take it off line to see what it does when it can be heard again.