ADR problems - node of the same type have different ADR behaviour

[gigi130358 (Gianluigi)]

Many thanks Arjan.
Unfortunaltely the console output isn’t no more available because after about half an hour I always get this error (“Something went wrong refreshing the token: oauth2: cannot fetch token: 400 Bad Request Response: {“error”:“invalid_grant”,“error_description”:“Refresh token not found”}”) and the page disapppear.
Do you know if there is any way to avoid this timeout viewing the console output ?

Anyway logging the download trafic with mosquitto, the 09:29:46 msg of the second node report this contents (timestamp hour it’s coming from an AWS clocks with an our before)
Feb 17 08:29:46 senlab-t/devices/1060d9/events/down/sent {“payload”:“YEcmASaAAgACoO7QyXTRd8WODG+faDszaKBELcM=”,“message”:{“app_id”:“senlab-t”,“dev_id”:"1060d ",“port”:2,“payload_raw”:“AQcBAQCWAHECAJYAJgkADw==”},“gateway_id”:“eui-1dee0eb900028599”,“config”:{“modulation”:“LORA”,“data_rate”:“SF9BW125”,“airtime”:226304000,“counter”:2,“frequency”:869525000,“power”:27}}
I hope the evaluation of this msg with the lorawan-packet-decode could help to evaluate the contents.

About the download cnt, I try to sent some download msg before the one reported in the screenshot.

Considering the interesting Mario’s post https://mariozwiers.de/2019/01/04/adaptive-data-rate-the-things-uno-rn2483/, do you think that after more 64 up msg, TTN server should send the msg to set the adr ?

Gianluigi

[arjanvanb (Arjan)]

@gigi130358

YEcmASaAAgACoO7QyXTRd8WODG+faDszaKBELcM=

Indeed, that has no ADR MAC commands in FOpts there. However, it does carry application data; is that what you expected? (So, you did not send an empty downlink? That’s fine, of course, just want to be sure.)

@gigi130358

do you think that after more 64 up msg, TTN server should send the msg to set the adr

I’m not sure. Even after the device has sent an ADRAckReq then TTN is not even required to reply immediately, and can again wait for at most 32 uplinks, to piggy-back on a regular downlink. Also, things are different when the node is at SF12, which TTN tries to avoid. For that, it might show “avoid high sf” in the Trace part in TTN Console, and not await a device’s ADRAckReq at all. So while at SF12 you should not have to wait that long. Also, your devices are all the same.

As for TTN Console timeouts: I’ve seen it many times too. MQTT is indeed much more reliable when debugging, but then you don’t have the Trace parts which can be really helpful.

[arjanvanb (Arjan)]

@gigi130358

I’ve also imagined that some nodes could be put in a let’s say “adr blacklist” by the TTN server

As an aside: it seems that this blacklist is maintained by a simple counter in the device’s metadata, ADR.Failed. And it seems this counter is only incremented if a device does send a LinkADRAns MAC command, but in that command does not confirm all settings.

There is code to detect a missing LinkADRAns, but that code does not seem to increment that counter. So, if the node does not send such MAC command at all (including if it never received it to start with), it seems it might not be registered as a failed ADR command. Hence, if the node does not receive the ADR Request from the server, then I don’t think it will be banned…

[gigi130358 (Gianluigi)]

@arjanvanb

Indeed, that has no ADR MAC commands in FOpts there. However, it does carry application data; is that what you expected? (So, you did not send an empty downlink? That’s fine, of course, just want to be sure.)

I confirm that I sent an application command in order to modify the reporting time of the node.

@arjanvanb

As for TTN Console timeouts: I’ve seen it many times too. MQTT is indeed much more reliable when debugging, but then you don’t have the Trace parts which can be really helpful.

I agree, the console could really help in debug, but sometimes (… when I need it ) it diseappears.
In the meantime I will continue to collect the downlink msg with mosquitto and let’s see if something will arrive from TTN backend …

Gianluigi

[arjanvanb (Arjan)]

@gigi130358

“ADR requests are no longer scheduled or sent if the device refused several ADR requests” ?
Does it mean that If a node refused several (how many ?) ADR request, it will banned for ever from TTN ADR managment or for example for some time or till it will be reset ?

Given my previous answer (which I posted while you writing yours) I’d say it’s banned until deleted and re-created. The maximum failures is 3. But again: I think it’s only banned when in the next uplink it does acknowledge it received the ADR request, but then tells TTN it refused one of the settings, being data rate, power and/or channels. In other words: if it could not comply to TTN’s request, in which case repeating it makes no sense. This would better be described as “rejected” rather than “failed”.

Keep the gateway’s Traffic page in TTN Console open to be able to see the Trace part when clicking an uplink or downlink!

[arjanvanb (Arjan)]

@gigi130358

have good snr and are really close to the gateway

Beware:

@kersing

Make sure the node is at least 3 meters away from the gateway or the signal might be too strong resulting in CRC errors.

And:

@kersing

Make sure to keep at least 8 feet between a node and gateway to prevent cross talk caused by strong RF signals. (Which can cause the gateway to receive uplinks on a different channel, resulting in a reply on a channel on which the node does not listen for an answer)

In your case, the latter seems okay, as the downlinks are in RX2, 869.525/SF9, which is different from the uplink channel anyway. The first might apply though, making the ADR downlink not being handled on the device.

Are all nodes using OTAA? Or when using ABP, did you adjust the RX2 settings to match the non-standard TTN settings? (I guess this is fine, as it would apply to both devices, and one works just fine.)

Oh, and all of the above might not apply to you, as we already determined that at least one of the downlinks did not hold any ADR command at all…

[gigi130358 (Gianluigi)]

The gw is about 15 meter from the nodes and I’m only using OTAA.

I’m attaching the trafic log files about the 10609d node both in up and down direction, with some comments inside.
No downlink messagge from yesterday midday, till this morning when I moved the nodes from home to work site. When I arrived at work, one downlink was sent by TTN, but SF is always 12.

1060d9-up-18022019.txt (127.8 KB)
1060d9-down-18022019.txt (1.8 KB)

[gigi130358 (Gianluigi)]

I try something more with another node, 1060bd.
Yesterday this node opeartes with a SF7, then I shielded it for all the night and this morning I remove the shield and I found it with a SF12.
Then TTN start to send downlink msg every time the node send an uplink (every hour).
This is the mosquitto downlink log from yesterday till now:

Feb 17 09:18:27 senlab-t/devices/1060bd/events/down/sent {“payload”:“YOEvASaAFwACYNKhHPXwO+Zs3isTHM1fx1Ffv0A=”,“message”:{“app_id”:“senlab-t”,“dev_id”:“1060bd”,“port”:2,“payload_raw”:“AQcBAQcIBuMCBwgDXwkADw==”},“gateway_id”:“eui-1dee0eb900028599”,“config”:{“modulation”:“LORA”,“data_rate”:“SF7BW125”,“airtime”:66816000,“counter”:23,“frequency”:867700000,“power”:14}}
Feb 18 06:49:52 senlab-t/devices/1060bd/events/down/sent {“payload”:“YOEvASaFGAADVf8AAzz/Bqw=”,“message”:{“app_id”:“senlab-t”,“dev_id”:“1060bd”,“port”:0},“gateway_id”:“eui-1dee03a372f1126a”,“config”:{“modulation”:“LORA”,“data_rate”:“SF9BW125”,“airtime”:164864000,“counter”:24,“frequency”:869525000,“power”:27}}
Feb 18 07:49:39 senlab-t/devices/1060bd/events/down/sent {“payload”:“YOEvASaKGQAFA9KthANV/wADLIV4/Q==”,“message”:{“app_id”:“senlab-t”,“dev_id”:“1060bd”,“port”:0},“gateway_id”:“eui-1dee03a372f1126a”,“config”:{“modulation”:“LORA”,“data_rate”:“SF12BW125”,“airtime”:1482752000,“counter”:25,“frequency”:869525000,“power”:27}}
Feb 18 08:49:42 senlab-t/devices/1060bd/events/down/sent {“payload”:“YOEvASaKGgAFA9KthANV/wADYCLDlQ==”,“message”:{“app_id”:“senlab-t”,“dev_id”:“1060bd”,“port”:0},“gateway_id”:“eui-1dee1a0a6e94d4bf”,“config”:{“modulation”:“LORA”,“data_rate”:“SF12BW125”,“airtime”:1482752000,“counter”:26,“frequency”:869525000,“power”:27}}
Feb 18 09:49:50 senlab-t/devices/1060bd/events/down/sent {“payload”:“YOEvASaKGwAFA9KthANV/wADJbGK9Q==”,“message”:{“app_id”:“senlab-t”,“dev_id”:“1060bd”,“port”:0},“gateway_id”:“eui-1dee1a0a6e94d4bf”,“config”:{“modulation”:“LORA”,“data_rate”:“SF12BW125”,“airtime”:1482752000,“counter”:27,“frequency”:869525000,“power”:27}}
Feb 18 10:49:41 senlab-t/devices/1060bd/events/down/sent {“payload”:“YOEvASaKHAAFA9KthANV/wADHOnA3Q==”,“message”:{“app_id”:“senlab-t”,“dev_id”:“1060bd”,“port”:0},“gateway_id”:“eui-1dee1a0a6e94d4bf”,“config”:{“modulation”:“LORA”,“data_rate”:“SF12BW125”,“airtime”:1482752000,“counter”:28,“frequency”:869525000,“power”:27}}
Feb 18 11:50:04 senlab-t/devices/1060bd/events/down/sent {“payload”:“YOEvASaKHQAFA9KthANV/wADtk6xvQ==”,“message”:{“app_id”:“senlab-t”,“dev_id”:“1060bd”,“port”:0},“gateway_id”:“eui-1dee1a0a6e94d4bf”,“config”:{“modulation”:“LORA”,“data_rate”:“SF12BW125”,“airtime”:1482752000,“counter”:29,“frequency”:869525000,“power”:27}}
Feb 18 12:49:36 senlab-t/devices/1060bd/events/down/sent {“payload”:“YOEvASaKHgAFA9KthANV/wADcpP3+A==”,“message”:{“app_id”:“senlab-t”,“dev_id”:“1060bd”,“port”:0},“gateway_id”:“eui-1dee1a0a6e94d4bf”,“config”:{“modulation”:“LORA”,“data_rate”:“SF12BW125”,“airtime”:1482752000,“counter”:30,“frequency”:869525000,“power”:27}}
Feb 18 13:49:56 senlab-t/devices/1060bd/events/down/sent {“payload”:“YOEvASaNHwACFwEFA9KthANV/wADQ9uFew==”,“message”:{“app_id”:“senlab-t”,“dev_id”:“1060bd”,“port”:0},“gateway_id”:“eui-1dee1a0a6e94d4bf”,“config”:{“modulation”:“LORA”,“data_rate”:“SF12BW125”,“airtime”:1482752000,“counter”:31,“frequency”:869525000,“power”:27}}

I’ve also catched the last one downlink (13:49:56) with the TTN console, that shows the ADR commands:

I don’t know if the link-adr commands are correct, from the mosquitto log there is a “config” payload that I don’t know if refers to the current condition or represent a configuration command.
In case the adr command are really trying to lower the SF, it could be that the problem is the gateway or the node.

Gianluigi

[arjanvanb (Arjan)]

Given the frequency 869.525, the downlinks are all in RX2, except for the very first one, which is in RX1 (and is the Join Accept). Of those RX2 downlinks, only the first is sent at the TTN-specific SF9 (as expected), all others use the LoRaWAN default SF12 (as a workaround).

Now, an example has proved that, when using RX2, TTN surely is smart enough to repeat ADR on SF12 if nothing changed when the node ignored the ADR downlink in the documented SF9. The same can been seen in the code. But I feel TTN should not keep using SF12 for ADR in RX2 then^†; assuming your node is using the TTN specific settings that it got in the EU868 OTAA Join Accept, it will never receive the RX2 downlinks when TTN uses SF12…

On the other hand, maybe the bug is that it should not repeat forever, but only once. And maybe it uses SF9 again when the node sets ADRAckReq at some later time again. And the other nodes do accept the RX2 SF9 downlink?

@gigi130358

from the mosquitto log there is a “config” payload

That’s the gateway’s configuration for the downlink. Like frequency, data rate and power that is to be used for the transmission. It’s not some configuration payload that is sent to the node.

The data sent to the node is taken from, e.g., "payload": "YOEvASaFGAADVf8AAzz/Bqw=" for the SF9 downlink, which indeed is a MAC command in FOpts = 0355FF0003, which according to the 1.0.2 specification and the EU868 regional parameters decodes to:

• 0x03 = LinkADRReq MAC command
• 0x55 = DataRate_TXPower: data rate 5 (SF7/125KHz); TX power 5 = 6 dBm
• 0xFF00 (LSB) = 0b00000000 11111111 (MSB) = ChMask = channels 1 thru 8
• 0x03 = 0b00000011, so ChMaskCntl = 0b000, so indeed the bits in ChMask above directly refer to individual channels. This also sets NbTrans = 3, telling the node that each unconfirmed uplink should be transmitted 3 times. (I don’t understand that, but I assume it’s unrelated to your problems, as I assume it’s the same for the other nodes…)

And the first SF12 downlink, which your node will not have received, gives FOpts = 0503D2AD84 0355FF0003, which indeed first tells the node about the TTN specific RX2 settings:

• 0x05 = RXParamSetupReq MAC command
• 0x03 = DLsettings: RX1DRoffset = 0 (RX1 downlink uses the same data rate as the preceding uplink) and RX2DataRate = 3 (SF9/125kHz for RX2)
• 0xD2AD84 (LSB) = 0x84ADD2 (MSB) = RX2 frequency 869.5250 MHz

…followed by the LinkADRReq MAC command 0x0355FF0003 as described above.

So, all above aside: why didn’t that specific node receive or process the SF9 downlink? (As other nodes do process it, so certainly receive it, I doubt the gateway is to blame.)

---

^† It seems that for OTAA, this was fixed in October 2019: RX2 fallback leads to endless downlinks · Issue #767 · TheThingsArchive/ttn · GitHub, for devices that joined after that fix.

[arjanvanb (Arjan)]

@arjanvanb

This also sets NbTrans = 3, telling the node that each unconfirmed uplink should be transmitted 3 times?

Ah, the code seems to show that this number is increased when packet loss has been detected. Surely shielding all night has made TTN detect that. In time, this number will decrease again…

[gigi130358 (Gianluigi)]

Thanks Arjan fo your detailed and really interesting explanation, I’ll let you know about any news.
Gianluigi

[gigi130358 (Gianluigi)]

In order to identify where is the problem, I’m trying debugging the connection with a ThingsNode.
When I download a byte on the port 0 of the node, I’m able to see it in the log of the node, even if the msg will be reported related to port 2.

When I send to port 2 it’s correctly reported oin port 2

Setting the adr on in the node (ttn.reset(true)), when the msg cnt is 64, TTN correctly send an adr command to the node (14:01:42), but there isn’t any trace of this msg in the node debug console.

Considering that in the node code I’m using the method “ttn.onMessage(message)” for receiving the download msg, is it correct that it catchs only the application level or should it also get the mac msg ?

Gianluigi

[arjanvanb (Arjan)]

@gigi130358

When I download a byte on the port 0 of the node

That’s not supported; port 0 is reserved for downlinks without any application payload. That is: downlinks that only hold network MAC commands.

I assume TTN actually fixes this for you. It surely does that if it includes a MAC command.

@gigi130358

“ttn.onMessage(message)” […] catchs only the application level

Yes. Any MAC commands will be handled by the RX2xx3 chip.

[gigi130358 (Gianluigi)]

Thanks Arjan,
so if I correctly understand, I’m not able to verify if the gateway forward the adr request to the node by mac command.
Anyway at the beginning of the log the mac command sent by the node are reported.

[arjanvanb (Arjan)]

The gateway won’t change whatever is in the downlink. It cannot even read all its contents and surely cannot change the MIC, as it does not know the secrets.

But: you can decipher the downlink using the gateway’s log, or the gateway’s Traffic page in TTN Console. So if you see that a downlink was received in the node’s log, you’ll also know if that included any MAC commands. Of course, you won’t know if the RN2xx3 handled those…

You started with:

@gigi130358

I’ve 4 nodes of the same type

Maybe they actually have different versions of the RN2xx3 chips, or their firmware?

The version 1.0.4 you show in your last screenshot is slightly outdated but probably good enough.

[gigi130358 (Gianluigi)]

Sorry, may be I wasn’t clear enough.
The last test wasn’t carried out with one of the four nodes that give me the original problem, because I could’t conect with them to see theirs log.
For check if the adr commnad sent by TTN reach the node, I used another lora node (“The Things Node”).
In the Arduino IDE debug console of this node, I’m able to see the downlink msg that I manually sent from the TTN console, but when the TTN console shows an adr command sent from TTN to “The Things Node” after 64 uplinks, there is not trace of this msg in the Arduino IDE debug console of the node.

[arjanvanb (Arjan)]

Yup, it depends on the LoRaWAN implementation if you can see any MAC commands in the log. For example, for LMIC on Arduino one can enable verbose logging to see ADR MAC commands.

[gigi130358 (Gianluigi)]

It’s probable that the mac msg are not logged on the node console.
I set SF 8 and enabled adr and after 64 msg it correctly swith from SF 8 to 7 even if the mac command it’s not reported in the log.

[gigi130358 (Gianluigi)]

Always using the same setup based on “The Things Node” I investigate the behaviour with different SF initial setup obtained defining the “sf” with the method:
TheThingsNetwork ttn(loraSerial, debugSerial, freqPlan, sf);

Starting from SF7 to 10 included, both the Join Request and Accept are always with the SF set up by the above instruction.
These are the output of the TTN gateway console and the Things Node debug console:

…

but if I set SF to 11 or 12, the Join Accept answer is always with SF12


For these two last conditions with SF11 and 12, the node try 3 times to join without any success and then try to send messagge but of course it was not able to.

I don’t know the detail of the lorawan at this level, it’s normal this behaviour in your opinion ?
Thanks,
Gianluigi

P.S.
One more think that seems confirmed is that both Things Node and another type of adr functioning node set the first uplink with counter 0, while the four nodes that give adr problems start with cnt 1.

[kersing (Jac Kersing)]

@gigi130358

normal this behaviour in your opinion

Please search the forum for rn2483 and SF12 and take a look at the discussions regarding the use of RX2.

With regards to the failing joins at SF11 and SF12, check this forum message and the ones below.