Issue with confirmed messages

[fhander]

I just received my TTN Indoor GW today and setup went smoothly and successful.
I’m now able to send data from my LoStik lorawan usb-dongle and see the traffic in the TTN Gateway console as well as in my TTN Device console.

When I send an unconfirmed message, I get a mac_tx_ok response.
But when I send a confirmed messages I can also see the messages in the console, but my LoStik device apparently never receives the conifrmation and retransmit the message 7 times and then I get a mac_error response.

My LoStik is based on the RN2483 chip and I interact with the device using Termite.
Some command excerpt of device interaction:
mac tx uncnf 1 010210
ok
mac_tx_ok
…
mac tx cnf 1 020210
ok
mac_err

I understand that there is a limitation on the number of confirmed messages one may send per day, but this error shows up on all, also the first.

Any help welcome.

[cslorabox]

You need to figure out if TTN is generating a downlink message or not.

If not, something is wrong with your uplink.

If so, then there is a slight chance the gateway is failing to transmit but a much larger chance that something about how the node is trying to receive is incorrect, for example listening on the wrong frequency or spreading factor or listening at slightly the wrong time.

Generally speaking you should not use confirmed messages anyway. On TTN you’d rapidly exceed the usage policy if you do so. And even on the private network, the protocol is poorly thought out since if it happens to be the confirmation that is lost, the retries won’t get a confirmation either.

Generally it’s better to send fresher data at application level rather than try to resend old data, and if there is occasional information you absolutely need to confirm got through, confirm that at application level rather than network protocol level.

But of course you first have to get downlink working at all, issues there are your most likely problem.

[fhander]

I see downlink messages in the TTN Gateways console. I think one for each uplink message.
But I don’t know how the interpret the messages as confirmation messages.
Yes, I will probably not use confirmed messages. I just wanted to verify my device/gateway setup including confirmed messages, although these may be of little use.
How would I know the correct setup of my device if it must conform to some TTN GW setup? I would have thought that the RN2483 would work without some dedicated configuration?

[arjanvanb (Arjan)]

@fhander

I don’t know how the interpret the messages as confirmation messages.

Either decode the full LoRaWAN packet that you see in the gateway’s Traffic page (like in your screenshot). Or look at the application’s or device’s Data page, so see TTN Console already giving you some more descriptive details about the downlink.

Given the size of 22 bytes, it’s not just a confirmation for sure.

By the way: I wonder why all downlinks are on SF12, and use the same frequency. Is the node set to use a single frequency? And why SF12?

[fhander]

Ok, learning a lot these hours

When I decode in node.js the downlink message from the Gateways console view, I get:
FRMPayload =

            ( FHDR = DevAddr[4] | FCtrl[1] | FCnt[2] | FOpts[0..15] )
           DevAddr = 2601182B (Big Endian)
             FCtrl = AA
              FCnt = 0038 (Big Endian)
             FOpts = 0503D2AD840351070001

      Message Type = Unconfirmed Data Down
         Direction = down
              FCnt = 56
         FCtrl.ACK = true
         FCtrl.ADR = true

So drom the FCtrl = 10101010 it seems that the ACK bit (5) is set but also that the length bits are 1010 (i.e. 10).

So is this a confirmation message? And if it is, why doesn’t my LoStik (RN2483) receive it?

Wrt to the SF = 12, I have no idea from where that comes.
On my device I get:
radio get sf
sf9

So should I set SF to 12 as indicated in the messages? Or ?

[cslorabox]

That downlink message has an ADR command and channel map in it too, which your node has probably not honored yet.

Key question at this point would be what the uplink SF & frequency are. If they are all the same frequency, your node is not operating correctly.

If the uplink frequency is varying but the downlink is stuck on SF12 869.525, then it’s possible these are all RX2 downlinks with the default SF12 (vs the TTN specific SF9) being reverted to after the node missed too many ones SF9. Though it would be unclear why you are seeing only RX2 downlinks.

Knowing the settings of the uplink is key at this point. Even better, an uplink-downlink pair showing both settings and timestamps.

[kersing (Jac Kersing)]

@cslorabox

Though it would be unclear why you are seeing only RX2 downlinks.

These are RX2 downlinks (this frequency is only used for RX2). TTN uses RX2 for all downlinks when the uplinks are SF10 - SF12.

[cslorabox]

@kersing

TTN uses RX2 for all downlinks when the uplinks are SF10 - SF12.

How does that possibly work in a situation where SF12 is actually needed to get the signal through?

Normal TTN SF9 RX2 does not seem like a strategically appropriate response to an SF12 uplink at all - I guess they are counting on hammering it with enough extra transmit power to overcome the reduced coding gain?

[arjanvanb (Arjan)]

@kersing

this frequency is only used for RX2

Almost right, and I almost made the same mistake in my earlier answer: you’re confused with 869.525 - SF9BW125 (for EU868).

[cslorabox]

@arjanvanb

Almost right, and I almost made the same mistake in my earlier answer: you’re confused with 869.525 - SF9BW125 (for EU868).

“Almost right” it reportedly falls back to the standard SF12 eventually

[arjanvanb (Arjan)]

@cslorabox

it reportedly falls back to the standard SF12 eventually

True:

@arjanvanb

Now, an example has proved that, when using RX2, TTN surely is smart enough to repeat ADR on SF12 if nothing changed when the node ignored the ADR downlink in the documented SF9. The same can been seen in the code. But I feel TTN should not keep using SF12 for ADR in RX2 then; assuming your node is using the TTN specific settings that it got in the EU868 OTAA Join Accept, it will never receive the RX2 downlinks when TTN uses SF12…

On the other hand, maybe the bug is that it should not repeat forever, but only once. And maybe it uses SF9 again when the node sets ADRAckReq at some later time again. And the other nodes do accept the RX2 SF9 downlink?

So, @fhander, indeed:

@cslorabox

Key question at this point would be what the uplink SF & frequency are.

@cslorabox

Normal TTN SF9 RX2 does not seem like a strategically appropriate response to an SF12 uplink at all - I guess they are counting on hammering it with enough extra transmit power to overcome the reduced coding gain?

Also true, see:

@arjanvanb

@telkamp

in RX2 we transmit at 27 dBm, and the increase in power (over the 14dBm uplink) offsets the loss of sensitivity going from SF12 to SF9

In Slack, someone wrote that many gateways only support 20 dBm:
…

[fhander]

It seems, from the Gateways console log, that both uplink and downlink messages uses:

lora": {
“spreading_factor”: 12,
“bandwidth”: 125,

Not, that it is clear to me if that is a problem or not ?

[arjanvanb (Arjan)]

When the uplink is SF12, then a downlink in RX1 should be SF12 too. The SF12 indeed suggests it’s RX1, though for an SF12 uplink I’d expect TTN to use RX2 for the downlink instead (which for EU868 then uses 869.525 SF9BW125).

So, SF12 for both uplink and downlink should work, though it’s a bit unexpected.

Still then: what about the uplink frequencies? Are they indeed always 869.5? A proper RN2483 will at least hop between 868.1, 868.3 and 868.5, and if it processed the OTAA Join Accept, or if it was properly configured for ABP, then it should also use some other frequencies. It should not use the same frequency for each uplink.

Are you using OTAA or ABP? For ABP you need to set the frequencies and RX2 settings manually.

And what version of the RN2483 are you using? I’ve read that SF12 is troublesome for older versions, but maybe that only applied to OTAA.

If all this is done by manually entering commands through a serial connection, then maybe it’s time to factory reset the RN2483 and delete and re-add the device in TTN Console.

[arjanvanb (Arjan)]

@arjanvanb

Almost right, and I almost made the same mistake in my earlier answer: you’re confused with 869.525 - SF9BW125 (for EU868).

Uh, wow, that is actually the frequency that was reported. I was wrong, @kersing is right! And the SF12 is wrong…?

[kersing (Jac Kersing)]

No, I am right. 869.525 is only used for RX2. And as mentioned by @cslorabox

@arjanvanb

When the uplink is SF12, then a downlink in RX1 should be SF12 too. The SF12 indeed suggest it’s RX1, though for an SF12 uplink I’d expect TTN to use RX2 for the downlink instead (which for EU868 then uses 869.525 SF9BW125)

When an uplink is SF12 TTN will always respond in RX2 (check the code if you don’t believe this). This is chosen to preserve airtime on the other frequencies (RX2 frequency has different allowances).
The SF12 issue in RX2 is caused by failing logic in the back end that falls back to SF12 assuming that a node that does not accept ADR settings when using SF9 might receive them using SF12 and the back-end does not recover from this setting.
Deleting the node from the application and re-adding it seems to the the only solution. (The is/was an issue in GitHub for this behavior of the V2 stack)

[arjanvanb (Arjan)]

For OTAA, I think the issue might/should have been resolved:

RX2 fallback leads to endless downlinks

But beware, its comments say:

@jpmeijers commented on 15 Oct 2019

I just realised this fix will only be applied to devices that join after the fix is applied. Devices that are using a session from a join before today will still suffer from this bug.

[fhander]

I have joined in ABP mode.

The uplink freq is: “frequency”: 868500000
and the downlink freq is: “frequency”: 869525000
as reported by the TTN Gateways console log.

[fhander]

I reasoned from your comments that possbly the sf12 setting was wrong. Hence I changed the radio setup to use sf7 and after joining ABP the confirmed as well as unconfirmed messages work.
Thanks for your help, although I don’t quite understand why there should be a problem using sf12?

[kersing (Jac Kersing)]

With some versions of the firmware RN2483 modules have issues receiving data using SF12. Search the forum if you want to know details.

[arjanvanb (Arjan)]

Generic SF12 problems with some RN2483 firmware versions aside: every properly configured ABP device that uses ADR might run into the same issues for RX2, whenever just one ADR downlink has not been received?

A long read to explain what probably happened, and might happen again:

Your node has ADR enabled, so tells TTN that the network should control the device’s transmission settings (like channels, SF and power). As you were using SF12 (which is a huge waste of air time), TTN will very soon have sent an ADR downlink, to command your device to use, say, SF8 instead. But somehow your device missed that ADR downlink and continued using the old settings.

Now, in EU868, TTN uses specific settings for RX2: instead of the LoRaWAN default of SF12 (on 869.525 MHz), TTN uses SF9 (also on 869.525, but with a higher transmission power). When using OTAA in EU868, a device gets this network settings automatically in the Join Accept. But for ABP this needs to be programmed into the device, and some people forgot to make this change and kept wasting valuable shared airtime. To mitigate that, in September 2017 TTN added a fallback to use the default LoRaWAN RX2 settings (and enhance the ADR downlink to tell the node about the proper TTN RX2 settings) when it notices that a device has enabled ADR, but does not adjust its settings after an ADR downlink. This indeed works for devices that are not configured properly.

However, for properly configured ABP devices, TTN also (erroneously) assumes a device is using the LoRaWAN RX2 defaults when it somehow failed to receive an ADR downlink. And as of that point, TTN keeps using those defaults for all subsequent ADR downlinks, even though a properly configured device will never receive RX2 on SF12. At this point the device is stuck at SF12 even though it has ADR enabled, while TTN keeps repeating the ADR downlinks on the wrong SF.

Next, when manually switching the device to use SF7, TTN will switch from RX2 back to RX1, and all is fine again: the device will receive the pending ADR downlink, and TTN sees all is fine because the next uplink uses the expected settings.

However, if the reception quality drops some time later (including when moving further away from the best gateway, or when a nearby gateway is taken offline), the device might notice that its uplinks are no longer received. (An ADR-enabled device expects to see some downlink activity every now and then.) This will make the node increase its SF, until it knows that its transmissions are received again. In bad conditions (including when the network is down for a long time), this means that at some point it may end up at SF11 or SF12 again. If some time later the circumstances improve again, and TTN uses RX2 to send an ADR downlink to command the node to improve its settings, the whole problem might start again.

In October 2019, this has been fixed for OTAA, where TTN no longer assumes the LoRaWAN defaults for devices that have joined after that fix. But for ABP I think there’s always the risk that ADR will mess up?

In short: use OTAA. If you really must use ABP, then disable ADR. (Note that using SF11 or SF12 is not allowed then.)

 

@kersing

When an uplink is SF12 TTN will always respond in RX2 (check the code if you don’t believe this).

Even more so: browsing the latest v2.10.3 test code, the scoring code (including the comment “Impact on duty-cycle (in order to prefer RX2 for SF9BW125”) and the June 2016 commit title “Prefer RX2 over RX1 for SF9+ [EU]” that introduced the changes that are still in the latest version today, I’m tempted to say that ever since June 2016 for EU868 TTN prefers RX2 for SF9 and up?

I feel that doesn’t match the following from April 2017, unless the following statement assumes that in EU868 most uplinks use SF7-SF8 anyhow:

@htdvisser

We try to avoid RX2 if possible. For RX1 we have 8 channels available, while RX2 is fixed on 1 channel and a very inefficient data rate.

Maybe the test code fools me as the mock gateway has no other things to compare (such as how busy a channel is with receiving uplinks). I guess I need to do some real testing rather than browse the code…