If devices stopped working after TTN upgrade on Jan 31st, 2022

TTN Network and Services
1

Hello,

In case somebody else has the same problem:

I have some Heltec cubecells which use the LoRaWAN 1.0.2 protocol.
My devices stopped sending after The Things Network upgrade on January 31st.
Afterwards there were a lot of join errors from each device, either “MIC mismatch” or “DevNonce has already been used”.

With the TTN upgrade there was a fix (mentioned on the upgrade announcement Upgrade to The Things Network v3.17.2 (completed)):

  • When an end device has both NwkKey and AppKey provisioned in the Join Server, NwkKey is used for MIC and session key derivation when activating the device in LoRaWAN 1.0.x. This is per LoRaWAN 1.1 specification.

I didn’t care much for NwkKey yet, but this “fix” seems to have caused my problem.

I could finally bring the devices up again, by setting the NwkKey to the same value as the AppKey.

In case, anybody has the same problem, I used this script for it:

ttncli() {
  PATH_TO/ttn-lw-cli \
    --network-server-grpc-address eu1.cloud.thethings.network:8884 \
    --join-server-grpc-address eu1.cloud.thethings.network:8884 \
    --gateway-server-grpc-address eu1.cloud.thethings.network:8884 \
    --application-server-grpc-address eu1.cloud.thethings.network:8884 \
    --identity-server-grpc-address eu1.cloud.thethings.network:8884 \
    $@
}
APPID="--application-id XXX_my_application_id"
for i in $(ttncli end-devices list $APPID --limit 1000 | jq -r .[].ids.device_id); do
    ttncli end-devices set $APPID --device-id $i \
        --root-keys.nwk-key.key $(ttncli end-devices get $APPID --device-id $i --root-keys.app-key | jq -r .root_keys.app_key.key);
done
1 Like
2

Excellent share, thank you.

I sanitised the title - the devices are still sending but the stack is not able to process them as we’d expect.

For those who’s CLI foo isn’t yet strong, the script is a shell script, the $@ adds the arguments (CLI parameters) to what is effectively a function call, the loop then calls that function as required and the jq is the JSON Parser program that will need to be installed for it to automagically pick out the information required from the responses and turn it in to an update.

1 Like
3

I just found out that I can just delete the NwkKey.

The script is a bit simpler:

ttncli() {
  PATH_TO/ttn-lw-cli \
    --network-server-grpc-address eu1.cloud.thethings.network:8884 \
    --join-server-grpc-address eu1.cloud.thethings.network:8884 \
    --gateway-server-grpc-address eu1.cloud.thethings.network:8884 \
    --application-server-grpc-address eu1.cloud.thethings.network:8884 \
    --identity-server-grpc-address eu1.cloud.thethings.network:8884 \
    $@
}
APPID="XXX_my_application_id"
for i in $(ttncli end-devices list $APPID --limit 1000 | jq -r .[].ids.device_id); do
    ttncli end-devices set $APPID $i --unset root-keys.nwk-key;
done

In case you wonder why I ran into this problem:
I register my devices using the command line ttn-lw-cli with params end-devices create [....] --with-root-keys.
This creates both the AppKey and the NwkKey. But my devices only use the AppKey.

4

You are right, this is a regression, my bad. I initially implemented --with-root-keys three years ago, generating AppKey and NwkKey regardless of LoRaWAN version, and recently changed behavior in Join Server to interpret existence of both root keys as LoRaWAN 1.1 provisioned. Disclaimer: that root key generation was written during some coding evening the week before launching The Things Stack at The Things Conference 2019; https://github.com/TheThingsNetwork/lorawan-stack/commit/c5f622c6de0a04773ad502e312fbc47ee49bb730

Anyway, we will try to fix this with a work-around in the Join Server and a permanent fix for the CLI not to generate both root keys when the LoRaWAN version is specified and not 1.1 or higher. For the time being, indeed remove NwkKey.

1 Like