Multitech released their latest firmware 3.3.5, which bumps up to LoRaWAN 1.0.4 and i’m having difficulty getting data to my application… join works OK and data makes it to the gateway OK, but never shows up on the application. If I downgrade to their 3.2.1 version which is LoRaWAN 1.0.3rA then everything works fine… any thoughts on settings that may have changed with their latest firmware?
https://os.mbed.com/teams/MultiTech/wiki/Dot-firmware-change-log
TTN does not (yet) implement the 1.0.4 standard and it seems not to be backwards compatible?
Looking at the release notes my guess is that
Join App Nonce is validated to increment with each Join Accept received
might cause issues. The 1.0.3 standard allows for use of a random value (or some sort of unique ID).
I don’t know what the 1.0.4 standard requires as it seems not to be publicly available yet. From what I understood during the virtual conference yesterday the standard was not final yet and no test suite is available yet so I am surprised multitech released firmware implementing it.
From discussions with L-A staff at TTConf in Jan I understood 1.04 (adding additional country plans I had potential clients interested in etc.*) had been agreed a few days before the Conf, potentially allowing some suppliers to move quickly to new firmware by now - but then again I’m usually wrong and as you say it may have taken time to then get test suite out though would have expected that would also be agreed as part of the new spec release? 
*or was it just an update to the RP’s? Maybe someone from TT Core team can advise/is aware of potential compatability issues?
Note that 1.1.x devices will (should) be backwards compatible with 1.0.x networks: during the Join Accept a 1.1.x device will see that a 1.0.x network responded, which should make it fall back to 1.0.x as of that point. I know that’s not your use case. But given that, I’d really expect 1.0.4 to be either compatible with earlier 1.0.x releases, or support detection of the network using 1.0.2 (which is what the TTN community network uses) and then fall back to 1.0.2 or 1.0.3 as well.
But: those are just assumptions.
So TTN accepted the join, and as the device starts to send data uplinks it has handled the OTAA Join Accept. Another assumption: I’d guess that TTN fails to validate the MIC of those uplinks, and hence silently drops those. The MIC is calculated using the secret NwkSKey that the node will have calculated from the OTAA Join Accept. Assuming backwards compatibility, so assuming that a 1.0.4 device should work with a 1.0.2 network: if the device would have calculated a different NwkSKey or MIC then that would be a bug in the new firmware?
If you want to debug, you could capture some full LoRaWAN data uplink packets from the gateway’s Traffic page in TTN Console for both 1.0.3 and 1.0.4, and compare using an online decoder. The decoder will assume 1.0.2, just like the TTN community network does.
Be sure to make the decoder validate the MIC too, using the secret NwkSKey (and secret AppSKey) from TTN Console. (Note that the DevAddr, NwkSKey and AppSKey change for each successful OTAA join.) If validation indeed fails for 1.0.4 using the NwkSKey from TTN Console, but if the uplinks’ structures look otherwise similar, then maybe you could make the device print the secrets it calculated from the Join Accept?
I’ve not read the change log you provided, but in an unrelated other example some funny configuration like #define LORAWAN_NWK_KEY LORAWAN_APP_KEY was required to get the correct MIC for an OTAA join. Again, that’s just an unrelated example, but I would not rule out that the configuration for your device might have changed for the new firmware…
The website of the alliance still links to 1.0.3 so I expect that to be the current version…
The Dot is sending a DeviceTimeReq with the uplinks. This causes the uplinks to be rejected by the server as they are new to LoRaWAN 1.0.4
at+mac
0dOK
Disable FOTA and Join Nonce Validation before joining. TTN is sending a random AppNonce value in the Join Accept so validation needs to be disabled. Otherwise joins will randomly fail. The FOTA settings is not saved to flash, it will need to be set after a device reset.
at+fota=0
at+jnv=0
at+join
So we can assume that there is no detection of a 1.0.2 network in 1.0.4 devices? 
Man… Multitech really sneaks in some new AT commands without clear guidance on how they affect a current deployment. thanks for the advice… i’ll try this next week.
That would defeat the security enhancements of the updated protocol features. The join nonce counters allows protection from replays.
True, but it’s not like one can accidentally connect to some unknown network. (The device’s secret AppKey will always need to be registered to the network of choice.) So, I’d say that an automatic fallback is better than requiring configuration in the device.
I think many users won’t even know that TTN’s community network is using a less-secure 1.0.2, and I’m afraid (and totally understand) that people just buy an off-the-shelf device that happens to be shipped with 1.0.4, which then does not work on the older network. Or worse: that works just fine the first time, but take much longer to join every next time.
I assume the LoRa Alliance has given this breaking change much thought, but 1.0.4 devices requiring configuration in the device to work on a 1.0.2/1.0.3 network is not something that helps the market for off-the-shelf devices.
I agree that automatic detection of lorawan version would aid easier deployment.
TTN could update the JS to use counters as recommended by the lora alliance since 2018.
See page 8.
https://lora-alliance.org/resource-hub/technical-recommendations-preventing-state-synchronization-issues-around-lorawanr-10x
This could get very messy and disruptive very quickly! 
Sounds like we will have to be wary of what we buy and ask questions and take care with any firmware updates. One problem is some updates may include feature or security updates/changes but be wrapped up with move to 1.04 forcing some difficult choices (starts to sound like MSFT Windows updates! 
)
@johan what is TTN/TTI take/position on this? given Jason’s last comment