Using the MQTT API you might also see the downlinks, including the time they were queued, though I’m not sure ADR is included in that (maybe that only includes application downlinks). That might also tell you if things are queueing up (for which the backend should set the “downlink pending” flag to make the node transmit another uplink if it wants to).
As the MIC validates, I guess we agree you’re seeing old (queued, repeated, …?) downlinks, given the lower frame counters. Next, I wonder if the 2nd line below tells one when the TTIG received the downlink from TTN:
It seems these lines are not logged when a “dnmsg” command is received by the TTIG, but only when it’s handling its internal TX queue. So it seems one cannot tell when the TTIG receives commands from TTN.
If lines like this are logged upon arrival of the downlink commands, then the time those arrive at the TTIG clearly is weird. If, like for other forwarders, the Basic Station protocol includes some relative time with each uplink, and the backend just adds 1,000,000 or 2,000,000 microseconds to that value to tell the TTIG when to transmit, then I could imagine that relative timestamps in the metadata of old queued downlinks mess up the time a downlink is actually transmitted, like due to rollover of the timers. But would that also make the backend (or the intermediate bridge that sits between the Basic Station protocol and the V2 backend) forward the downlink to the TTIG so late after an uplink?
Any chance there’s something different for the first downlink that went wrong? Or maybe sometimes the very same downlink is forwarded to the TTIG multiple times?
Also, I think we need @KrishnaIyerEaswaran2 here.