Class B Downlink fails: (in 2 different ways)

I am experimenting with Class B operation on the V3 servers. Should this be working?

So far I can place my device in Class B operation and receive beacons OK. Uplink packets reach the server. Downlink packets reach the device in Class A but not in Class B. (I generate a single byte payload on port 2 with the “Schedule Downlink” feature in the console.) AFAIK the server is not sending instructions to the gateway to transmit the packet to the device.

There seem to be (at least) two situations: (1) the message is apparently processed but does not reach the gateway (in fact, does not appear in the V3 console Gateway Live Data feed), and (2) the message is rejected (in the V3 Device Live Data feed) because “gateway time is not GPS synchronized” (AFAIK the GPS is working fine).

json messages for each of these cases follow.

Any comments on whether Class B is working, even in beta, and what might be required to make it work for me, would be much appreciated.

case (1) - this message appears in the “End devices” console, but AFAIK no equivalent message shows up in the corresponding “Gateway” console :

{
  "name": "as.down.data.forward",
  "time": "2021-03-23T07:06:33.407155207Z",
  "identifiers": [
    {
      "device_ids": {
        "device_id": "ww120-48-03",
        "application_ids": {
          "application_id": "wetawatcher-1"
        }
      }
    }
  ],
  "data": {
    "@type": "type.googleapis.com/ttn.lorawan.v3.ApplicationDownlink",
    "f_port": 2,
    "frm_payload": "4A==",
    "correlation_ids": [
      "as:downlink:01F1EYG4HN5G4DC20XPM939J5P",
      "rpc:/ttn.lorawan.v3.AppAs/DownlinkQueueReplace:93925844-9c24-4684-bb7b-f2812928aedd"
    ]
  },
  "correlation_ids": [
    "as:downlink:01F1EYG4HN5G4DC20XPM939J5P",
    "rpc:/ttn.lorawan.v3.AppAs/DownlinkQueueReplace:93925844-9c24-4684-bb7b-f2812928aedd"
  ],
  "origin": "ip-10-102-15-15.ap-southeast-2.compute.internal",
  "context": {
    "tenant-id": "CgN0dG4="
  },
  "visibility": {
    "rights": [
      "RIGHT_APPLICATION_TRAFFIC_READ"
    ]
  },
  "authentication": {
    "type": "Bearer",
    "token_type": "AccessToken",
    "token_id": "NBYXVNEL2IST2MHNLW7WZVC7ZBW3F24HGM5G2CI"
  },
  "remote_ip": "118.148.81.24",
  "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0) Gecko/20100101 Firefox/86.0",
  "unique_id": "01F1EYG4HZJNWHYRYTQQFG8CAX"
}

case (2) - I get a failure message in the “End devices” console, despite the gateway having a decent GPS connection, and the beacons being transmitted from the gateway to the device:

{
  "name": "ns.down.data.schedule.fail",
  "time": "2021-03-23T06:55:02.859363285Z",
  "identifiers": [
    {
      "device_ids": {
        "device_id": "ww120-48-03",
        "application_ids": {
          "application_id": "wetawatcher-1"
        },
        "dev_eui": "0080E11500074803",
        "join_eui": "70B3D57ED003C8D0",
        "dev_addr": "260DA0B7"
      }
    }
  ],
  "data": {
    "@type": "type.googleapis.com/ttn.lorawan.v3.ErrorDetails",
    "namespace": "pkg/gatewayserver",
    "name": "schedule",
    "message_format": "failed to schedule",
    "correlation_id": "2a2a080fcf164ca79f389dd19b8d99f0",
    "code": 10,
    "details": [
      {
        "@type": "type.googleapis.com/ttn.lorawan.v3.ScheduleDownlinkErrorDetails",
        "path_errors": [
          {
            "namespace": "pkg/gatewayserver",
            "name": "schedule_path",
            "message_format": "failed to schedule on path `{gateway_uid}`",
            "attributes": {
              "gateway_uid": "acutetech-rak7244-for-v3@ttn"
            },
            "cause": {
              "namespace": "pkg/gatewayserver/io",
              "name": "tx_schedule",
              "message_format": "failed to schedule",
              "code": 10,
              "details": [
                {
                  "@type": "type.googleapis.com/ttn.lorawan.v3.ScheduleDownlinkErrorDetails",
                  "path_errors": [
                    {
                      "namespace": "pkg/gatewayserver/io",
                      "name": "rx_empty",
                      "message_format": "settings empty",
                      "code": 9
                    },
                    {
                      "namespace": "pkg/gatewayserver/io",
                      "name": "no_gps_sync",
                      "message_format": "gateway time is not GPS synchronized",
                      "code": 9
                    }
                  ]
                }
              ]
            },
            "code": 10,
            "details": [
              {
                "@type": "type.googleapis.com/ttn.lorawan.v3.ScheduleDownlinkErrorDetails",
                "path_errors": [
                  {
                    "namespace": "pkg/gatewayserver/io",
                    "name": "rx_empty",
                    "message_format": "settings empty",
                    "code": 9
                  },
                  {
                    "namespace": "pkg/gatewayserver/io",
                    "name": "no_gps_sync",
                    "message_format": "gateway time is not GPS synchronized",
                    "code": 9
                  }
                ]
              }
            ]
          }
        ]
      }
    ]
  },
  "correlation_ids": [
    "as:downlink:01F1EXV1Y3BTZ4GYRMVSWZ25YN",
    "ns:downlink:01F1EXV269D4BS8N8Z4RNCSSE0",
    "rpc:/ttn.lorawan.v3.AppAs/DownlinkQueueReplace:fa9d0aec-a20d-4c62-bdec-b6772d33f101"
  ],
  "origin": "ip-10-102-7-115.ap-southeast-2.compute.internal",
  "context": {
    "tenant-id": "CgN0dG4="
  },
  "visibility": {
    "rights": [
      "RIGHT_APPLICATION_TRAFFIC_READ"
    ]
  },
  "unique_id": "01F1EXV26B6W3G8B0ZK6WPAXJD"
}

Can you show some raw uplink reports from this gateway and check if they include valid GPS times?

Hmmm… behaviour is complex and I get different results with different attempts at sending the downlink message. Once the downlink packet arrived. Usually it does leave TTN. It is difficult for me to keep an eye on my device’s console, my local gateway packet forwarder and two TTN consoles (gateway and device). AFAIK the GPS is reporting valid location and time to the server in the JSON up messages, at least that is what I understand the “tmms” field represents. (Though I do get “WARNING: [gps] GPS out of sync, keeping previous time reference”)

Just one example, with a new failure mode: I attempted to send a single byte from the TTN device console. The message sequence shows 6 attempts/failures as scheduling this message. The failure report includes “gateway time is not GPS synchronized”, despite every “rxpk” message including a “tmms” field (is that how your server is supposed to determine if my gateway has the GPS time? If not, how?).

I then asked my devce to send an upstream packet, which arrived. This prompted a successful scheduling of the downlink message. These messages are shown here (oldest at the bottom):

Successfully scheduled data downlink for transmission on Gateway Server
Forward data message to Application Server
Schedule data downlink for transmission on Gateway Server
Forward uplink data message
Receive uplink data message
Successfully processed data message
Receive data message
Failed to schedule data downlink for transmission on Gateway Server
Schedule data downlink for transmission on Gateway Server
Failed to schedule data downlink for transmission on Gateway Server
Schedule data downlink for transmission on Gateway Server
Failed to schedule data downlink for transmission on Gateway Server
Schedule data downlink for transmission on Gateway Server
Failed to schedule data downlink for transmission on Gateway Server
Schedule data downlink for transmission on Gateway Server
Failed to schedule data downlink for transmission on Gateway Server
Schedule data downlink for transmission on Gateway Server
Failed to schedule data downlink for transmission on Gateway Server
Schedule data downlink for transmission on Gateway Server
Forward downlink data message
Receive downlink data message

Over on the TTN gateway console there is no sign of the failed attempts at scheduling the download. It does see the uplink message, processes it, then gets and sends the downlink message. These messages are shown here (oldest at the bottom):

Disconnect gateway  {I had stopped the gateway packet forwarder}
Transmit downlink message failure
Send downlink message
Forward uplink message
Forward uplink message
Receive uplink message
Forward gateway status
Forward gateway status

And at the gateway itself I get the following: note the status message, the outgoing beacon, the uplink message (when I asked my device to send an uplink packet), the downlink message from the server, followed by “ERROR: Packet REJECTED”. My ping slots are 8s apart. I have obscured my GPS coordinates.

JSON up: {"stat":{"time":"2021-03-23 23:02:04 GMT","lati":xxx,"long":xxx,"alti":10,"rxnb":2,"rxok":0,"rxfw":0,"ackr":100.0,"dwnb":0,"txnb":0}}
INFO: [up] PUSH_ACK received in 74 ms
INFO: Beacon dequeued (count_us=1983941148)
INFO: tx_start_delay=1495 (1495.500000) - (1497, bw_delay=1.500000, notch_delay=0.000000)
INFO: Beacon queued (count_us=2367941236, freq_hz=923400000, size=17):
   => 00 00 80 37 85 4D 09 E0 00 E3 38 0E E3 38 0E D2 7D
INFO: [down] PULL_ACK received in 70 ms

INFO: Received pkt from mote: 260D6B94 (fcnt=17)

JSON up: {"rxpk":[{"tmst":2013889811,"time":"2021-03-23T23:02:34.948258Z","tmms":1300575773948,"chan":0,"rfch":0,"freq":923.200000,"stat":1,"modu":"LORA","datr":"SF7BW125","codr":"4/5","lsnr":9.0,"rssi":-43,"size":21,"data":"QJRrDSaQEQACpPAgnxmpSoye2sR2"}]}
INFO: [up] PUSH_ACK received in 77 ms
INFO: [down] PULL_ACK received in 71 ms
INFO: [down] PULL_RESP received  - token[0:14] :)

JSON down: {"txpk":{"imme":false,"tmst":2018889811,"freq":923.2,"rfch":0,"powe":13,"modu":"LORA","datr":"SF7BW125","codr":"4/5","ipol":true,"size":14,"ncrc":true,"data":"YJRrDSaADAACj9MPmjI="}}
INFO: == used txlut index:8
src/jitqueue.c:251:jit_enqueue(): ERROR: Packet REJECTED, timestamp seems wrong, too much in advance (current=2019984514, packet=2018889811, type=0)
ERROR: Packet REJECTED (jit error=2)

I am happy to work with TTN on this, but not this way: it has taken an hour to write this post. Perhaps you can watch my consoles from your office, while I push buttons on my device, and share a screen?

It would be good to know the extent to which Class B has been tested…

You are aware you are using the community network for free and 99% of the people on this forum are users like you? If you want support you need to contact TTI and get a commercial deployment. If you want to work with the community you’re welcome on this forum but you will need to do your part of the work.

That’s not good. In the statistics overview, does the packet forwarder state it has a valid GPS position?

This is not good either. The data arrives after it should have been sent. Did you accidentally set ‘schedule downlink late’ in the LoRaWAN options for the gateway?

Here is some more information. It is only a partial picture, but it accounts for some of what I am seeing and perhaps points towards an explanation and solution.

First, Jac asks:

I can see correct GPS coordinates in the “stat” messages, and a “tmms” field in the “rxpk” message, which is “GPS time of pkt RX…” I would have thought these would indicate to the server that my gateway has a GPS fix. The GPS engine is delivering $GPGGA messages with a Fix Quality value of 2 “= DGPS fix” (as well as lat, long, time).

The “Scheduled downlink late” is not checked.

Further results and analysis:

When the device is in Class B mode and you attempt to send a downlink message using the TTN device console, the server that handles this task attempts to schedule the downlink message. Soon afterwards it decides (why?) that “gateway time is not GPS synchronized.” The message does not appear in TTN’s gateway console. The scheduling fails and after a delay it tries again. It loops continuously, until the device sends an uplink message.

At that point it decides (why?) that the gateway has a GPS time, and it can send the downlink message. I speculate that this is because the uplink message from the gateway contains the “tmms” message that indicates a valid GPS time. Then the downlink message appears in TTN’s gateway console and it is passed to the gateway.

In some cases (not sure which yet) the gateway decides the downlink message is OK and sends it to the device (maybe in the RX1 or RX2 slots scheduled after the uplink message?). In other cases the gateway decides that the timestamp in the message is wrong (perhaps the scheduled ping slot contained in the delayed downlink message is in the past?) and rejects it with “ERROR: Packet REJECTED, timestamp seems wrong”.

The most interesting question for me now is how the TTN server (which?) decides that “gateway time is not GPS synchronized”. Does it indeed look for “tmms” fields? Does it expect “tmms” field in the “stat” messages? (they are not supposed to be there). Does it compare the time in the “stat” messages with its own GPS time? I am in New Zealand communicating with Australian servers: is there a longer delay in my link than is expected?

In the failure message I see two error reasons:

                  "path_errors": [
                    {
                      "namespace": "pkg/gatewayserver/io",
                      "name": "rx_empty",
                      "message_format": "settings empty",
                      "code": 9
                    },
                    {
                      "namespace": "pkg/gatewayserver/io",
                      "name": "no_gps_sync",
                      "message_format": "gateway time is not GPS synchronized",
                      "code": 9
                    }

What is this “settings empty” message, and does it provide a clue?

I thought I would check if ACK downlink packets worked OK. They do. When in class B the server appears to receive and respond to both unconfirmed and confirmed uplink messages. That is, it sends downlink packets to the gateway with the ACK response and these are passed to my device. It looks like the downlink messages are delivered in the RX1 slot following the uplink, and not in the class B ping slots. Periodic ADR requests are also delivered.

As before, an attempt to send a downlink message from the TTN device console gives the “Failed to schedule” error with “gateway time is not GPS synchronized”.

Note that the downlink message can be delivered if I send an (unconfirmed) uplink message. It seems to be delivered in the RX1 slot.

I guess this all makes sense: Class A uplinks and corresponding RX1 downlinks behave as before. The server only fails when trying to use the Class B ping slots.

When running these tests, sending unconfirmed uplink messages, I sometimes see the gateway console reporting “Drop uplink message” and “Transmit downlink message failure” The uplink messages reach the TTN Device console OK. Details follow. (Perhaps these deserve their own thread?)

      {
  "name": "gs.up.drop",
  "time": "2021-03-25T04:52:06.753003141Z",
  "identifiers": [
    {
      "gateway_ids": {
        "gateway_id": "acutetech-rak7244-for-v3",
        "eui": "DCA632FFFE36FAC2"
      }
    }
  ],
  "data": {
    "@type": "type.googleapis.com/ttn.lorawan.v3.ErrorDetails",
    "namespace": "pkg/gatewayserver",
    "name": "host_handle",
    "message_format": "host `{host}` failed to handle message",
    "attributes": {
      "host": "cluster"
    },
    "cause": {
      "namespace": "pkg/networkserver",
      "name": "device_not_found",
      "message_format": "device not found",
      "correlation_id": "ec5d48df91124480b219fb8b8a0761d8",
      "code": 5
    },
    "code": 5
  },
  "correlation_ids": [
    "gs:conn:01F1KQBB0V0JM36GBQ0DX6RQD5",
    "gs:up:host:01F1KQBB9GY8R0WX8PZ2R0STE0",
    "gs:uplink:01F1KVKCYVZD1M9Y07WMEJ2NKC"
  ],
  "origin": "ip-10-102-13-163.ap-southeast-2.compute.internal",
  "context": {
    "tenant-id": "CgN0dG4="
  },
  "visibility": {
    "rights": [
      "RIGHT_GATEWAY_TRAFFIC_READ"
    ]
  },
  "unique_id": "01F1KVKCZ1GZ4W5PTSCQYPPV8X"
}

and


{
  "name": "gs.down.tx.fail",
  "time": "2021-03-25T04:54:02.707297018Z",
  "identifiers": [
    {
      "gateway_ids": {
        "gateway_id": "acutetech-rak7244-for-v3",
        "eui": "DCA632FFFE36FAC2"
      }
    }
  ],
  "data": {
    "@type": "type.googleapis.com/google.protobuf.Value",
    "value": 3
  },
  "correlation_ids": [
    "gs:conn:01F1KQB9P39CJHN4HN0HN5DMBW",
    "gs:tx_ack:01F1KVPY6K6HBKFMGD8M3FXBQD"
  ],
  "origin": "ip-10-102-6-231.ap-southeast-2.compute.internal",
  "context": {
    "tenant-id": "CgN0dG4="
  },
  "visibility": {
    "rights": [
      "RIGHT_GATEWAY_TRAFFIC_READ"
    ]
  },
  "unique_id": "01F1KVPY6K5RMWNQPY4X9F1DQB"
}

It seems like you see this when there isn’t a recent uplink with GPS timing information in it - so it seems the network can only get that from actual uplinks and not stats messages. I’m not entirely surprised; in general stats messages seem to be given little weight.

When running these tests, sending unconfirmed uplink messages, I sometimes see the gateway console reporting “Drop uplink message” and “Transmit downlink message failure” The uplink messages reach the TTN Device console OK.

As I think was explained before, this happens when the transmit request implementing the downlink doesn’t reach the gateway until after the time at which it was supposed to transmit.

Perhaps you have unreliable Internet connectivity between your gateway and the TTN servers? That could also be making the GPS times in the uplinks look “too erratic” if they’re suffering substantial delay.

I looked at what the packet forwarder messages contain with and without GPS. For the “stat” messages the only difference is the “lat”, “long” and “alti” fields which are absent without GPS. In both cases the “stat” messages contain “time” fields which I guess must come from somewhere else in the absence of GPS.

In the case of the “rxpk” messages, they always contain “tmst” fields. With GPS they also contain “tmms” and “time” fields.

According to PROTOCOL.TXT, the “tmms” field is “GPS time of pkt” and is ONLY expected in the “rxpk” messages, never in the “stat” messages. So it seems to me that the gateway is providing TTN with all that it can, in both stat and rxpk messages. AFAIK it is a completely standard packet forwarder, and it does get a GPS fix, so I can’t see a problem at my end.

As I asked before, IMHO the most important next step is to find out why TTN’s servers report a problem: what is the algorithm? (is it open-source?).

Also, IMHO, the server should not withold the downlink message. If it puts the right timestamp in it, it can rely on the gateway to deliver that to my device at the correct ping slot. My device has already synchronised with the beacon. Should work…

As far as the internet connection is concerned, I have a reliable connection, though it is true that (a) I am using a cellphone (GSM) comms connection from the gateway to the Internet, and (b) I am in New Zealand and TTN’s servers are in Australia (I assume). But that is a quite reasonable use case, and I would have thought that TTN should be able to work with that kind of latency…

The back-end code for V3 is open source and has been so from the start. (As was most of V2 and predecessors)

If it thinks your gateway’s GPS fix is wrong, then, no, it most definitely needs to witthold the request, transmitting a timed downlink at even slightly the wrong time would be quite problematic.

As far as the internet connection is concerned, I have a reliable connection, though it is true that (a) I am using a cellphone (GSM) comms connection from the gateway to the Internet

You’ve posted error messages that show a packet not getting to the gateway until after the time when it needed to be transmitted, suggesting that there’s some substantial Internet delay occurring at times.

While I don’t know for a fact that there’s such a check, if there is being delay on the uplink side too, then beyond a reasonable allowance this could make the reported GPS time look wrong to the server, since the timestamps would be too stale to look current and accurate.

I have looked at the source and can see code that generates the errors. However there is no way I can trace the actual path through the code in my case.

The errors come from the file io.go. The “gateway time is not GPS synchronized” error comes via IsGatewayTimeSynced() in scheduler.go and seems to report that “synced” in clock.go is false. The “settings empty” error seems associated with Rx1 and Rx2 downlink frequencies, which are not applicable to Class B downlink messages, AFAIK.

It looks like synced can be set by a call to either Sync() or SyncWithGatewayConcentrator(). There is a HandleUp() in io.go that calls SyncWithGatewayAbsolute() that calls Sync() that should set “synced”. (It could call Sync() directly).

SyncWithGatewayConcentrator() says “synchronizes the clock using the given token. If the given token does not provide enough information or if the latest
clock sync is more recent, this method returns false”. There seems to be 4 reasons why it might fail…

Elsewhere, there is a comment “uplinkToken is always present with class A downlink, but may be nil otherwise”

There is no code that I can see that tests for “stability” or “quality” of the timestamps in the “rxpk” or “stat” messages.

I wonder:
(a) Has anyone managed to do what I am doing, without error? i.e. has this code been tested by sending Class B downlink messages through the console?
(b) Is there anyone with an understanding of the code in these files who can propose the failure mechanism that I get?
© How might I test Class B ping slot downlink messages, other than through the console?

I have the problem too, When I test my class B device, I downlink a message in the device console, The NS server can’t downlink the message to my gateway, and report an error “Gateway time is not GPS synchronized”, I do not know how to resolve the problem.

imho class B is not supported by TTS CE in the moment. Only class A and class C.

Thank you! I’m using V3 version of TTS, and connected to nam1.cloud.thethings.network server.