"no clock sync" when queuing a class C multicast message

Hello there, I’ve been having an issue since this morning that has been driving me crazy. I’ve looked everywhere, but I can’t find anyone with a similar issue to this one.

I have a class C multicast group set up in TTN, and I’m trying to queue a message to it. However, it always fails like so, constantly retrying to upload the message:

The “Schedule” message contains details about the error. Here’s the event:

{
  "name": "ns.down.data.schedule.fail",
  "time": "2025-03-18T09:08:25.057271779Z",
  "identifiers": [
    {
      "device_ids": {
        "device_id": "multicast-group",
        "application_ids": {
          "application_id": "gie-fuota-app"
        },
        "dev_eui": "70B3D57ED800405A",
        "dev_addr": "DEADBEEF"
      }
    }
  ],
  "data": {
    "@type": "type.googleapis.com/ttn.lorawan.v3.ErrorDetails",
    "namespace": "pkg/gatewayserver",
    "name": "schedule",
    "message_format": "schedule",
    "correlation_id": "7ac4369a39844182a9743feb00524afe",
    "code": 10,
    "details": [
      {
        "@type": "type.googleapis.com/ttn.lorawan.v3.ScheduleDownlinkErrorDetails",
        "path_errors": [
          {
            "namespace": "pkg/gatewayserver",
            "name": "schedule_path",
            "message_format": "schedule on path `{gateway_uid}`",
            "attributes": {
              "gateway_uid": "kona-enterprise-1@giefuota"
            },
            "correlation_id": "cc002b95bcc64c59917ac18f4b44d6aa",
            "cause": {
              "namespace": "pkg/gatewayserver/io",
              "name": "tx_schedule",
              "message_format": "schedule",
              "correlation_id": "82dfb1470c284b4da29317d65c930f9a",
              "code": 10,
              "details": [
                {
                  "@type": "type.googleapis.com/ttn.lorawan.v3.ScheduleDownlinkErrorDetails",
                  "path_errors": [
                    {
                      "namespace": "pkg/gatewayserver/io",
                      "name": "rx_empty",
                      "message_format": "settings empty",
                      "correlation_id": "39374b03b9954850bab3a1c440d925b9",
                      "code": 9
                    },
                    {
                      "namespace": "pkg/gatewayserver/io",
                      "name": "rx_window_schedule",
                      "message_format": "schedule in Rx window `{window}` failed",
                      "attributes": {
                        "window": 2
                      },
                      "correlation_id": "571165e0e9ea4345bdf8d80aec2457a2",
                      "cause": {
                        "namespace": "pkg/gatewayserver/scheduling",
                        "name": "no_clock_sync",
                        "message_format": "no clock sync",
                        "correlation_id": "eb4a23436c3c4c1f84b78351e1a61764",
                        "code": 14
                      },
                      "code": 14
                    }
                  ]
                }
              ]
            },
            "code": 10,
            "details": [
              {
                "@type": "type.googleapis.com/ttn.lorawan.v3.ScheduleDownlinkErrorDetails",
                "path_errors": [
                  {
                    "namespace": "pkg/gatewayserver/io",
                    "name": "rx_empty",
                    "message_format": "settings empty",
                    "correlation_id": "39374b03b9954850bab3a1c440d925b9",
                    "code": 9
                  },
                  {
                    "namespace": "pkg/gatewayserver/io",
                    "name": "rx_window_schedule",
                    "message_format": "schedule in Rx window `{window}` failed",
                    "attributes": {
                      "window": 2
                    },
                    "correlation_id": "571165e0e9ea4345bdf8d80aec2457a2",
                    "cause": {
                      "namespace": "pkg/gatewayserver/scheduling",
                      "name": "no_clock_sync",
                      "message_format": "no clock sync",
                      "correlation_id": "eb4a23436c3c4c1f84b78351e1a61764",
                      "code": 14
                    },
                    "code": 14
                  }
                ]
              }
            ]
          }
        ]
      }
    ]
  },
  "correlation_ids": [
    "as:downlink:01JPM85RMBJRXDPR9Z91T2X3KC",
    "ns:transmission:01JPM85RV0NVKRMWND9RVAT9YK"
  ],
  "origin": "ip-10-23-4-101.eu-west-1.compute.internal",
  "context": {
    "tenant-id": "CghnaWVmdW90YQ=="
  },
  "visibility": {
    "rights": [
      "RIGHT_APPLICATION_TRAFFIC_READ"
    ]
  },
  "unique_id": "01JPM85RV14W94CJ3WBHTGHYKA"
}

The only thing I was able to find about this issue was this code which I’m guessing is the source for the TTN message server. If so, then I understand the following:

  • The rx_empty error makes sense because it cannot use the rx1 window as it is class C only, so it is probably misconfigured and has its frequency set to 0
  • The rx_window_schedule error, now, I have no clue. The cause is a “no_clock_sync” exception, but looking at the code it’s not clear to me what this clock means, why it needs to be synchronized, or how to synchronize it.

Here’s more background info in case it is useful:

I’m new to the world of LoRa and radio communication in general so any information merely tangential to the issue would be greatly appreciated. And if you’ve read thus far - thank you for your time :slight_smile:

Alejandro

I think the first thing to try is to move your device off of the “Europe 868.1 MHz” plan, as that is a single channel only frequency plan, which is rather experimental and definitely not one you would want to use in production deployments.
“Europe 863-870 MHz (SF9 for RX2 - recommended)” is really the recommended channel plan here. I’m not fully aware of the consequences of channel plan on Class C behaviour, but it may very well help selecting the recommended plan.

1 Like

Whilst potentially hugely amusing to some to set up the DevAddr to DE AD BE EF, if you are new to LoRaWAN, why not keep things simple by leaving the defaults alone.

I can count on the fingers of one hand how many people I know who use Class C - it’s an interesting extension but the vast majority start out with good old fashioned Class A before finding all the corner cases of use.

The hijack of the DevAddr, the value range which is assigned to TTI by the LoRa Alliance, that is, it’s an official number that has significance, indicates you may benefit from some additional info - the Learn section is super useful: https://www.thethingsnetwork.org/docs/lorawan/

Good spot! Reinforces the idea of going with the normal (recommended) before trying the special!

Hopefully the gateway is on the recommended channel plan as well.

1 Like

Thank you for your quick replies!

I had no clue – That might answer a lot of the issues I have had for the past months with antennas sometimes not receiving or queued messages being silently ignored. I’ll try changing it and see what happens.

Right, I completely forgot to mention – I’m working with custom firmware here, creating my own device. Been doing so for the past few months. As such, I don’t think there was a default value in this context. I just chose an arbitrary DevAddr - which I assumed to be the multicast address since TTN doesn’t appear to have a “multicast address” setting on multicast group “devices”. If there is a limitation set by the standard on the addresses available for multicast usage though, please let me know.

It’s set to 868.1MHz as well, but I’ll try changing it to the one @stevencellist mentioned.

Does this include writing your own LoRaWAN stack? :thinking: Or do you use a readily available one (in which case, which one)?

I’ve yet to find time to debug the 868.1 MHz channel plan, but some users/stacks make assumptions of things that are common - having just one channel definitely is not a common thing.

We’re using one done by a company named Stackforce, using the version 1.0.4 of the standard.

To be honest I get pretty lost when getting closer to the physical layer of the protocol. I chose 868.1 MHz only because we were previously using it on another setup with another server. I should probably finish reading the LoRaWAN introduction book by Sylvain Montagny at some point…

By the way - I just tried changing the channel plan and it started working again, thank you! I have yet to do intensive testing since I had a couple other issues before this one but I hope that those will be gone with the new frequency configuration.

There isn’t, but there is a Generate button and the DevAddr ranges are assigned by the LoRa Alliance as they are fundamental part of the uplink matching process. It’s not fatal to the system to create your own but it does make a mockery of the whole allocation system AND doesn’t help with many aspects of debugging, plus you may have used someone else’s prefix, but if you bump around the console setting things, it’s best to know what the ramifications are.

Because the web console exposes pretty much every option you can have, it’s like learning to drive by getting in to a multi-engined heavy goods vehicle with refrigeration when all you need is a speedometer.

Everyone uses custom firmware in so much as they have to add their own sensor / processing code. Only crazy Dutch CS students with time on their hands create stacks.

It should not be the only thing you read. The most important is the learn section, linked above, it covers the essentials. Then there is watching Johan’s video, one chunk at a time. If you are writing firmware, then having the specification to hand is good. TTN/TTI have been doing LoRaWAN since the beginning of LoRaWAN so their docs contain a wealth of information. As does this forum which you can search directly or ask Google to search it for you - because this is where most of the debugging has been done since the start.

Noted. I took a look at the DevAddr assignment section of the Backend Interfaces standard and see what you mean now; I’ve changed the address for an auto-generated one instead.

Yes, I’m guilty of trying to get things to work without reading the underlying theory first - Since we had a tight deadline I succumbed to only reading the standard for the parts I needed which left a lot of gaps in my knowledge. I’ve dedicated this morning to watching and taking notes from the Johan Stokking presentation. Thanks for sharing it; it’s a great video and I think it’ll be really useful for the rest of my tasks. I’ll probably spend the rest of today or tomorrow reading the remaining contents of the learn page.

2 Likes

If the deadline for getting Class C multicast working is mid-summer, you should be OK.

LW is not WiFi, it’s not got much any warm & fuzzy wrapping.

Feel free to outline what you are trying to achieve, including why LW, and perhaps we can outline a path to get the job working sooner rather than later.

Thank you very much for your support. I’ll try my best to explain the situation concisely.

We’re working on a project that has LoRa as a requisite. We’re using it mostly because of its low energy consumption and range characteristics. We already got class A working, however the project requires us to implement class C & B as well. The meters need to have some form of FUOTA implementation as well, for which we had planned to use class C.

I’m currently working on the FUOTA implementation. We’ve been basing our work on the TR002 technical recommendation. Stackforce provides an application layer specifically for FUOTA that adds support for the Remote Multicast Setup, Clock Sync & Fragmented Data Block Transport specifications so we’re mostly covered on firmware side.
As for the FUOTA server side of things, I might’ve gone overboard, but I ended up building one from scratch, using TTN as the network server and a webhook integration to connect both.
The FUOTA process needs a multicast session to be set up so I created a multicast group in TTN and calculated the keys using a small script I wrote based on Sylvain Montagny’s advanced LoRaWAN book.
I’ve got everything related to FUOTA working on the end device now: Class C sessions being correctly set up, multicast messages being received without a MIC mismatch, etc.

The issue that I’ve been having for a few weeks now is that some multicast packets aren’t being sent, but as far as I know they aren’t being marked as “failed” either. They are queuing because I am getting the relevant webhook event and can see them on the multicast group log in the TTN console, but they aren’t being sent because I don’t get any send events on the webhook and see no activity on the end device either. If it helps, it seems to be more common when the queue is full and the gateway tries to send the messages ASAP. If I space the queuing out the issue seems to be a bit less prominent, but still happens enough to be problematic (around 1/6ths of the packets don’t get sent). We disabled the duty cycle limitations for testing if it was an issue with that but doesn’t seem to help. The nature of the error indicates that it is somewhere between the gateway and my server but I can’t tell exactly where.

Erm, please dont - no matter what the reason/motivation! Unless working in a proven Faraday cage (BTW When doing sensitivity tests on LoRa RF in the early days it was found that a single FC for the DUT wasn’t enough and the FC had to be put inside another FC for accurate sensitivity charaterisation!). If DC is a legaslative requirement in your jurasdiction then it’s an absolute otherwise… :police_car: :policeman: :judge: !

If you think that is an issue (and throughput may be a problem if issue presents more when Queues are filling up) then focus on trying to slow system down wrt throughput until reliable then look to optimise and speed up as and where you can. Remember the GW’s, as a radiating device, have their own DC limits and may (should/will!) throttle message when over capacity. Solution there is usually network densification so you have more available GW’s for any given end device deployment.

If you are working on a metering application/deployment then network densification is usually a must anyhow - helps limit time on air and improve battery life of deployed devices by ratcheting down SF and as then clearly a commercial venture recommend you engage properly with TTI core team and get implementation support (calling @rish1 ).

I understand this, but shouldn’t TTN be notifying of a failed send when reaching the DC limit, or delaying the message until the duty cycle requirements are met? I’m not exactly sure of the behavior it has in this context.

Not into the nuts and bolts of source behaviour (have you read source and investigated?) which is where my comment wrt TTI support comes into play (was editing message so we crossed!), perhaps yes but behaviour will likely be complex and will depend on all traffic underway and how packed the queues are (I suspect), wrt delaying until DC clear - how long a delay is reasonable and what can the application/deployment tolerate? Again how long may be impacted by other traffic being handled… Messages would need to be dropped/unqueued at some point I guess.

If you hammer the shared community provided-for-free servers by breaching the FUP, you’ll mostly get frosty stares from us - the community (aka non-commercial) applications for good causes rely on TTI being able to fund the servers - they can’t afford for TTI to have to close things down because the infrastructure has been over stressed. Please ensure you have read the FUP & are working to it while you move to a TTI instance.

Why? Meter reading is simple, put the effort in to testing so you don’t have to FUOTA!

You also need to test the entire FUOTA setup to within an inch of its life because the best way to brick a device is to have it all go horribly wrong. This includes the UI & human factors - part sending an update & then changing the firmware and finding out that the device has managed to piece together two different sets and fallen over makes for a very stressful meeting!

Going from Zero to FUOTA on LW is a tall order, if I was your PM, I’d ensure that Class A firmware is rock solid, get that tested in volume whilst the rest gets done. For FUOTA level you need to be able to quote the spec without looking at it yet you’ve yet to read some booklet from a university - which has lots of good info in it but has no commercial experience baked in.

The other reason for doing the learning first is that some of the protocols aren’t supported by TTS but TTI has a working server & device provided as a reference.

The WebHook can return detailed messages when a downlink fails so capturing those may assist.

One of the benefits of taking up a paid for instance is you get support from TTI engineers, so I’m sure @rish1 will be happy to advise.

@aperea And if its the one I think you are refering to it has some poor and challenged starting points as much discussed on the forum some years back when it 1st emerged - may have updated since then (Forum Search is your friend!).