Class C device receives the downlink frame after 2 minutes

We have a custom gateway and only one class C device. The chipsets are SX1302 and 2 SX1250. Sometimes we can see that the gateway has sent out the downlink frame immediately after it receives it from the server, but the end device receives it after ~2 minutes. Below is the log.

Oct 28 07:18:29 imx8mmxxx LoraService.sh[936]: JSON down: {“txpk”:{“imme”:true,“rfch”:0,“powe”:27,“ant”:0,“brd”:0,“freq”:869.525,“modu”:“LORA”,“datr”:“SF12BW125”,“codr”:“4/5”,“ipol”:true,“size”:21,“data”:“YGn9zgCACgIBGJdvBgVtuQrbzyh1”}}
Oct 28 07:18:29 imx8mmxxx LoraService.sh[936]: INFO: [down] a packet will be sent in “immediate” mode
Oct 28 07:18:29 imx8mmxxx LoraService.sh[936]: src/jitqueue.c:136:jit_enqueue(): Current concentrator time is 2613983026, pkt_type=2
Oct 28 07:18:29 imx8mmxxx LoraService.sh[936]: src/jitqueue.c:177:jit_enqueue(): DEBUG: insert IMMEDIATE downlink, first in JiT queue (count_us=2614063026)
Oct 28 07:18:29 imx8mmxxx LoraService.sh[936]: src/jitqueue.c:114:jit_sort_queue(): sorting queue in ascending order packet timestamp - queue size:1
Oct 28 07:18:29 imx8mmxxx LoraService.sh[936]: src/jitqueue.c:116:jit_sort_queue(): sorting queue done - swapped:0
Oct 28 07:18:29 imx8mmxxx LoraService.sh[936]: src/jitqueue.c:444:jit_print_queue(): INFO: [jit] queue contains 1 packets:
Oct 28 07:18:29 imx8mmxxx LoraService.sh[936]: src/jitqueue.c:445:jit_print_queue(): INFO: [jit] queue contains 0 beacons:
Oct 28 07:18:29 imx8mmxxx LoraService.sh[936]: src/jitqueue.c:448:jit_print_queue(): - node[0]: count_us=2614063026 - type=2
Oct 28 07:18:29 imx8mmxxx LoraService.sh[936]: src/jitqueue.c:314:jit_enqueue(): enqueued packet with count_us=2614063026 (size=21 bytes, toa=1483000 us, type=2)
Oct 28 07:18:29 imx8mmxxx LoraService.sh[936]: src/jitqueue.c:424:jit_peek(): peek packet with count_us=2614063026 at index 0
Oct 28 07:18:29 imx8mmxxx LoraService.sh[936]: src/jitqueue.c:440:jit_print_queue(): INFO: [jit] queue is empty
Oct 28 07:18:29 imx8mmxxx LoraService.sh[936]: src/jitqueue.c:358:jit_dequeue(): dequeued packet with count_us=2614063026 from index 0
Oct 28 07:18:29 imx8mmxxx LoraService.sh[936]: src/lora_pkt_fwd.c:3326:thread_jit(): lgw_send done on rf_chain 0: count_us=2614063026
Oct 28 07:18:29 imx8mmxxx LoraService.sh[936]: INFO: [down] PULL_ACK received in 822 ms

Oct 28 07:20:26 imx8mmxxx LoraService.sh[936]: JSON up: {“rxpk”:[{“jver”:1,“tmst”:2750148575,“chan”:3,“rfch”:0,“freq”:867.100000,“mid”: 8,“stat”:1,“modu”:“LORA”,“datr”:“SF7BW125”,“codr”:“4/5”,“rssis”:-39,“lsnr”:13.8,“foff”:3648,“rssi”:-39,“size”:21,“data”:“gGn9zgCASgECE6dSP6U8N6vvAZBJ”}]}

We did also check the end device that it really receive the downlink frame after 2 minutes. Since it is a class C device, and there is only one device in the network, we can’t identify what the problem is. Does anyone give us advice how to debug it? Thanks in advance!

Please format logs with the </> tool to make it easy for volunteers to read.

Nothing you have shown says that the device receives the downlink after 2 minutes.

All it shows is that the device sends back a response two minutes later - which isn’t an unreasonable amount of time to receive a downlink, process it and then put some jitter in to the response so that you don’t end up with all the Class C devices responding at the same time - the fact you have only one no part of the system will know so it won’t know that YOU want it to respond straight away.

I’d suggest you put a serial debug on a device and see what is happening at the device end.

We may be able to comment further if we knew what the device is &/or what firmware it is running.

Thanks for your reply. Sorry for the poor formatting. I will learn how to use the </> tool next time.
The logs are copied from the gateway, and therefore it just shows when it receives the json message from the server and the downlink response from the device.
Sorry for my unclear question. The gateway sends out the downlink to the SX1302 immediately after it receives it from the server. But from the device side, it receives the downlink after two minutes. The problem happens randomly. Oftentimes the device can receive the downlink immediately.
We have another RAK7268 gateway for comparison. The same device can always receive the downlink immediately. The RF chipsets are the same, SC1302 and 2 SX1250.
I have also compared the packet forwarder source code and the config file but only find little difference, which seems not related to the problem. Could you have more advice for me?

Show me the device logs!

Because if every Class C device in an area returned a response at the same time, it would break the gateway, the internet and possibly harm puppies.

If you are up to making your own concentrator board and poking around the source code, you could put in a debug line to show exactly when the transmission is performed rather than then the JSON was received!