Getting "LORA: TXPKT failed, too late!" in TTN Gateway's serial output

TTN Network and Services v2 Backend
1

This is merely to keep some insights from Slack for future reference, as Slack only shows the last 10,000 messages for free accounts.

Tonight, at 18:35 UTC, I got the following error in my TTN Gateway’s serial output, for the EU-region:

LORA: TXPKT failed, too late! tx: 230388 rx: 4293197684

Since July 15th, my logs recorded this error 189 times. Before that date, I’ve never seen it. (I installed the 1.0.5 firmware on July 17th; on July 15th it was still on 1.0.3.)

It seems there are two reasons for this error:

  1. Values like above seem to be caused by some unhandled integer overflow, rather than the packet really being too late. The maximum value of an unsigned integer is 4,294,967,295, so the above rx value is just below that maximum. I assume 230,388 should be interpreted as 4,294,967,295 + 230,388 = 4,295,197,683, hence leaving 4,295,197,683 - 4,293,197,684 = 1,999,999 microseconds or 1.99 seconds between rx and tx.

    In my old logs I see two other occurrences like that:

    [2018-08-12T15:19:18.062Z] LORA: TXPKT failed, too late! tx: 283227 rx: 4294250523
...
[2018-08-29T04:16:59.029Z] LORA: TXPKT failed, too late! tx: 263659 rx: 4294230955

    …which might both translate to 0.99 seconds? Reported as GitHub issue #53.

  2. The gateway truly receiving the downlink too late to be able to transmit it. Indeed the differences between tx and rx are not always as huge as above, and the delay is then nicely explained by @htdvisser’s reply below.

Sometimes, a few downlinks are okay, and then the error occurs again, like back in August:

[2018-08-26T15:17:29.167Z] MQTT: Received DOWNLINK
[2018-08-26T15:18:23.033Z] MQTT: Received DOWNLINK
[2018-08-26T15:31:32.876Z] MQTT: Received DOWNLINK
[2018-08-26T15:32:33.776Z] MQTT: Received DOWNLINK
[2018-08-26T15:32:33.781Z] LORA: TXPKT failed, too late! tx: 1290092140 rx: 1297861076
[2018-08-26T15:33:36.764Z] MQTT: Received DOWNLINK
[2018-08-26T15:34:41.766Z] MQTT: Received DOWNLINK
[2018-08-26T15:34:41.771Z] LORA: TXPKT failed, too late! tx: 1410113468 rx: 1411817244
[2018-08-26T15:35:48.359Z] MQTT: Received DOWNLINK
[2018-08-26T15:36:43.854Z] MQTT: Received DOWNLINK
[2018-08-26T15:36:43.858Z] LORA: TXPKT failed, too late! tx: 1530134820 rx: 1540630348
[2018-08-26T15:37:51.856Z] MQTT: Received DOWNLINK
[2018-08-26T15:37:51.860Z] LORA: TXPKT failed, too late! tx: 1590145500 rx: 1606155300
[2018-08-26T15:39:00.260Z] MQTT: Received DOWNLINK
[2018-08-26T15:39:00.265Z] LORA: TXPKT failed, too late! tx: 1650156148 rx: 1679068316

After reporting a few more of those in Slack’s #ops channel, on August 27th @htdvisser wrote:

We’ve had an unusual amount of high-latency periods on the MQTT bridge yesterday. Those typically resolve themselves, and otherwise the bridge should restart itself.

We do not have a fix for this yet, although I have an experiment/sideproject that may be helpful in resolving this

It’s not our highest priority though. This doesn’t happen often enough to become a real problem, and if it happens, it’s usually resolved within minutes

Here’s the Output/Input graph of our MQTT server in front of the EU bridge from yesterday morning until this morning (times are UTC). On the MQTT servers in front of the bridges this metric is typically around 1, meaning that every message that is pushed (by the gateways) to the MQTT server is subsequently pulled (by the bridges). A drop below 1 means that the gateways keep sending uplink, but the bridges can’t keep up anymore (because of those delays).

image
image3136×2448 297 KB

zoom image

All occurrences for my gateway so far

(Timestamps added by my own logging.)

[2018-07-15T11:40:54.716Z] LORA: TXPKT failed, too late! tx: 872588956 rx: 880091700
[2018-07-15T11:45:23.121Z] LORA: TXPKT failed, too late! tx: 1112599499 rx: 1154594316
[2018-07-15T11:46:50.911Z] LORA: TXPKT failed, too late! tx: 1172602147 rx: 1240591252
[2018-07-15T14:52:19.519Z] LORA: TXPKT failed, too late! tx: 3683181700 rx: 3742184347
[2018-07-15T14:53:23.622Z] LORA: TXPKT failed, too late! tx: 3743184347 rx: 3808155844
[2018-07-15T14:54:10.129Z] LORA: TXPKT failed, too late! tx: 3803186996 rx: 3862190563
[2018-07-15T15:52:54.222Z] LORA: TXPKT failed, too late! tx: 3108386004 rx: 3117392259
[2018-07-15T15:54:51.821Z] LORA: TXPKT failed, too late! tx: 3228391275 rx: 3237658404
[2018-07-15T15:56:53.522Z] LORA: TXPKT failed, too late! tx: 3348396563 rx: 3357626316
[2018-07-15T15:57:50.821Z] LORA: TXPKT failed, too late! tx: 3408399195 rx: 3416384660
[2018-07-15T18:37:35.842Z] LORA: TXPKT failed, too late! tx: 63934763 rx: 106927196
[2018-07-20T23:00:06.315Z] LORA: TXPKT failed, too late! tx: 3998565739 rx: 4040192604
[2018-07-20T23:01:18.518Z] LORA: TXPKT failed, too late! tx: 4059568396 rx: 4117571043
[2018-07-20T23:02:16.821Z] LORA: TXPKT failed, too late! tx: 4118571043 rx: 4177573676
[2018-07-20T23:03:33.336Z] LORA: TXPKT failed, too late! tx: 4179573676 rx: 4237576339
[2018-07-20T23:05:58.521Z] LORA: TXPKT failed, too late! tx: 3611611 rx: 75118244
[2018-07-20T23:07:01.815Z] LORA: TXPKT failed, too late! tx: 63614315 rx: 148570972
[2018-07-20T23:08:02.820Z] LORA: TXPKT failed, too late! tx: 123616971 rx: 182619595
[2018-07-20T23:09:23.920Z] LORA: TXPKT failed, too late! tx: 183619595 rx: 302624884
[2018-07-21T02:55:21.416Z] LORA: TXPKT failed, too late! tx: 3689046243 rx: 3689065804
[2018-07-21T04:23:24.732Z] LORA: TXPKT failed, too late! tx: 379350003 rx: 382479492
[2018-07-21T04:24:23.020Z] LORA: TXPKT failed, too late! tx: 439352715 rx: 439875948
[2018-07-21T05:03:33.532Z] LORA: TXPKT failed, too late! tx: 2779456747 rx: 2790997452
[2018-07-21T05:05:26.742Z] LORA: TXPKT failed, too late! tx: 2899461987 rx: 2904359427
[2018-07-22T00:15:07.720Z] LORA: TXPKT failed, too late! tx: 2128936028 rx: 2131463452
[2018-07-24T10:59:18.527Z] LORA: TXPKT failed, too late! tx: 1974151747 rx: 1977111691
[2018-08-01T04:57:18.417Z] LORA: TXPKT failed, too late! tx: 679407500 rx: 679814804
[2018-08-01T19:53:22.132Z] LORA: TXPKT failed, too late! tx: 2223716531 rx: 2223883348
[2018-08-01T19:55:24.525Z] LORA: TXPKT failed, too late! tx: 2343721851 rx: 2344582428
[2018-08-01T19:56:24.525Z] LORA: TXPKT failed, too late! tx: 2403724451 rx: 2403989499
[2018-08-05T23:13:56.419Z] LORA: TXPKT failed, too late! tx: 1426168603 rx: 1430158100
[2018-08-06T08:17:36.627Z] LORA: TXPKT failed, too late! tx: 115470307 rx: 534489827
[2018-08-08T04:57:02.078Z] LORA: TXPKT failed, too late! tx: 1515502491 rx: 1516161020
[2018-08-08T04:58:02.681Z] LORA: TXPKT failed, too late! tx: 1575505139 rx: 1575743308
[2018-08-11T13:03:25.163Z] LORA: TXPKT failed, too late! tx: 3764753211 rx: 3823755947
[2018-08-11T14:24:24.283Z] LORA: TXPKT failed, too late! tx: 95043811 rx: 95278068
[2018-08-11T14:26:42.863Z] LORA: TXPKT failed, too late! tx: 215049139 rx: 241001948
[2018-08-11T14:27:45.874Z] LORA: TXPKT failed, too late! tx: 275051779 rx: 288272996
[2018-08-11T14:28:49.878Z] LORA: TXPKT failed, too late! tx: 335054411 rx: 360784724
[2018-08-11T14:30:59.976Z] LORA: TXPKT failed, too late! tx: 455059771 rx: 489289444
[2018-08-11T14:32:15.764Z] LORA: TXPKT failed, too late! tx: 516062443 rx: 574065123
[2018-08-11T14:33:21.979Z] LORA: TXPKT failed, too late! tx: 575065123 rx: 634067779
[2018-08-11T14:34:20.865Z] LORA: TXPKT failed, too late! tx: 635067779 rx: 694070419
[2018-08-11T14:35:32.670Z] LORA: TXPKT failed, too late! tx: 695070419 rx: 757558012
[2018-08-11T14:36:55.762Z] LORA: TXPKT failed, too late! tx: 755073107 rx: 821186804
[2018-08-12T01:26:58.770Z] LORA: TXPKT failed, too late! tx: 3374866115 rx: 3416005779
[2018-08-12T03:30:55.270Z] LORA: TXPKT failed, too late! tx: 2225278411 rx: 2262735867
[2018-08-12T15:19:18.062Z] LORA: TXPKT failed, too late! tx: 283227 rx: 4294250523
[2018-08-16T04:10:36.909Z] LORA: TXPKT failed, too late! tx: 7625739 rx: 10065051
[2018-08-16T14:22:38.565Z] LORA: TXPKT failed, too late! tx: 338450091 rx: 338966195
[2018-08-16T14:31:35.163Z] LORA: TXPKT failed, too late! tx: 878475075 rx: 878647676
[2018-08-16T14:53:05.763Z] LORA: TXPKT failed, too late! tx: 2138533963 rx: 2163983396
[2018-08-16T14:54:02.355Z] LORA: TXPKT failed, too late! tx: 2198536627 rx: 2212317820
[2018-08-16T14:55:17.367Z] LORA: TXPKT failed, too late! tx: 2258539219 rx: 2284314148
[2018-08-16T14:57:12.466Z] LORA: TXPKT failed, too late! tx: 2378544595 rx: 2412487004
[2018-08-16T14:58:37.777Z] LORA: TXPKT failed, too late! tx: 2438547267 rx: 2497549939
[2018-08-16T18:48:41.488Z] LORA: TXPKT failed, too late! tx: 2739431843 rx: 2742588324
[2018-08-21T10:22:59.777Z] LORA: TXPKT failed, too late! tx: 1728155955 rx: 1732905059
[2018-08-23T15:27:07.063Z] LORA: TXPKT failed, too late! tx: 4159550723 rx: 4162364740
[2018-08-24T02:28:13.562Z] LORA: TXPKT failed, too late! tx: 2864670811 rx: 2869707116
[2018-08-24T06:26:08.662Z] LORA: TXPKT failed, too late! tx: 4260439011 rx: 4260909068
[2018-08-24T14:36:10.680Z] LORA: TXPKT failed, too late! tx: 2678564579 rx: 2680664716
[2018-08-24T14:37:12.497Z] LORA: TXPKT failed, too late! tx: 2738567251 rx: 2740742955
[2018-08-24T22:10:10.970Z] LORA: TXPKT failed, too late! tx: 1238508843 rx: 1240691676
[2018-08-25T09:38:19.882Z] LORA: TXPKT failed, too late! tx: 2103638363 rx: 2113713387
[2018-08-25T21:45:39.278Z] LORA: TXPKT failed, too late! tx: 578411324 rx: 595342588
[2018-08-25T21:47:27.883Z] LORA: TXPKT failed, too late! tx: 698416676 rx: 702129451
[2018-08-26T08:16:58.863Z] LORA: TXPKT failed, too late! tx: 1249765203 rx: 1250257188
[2018-08-26T08:19:39.164Z] LORA: TXPKT failed, too late! tx: 1369770531 rx: 1443013676
[2018-08-26T08:47:33.563Z] LORA: TXPKT failed, too late! tx: 3109851028 rx: 3111440340
[2018-08-26T08:49:28.359Z] LORA: TXPKT failed, too late! tx: 3229856372 rx: 3232131948
[2018-08-26T08:50:24.960Z] LORA: TXPKT failed, too late! tx: 3289859043 rx: 3292810483
[2018-08-26T08:58:47.769Z] LORA: TXPKT failed, too late! tx: 3769880435 rx: 3792100172
[2018-08-26T09:14:12.458Z] LORA: TXPKT failed, too late! tx: 374954187 rx: 405418828
[2018-08-26T09:15:22.288Z] LORA: TXPKT failed, too late! tx: 434956891 rx: 493959556
[2018-08-26T09:16:09.804Z] LORA: TXPKT failed, too late! tx: 494959556 rx: 533437444
[2018-08-26T09:17:19.178Z] LORA: TXPKT failed, too late! tx: 554962251 rx: 597392556
[2018-08-26T09:32:01.974Z] LORA: TXPKT failed, too late! tx: 1455003331 rx: 1466422884
[2018-08-26T10:04:34.563Z] LORA: TXPKT failed, too late! tx: 3435096491 rx: 3443950827
[2018-08-26T10:05:38.484Z] LORA: TXPKT failed, too late! tx: 3495099220 rx: 3508043356
[2018-08-26T10:08:57.691Z] LORA: TXPKT failed, too late! tx: 3675107227 rx: 3709051827
[2018-08-26T10:09:58.762Z] LORA: TXPKT failed, too late! tx: 3735109908 rx: 3746348364
[2018-08-26T10:12:25.768Z] LORA: TXPKT failed, too late! tx: 3855115243 rx: 3914118859
[2018-08-26T10:13:45.767Z] LORA: TXPKT failed, too late! tx: 3915118859 rx: 3987923348
[2018-08-26T10:17:49.264Z] LORA: TXPKT failed, too late! tx: 4215132292 rx: 4239685396
[2018-08-26T10:18:49.815Z] LORA: TXPKT failed, too late! tx: 4275134955 rx: 4277983644
[2018-08-26T10:20:14.065Z] LORA: TXPKT failed, too late! tx: 40170324 rx: 89503652
[2018-08-26T10:21:30.465Z] LORA: TXPKT failed, too late! tx: 100172947 rx: 159175683
[2018-08-26T10:23:44.066Z] LORA: TXPKT failed, too late! tx: 220178331 rx: 287250348
[2018-08-26T10:24:40.282Z] LORA: TXPKT failed, too late! tx: 280181019 rx: 352600043
[2018-08-26T10:25:26.375Z] LORA: TXPKT failed, too late! tx: 340183691 rx: 399186364
[2018-08-26T10:26:42.862Z] LORA: TXPKT failed, too late! tx: 400186364 rx: 480574652
[2018-08-26T10:31:33.360Z] LORA: TXPKT failed, too late! tx: 760269620 rx: 770267028
[2018-08-26T10:33:36.664Z] LORA: TXPKT failed, too late! tx: 880290956 rx: 890198380
[2018-08-26T10:36:04.467Z] LORA: TXPKT failed, too late! tx: 1000312308 rx: 1042484420
[2018-08-26T10:37:28.060Z] LORA: TXPKT failed, too late! tx: 1060322924 rx: 1119333652
[2018-08-26T10:53:26.261Z] LORA: TXPKT failed, too late! tx: 2080477332 rx: 2083260876
[2018-08-26T10:54:25.779Z] LORA: TXPKT failed, too late! tx: 2140488068 rx: 2141506827
[2018-08-26T11:24:54.463Z] LORA: TXPKT failed, too late! tx: 3940763196 rx: 3959188452
[2018-08-26T13:28:39.461Z] LORA: TXPKT failed, too late! tx: 2438784428 rx: 2443640276
[2018-08-26T13:34:04.581Z] LORA: TXPKT failed, too late! tx: 2738837812 rx: 2750021156
[2018-08-26T13:51:07.885Z] LORA: TXPKT failed, too late! tx: 3759001252 rx: 3801093259
[2018-08-26T13:53:07.878Z] LORA: TXPKT failed, too late! tx: 3879022596 rx: 3885079564
[2018-08-26T13:58:59.585Z] LORA: TXPKT failed, too late! tx: 4239086620 rx: 4272320164
[2018-08-26T14:00:29.969Z] LORA: TXPKT failed, too late! tx: 4130004 rx: 63140684
[2018-08-26T14:01:58.862Z] LORA: TXPKT failed, too late! tx: 64140684 rx: 123151292
[2018-08-26T14:23:33.265Z] LORA: TXPKT failed, too late! tx: 1444359116 rx: 1453044156
[2018-08-26T14:25:45.262Z] LORA: TXPKT failed, too late! tx: 1564380460 rx: 1581252300
[2018-08-26T14:28:01.482Z] LORA: TXPKT failed, too late! tx: 1684392812 rx: 1718417892
[2018-08-26T14:31:35.973Z] LORA: TXPKT failed, too late! tx: 1864424820 rx: 1923435492
[2018-08-26T14:33:00.885Z] LORA: TXPKT failed, too late! tx: 1924435492 rx: 1986251428
[2018-08-26T14:53:00.288Z] LORA: TXPKT failed, too late! tx: 3184650588 rx: 3199524884
[2018-08-26T14:55:03.368Z] LORA: TXPKT failed, too late! tx: 3304671940 rx: 3312984596
[2018-08-26T14:56:27.681Z] LORA: TXPKT failed, too late! tx: 3424693276 rx: 3427403339
[2018-08-26T15:10:51.279Z] LORA: TXPKT failed, too late! tx: 4264833676 rx: 4290654788
[2018-08-26T15:14:18.274Z] LORA: TXPKT failed, too late! tx: 149898396 rx: 200155228
[2018-08-26T15:15:33.181Z] LORA: TXPKT failed, too late! tx: 209909060 rx: 273532748
[2018-08-26T15:16:46.868Z] LORA: TXPKT failed, too late! tx: 269919964 rx: 346757748
[2018-08-26T15:32:33.781Z] LORA: TXPKT failed, too late! tx: 1290092140 rx: 1297861076
[2018-08-26T15:34:41.771Z] LORA: TXPKT failed, too late! tx: 1410113468 rx: 1411817244
[2018-08-26T15:36:43.858Z] LORA: TXPKT failed, too late! tx: 1530134820 rx: 1540630348
[2018-08-26T15:37:51.860Z] LORA: TXPKT failed, too late! tx: 1590145500 rx: 1606155300
[2018-08-26T15:39:00.265Z] LORA: TXPKT failed, too late! tx: 1650156148 rx: 1679068316
[2018-08-26T15:41:07.167Z] LORA: TXPKT failed, too late! tx: 1770177484 rx: 1807019356
[2018-08-26T15:42:14.666Z] LORA: TXPKT failed, too late! tx: 1830179124 rx: 1870227932
[2018-08-26T15:43:24.279Z] LORA: TXPKT failed, too late! tx: 1890189836 rx: 1949200508
[2018-08-26T15:44:50.174Z] LORA: TXPKT failed, too late! tx: 1951200508 rx: 2009211188
[2018-08-26T16:51:42.961Z] LORA: TXPKT failed, too late! tx: 1058575100 rx: 1072458139
[2018-08-26T16:52:50.659Z] LORA: TXPKT failed, too late! tx: 1119585764 rx: 1143744348
[2018-08-26T16:54:02.560Z] LORA: TXPKT failed, too late! tx: 1178596452 rx: 1215515835
[2018-08-26T16:56:07.969Z] LORA: TXPKT failed, too late! tx: 1299617796 rx: 1337943515
[2018-08-26T16:56:59.761Z] LORA: TXPKT failed, too late! tx: 1358628444 rx: 1377059932
[2018-08-26T16:58:23.886Z] LORA: TXPKT failed, too late! tx: 1418639132 rx: 1477640764
[2018-08-26T16:59:19.169Z] LORA: TXPKT failed, too late! tx: 1478640764 rx: 1489412988
[2018-08-26T17:18:34.662Z] LORA: TXPKT failed, too late! tx: 818537300 rx: 822673684
[2018-08-26T17:19:36.366Z] LORA: TXPKT failed, too late! tx: 878547964 rx: 886363404
[2018-08-27T14:33:44.001Z] LORA: TXPKT failed, too late! tx: 1057529068 rx: 1057832092
[2018-08-27T14:34:48.868Z] LORA: TXPKT failed, too late! tx: 1117539748 rx: 1122657676
[2018-08-27T14:35:49.256Z] LORA: TXPKT failed, too late! tx: 1177550420 rx: 1188006003
[2018-08-27T15:44:55.567Z] LORA: TXPKT failed, too late! tx: 1023238492 rx: 1036271388
[2018-08-27T15:47:01.873Z] LORA: TXPKT failed, too late! tx: 1143250820 rx: 1148933924
[2018-08-27T15:49:01.875Z] LORA: TXPKT failed, too late! tx: 1263272180 rx: 1277973820
[2018-08-27T15:50:05.782Z] LORA: TXPKT failed, too late! tx: 1323273868 rx: 1340391812
[2018-08-27T15:51:11.775Z] LORA: TXPKT failed, too late! tx: 1383284516 rx: 1411996012
[2018-08-27T15:53:15.077Z] LORA: TXPKT failed, too late! tx: 1503305844 rx: 1532264132
[2018-08-27T15:54:09.176Z] LORA: TXPKT failed, too late! tx: 1563307492 rx: 1591999580
[2018-08-27T15:57:29.967Z] LORA: TXPKT failed, too late! tx: 1743339540 rx: 1793733723
[2018-08-27T15:59:02.363Z] LORA: TXPKT failed, too late! tx: 1803349836 rx: 1871876740
[2018-08-27T16:00:24.264Z] LORA: TXPKT failed, too late! tx: 1863350908 rx: 1943928196
[2018-08-27T16:01:17.560Z] LORA: TXPKT failed, too late! tx: 1923361564 rx: 1982372228
[2018-08-27T16:01:55.670Z] LORA: TXPKT failed, too late! tx: 1983372228 rx: 2058346427
[2018-08-27T16:02:51.082Z] LORA: TXPKT failed, too late! tx: 2043373836 rx: 2102384596
[2018-08-27T16:04:20.083Z] LORA: TXPKT failed, too late! tx: 2103384596 rx: 2202578459
[2018-08-27T16:46:52.267Z] LORA: TXPKT failed, too late! tx: 388840180 rx: 456346068
[2018-08-27T16:47:52.462Z] LORA: TXPKT failed, too late! tx: 448850860 rx: 507861532
[2018-08-27T18:15:15.467Z] LORA: TXPKT failed, too late! tx: 1434741732 rx: 1467812083
[2018-08-27T18:16:21.379Z] LORA: TXPKT failed, too late! tx: 1494752412 rx: 1514839116
[2018-08-27T18:46:23.779Z] LORA: TXPKT failed, too late! tx: 3295045564 rx: 3334422300
[2018-08-27T19:52:51.667Z] LORA: TXPKT failed, too late! tx: 3020703292 rx: 3026128956
[2018-08-27T20:36:49.678Z] LORA: TXPKT failed, too late! tx: 1366151492 rx: 1374406803
[2018-08-27T22:24:51.661Z] LORA: TXPKT failed, too late! tx: 3492191052 rx: 3551201724
[2018-08-28T08:28:59.982Z] LORA: TXPKT failed, too late! tx: 1151941691 rx: 1154325620
[2018-08-29T04:16:59.029Z] LORA: TXPKT failed, too late! tx: 263659 rx: 4294230955
[2018-09-01T11:48:14.259Z] LORA: TXPKT failed, too late! tx: 1358469564 rx: 1358711076
[2018-09-03T00:51:28.309Z] LORA: TXPKT failed, too late! tx: 1115043035 rx: 1115633452
[2018-09-03T00:52:55.260Z] LORA: TXPKT failed, too late! tx: 1175045708 rx: 1189867940
[2018-09-03T01:23:49.476Z] LORA: TXPKT failed, too late! tx: 3035130531 rx: 3048924492
[2018-09-03T01:40:47.777Z] LORA: TXPKT failed, too late! tx: 4055178964 rx: 4072017028
[2018-09-03T01:42:45.959Z] LORA: TXPKT failed, too late! tx: 4175184316 rx: 4185801892
[2018-09-03T01:47:10.076Z] LORA: TXPKT failed, too late! tx: 60225035 rx: 157288380
[2018-09-03T02:03:06.962Z] LORA: TXPKT failed, too late! tx: 1080271460 rx: 1102287172
[2018-09-03T02:05:58.884Z] LORA: TXPKT failed, too late! tx: 1200276811 rx: 1259279483
[2018-09-03T02:06:57.277Z] LORA: TXPKT failed, too late! tx: 1260279483 rx: 1344344284
[2018-09-03T02:07:59.868Z] LORA: TXPKT failed, too late! tx: 1320283099 rx: 1409182508
[2018-09-03T02:09:39.980Z] LORA: TXPKT failed, too late! tx: 1380285827 rx: 1499291179
[2018-09-03T02:55:35.415Z] LORA: TXPKT failed, too late! tx: 4260417091 rx: 4263149340
[2018-09-03T02:56:27.968Z] LORA: TXPKT failed, too late! tx: 25452467 rx: 32692444
[2018-09-03T03:47:16.484Z] LORA: TXPKT failed, too late! tx: 3025588083 rx: 3032535044
[2018-09-03T03:48:39.777Z] LORA: TXPKT failed, too late! tx: 3085589763 rx: 3160673812
[2018-09-03T03:49:45.664Z] LORA: TXPKT failed, too late! tx: 3145592356 rx: 3222915908
[2018-09-03T04:06:55.973Z] LORA: TXPKT failed, too late! tx: 4225642523 rx: 4244252668
[2018-09-03T04:09:22.478Z] LORA: TXPKT failed, too late! tx: 50680596 rx: 109683243
[2018-09-03T04:10:28.485Z] LORA: TXPKT failed, too late! tx: 110683243 rx: 178055651
[2018-09-03T08:24:06.587Z] LORA: TXPKT failed, too late! tx: 2466483515 rx: 2508219763
[2018-09-03T08:25:33.881Z] LORA: TXPKT failed, too late! tx: 2526486147 rx: 2592846884
[2018-09-03T08:27:25.170Z] LORA: TXPKT failed, too late! tx: 2706494299 rx: 2707265987
[2018-09-03T08:45:12.280Z] LORA: TXPKT failed, too late! tx: 3726540691 rx: 3773832684
[2018-09-03T08:46:33.386Z] LORA: TXPKT failed, too late! tx: 3786543355 rx: 3848426748
[2018-09-26T18:35:51.432Z] LORA: TXPKT failed, too late! tx: 230388 rx: 4293197684