MatchX no TX

[kersing (Jac Kersing)]

@jmarcelino

Which gateways really don’t have enough memory to queue sufficient downlinks?

Every gateway running older packet forwarders like the stock basic/gps forwarder or early versions of poly forwarder because those versions of the software do not implement JIT queuing and send every packet received from the back-end to hardware which has no queue. So the last packet to arrive would be transmitted and earlier packets lost.

@jmarcelino

Sure application downlinks have tight scheduling constraints if they are to meet RX1

I’ve been told there should be a feature request allowing to specify the delay for RX1 for a node. Hopefully that will be implemented in the new back-end because the current window is far too small for any useful application processing (when combined with all other delays)

@jmarcelino

I’m very curious to see how a “fresh” TTN protocol implementation like the one on TTN’s own gateway reacts.

They suffer the same delay last time I checked, the reason I was told is the back-end does not distinguish between the two types of gateways. (semtech or ttn-gateway-connector protocol)

[arjanvanb (Arjan)]

@Verkehrsrot

On Twitter they guess that the 4,58 seconds downlink delay of TTN backend is deliberately to close out certain brands/types of gateways.

Wow, they are really hostile on Twitter. For me, a supplier accusing others in the open and then asking people to send an email if they want proof, would quickly drop from my possible supplier list, but well…

(This forum shows timing has always been tedious with TTN and now MatchX claims “a sudden deceitful 4.6s change”?)

[Verkehrsrot]

yes, the communication is terrible. I don’t know why the react in this assaulting manner. But the people behind there seem technically experienced.

[moeterbln (Thomas Graf)]

For me they don’t seem experienced since they obviously didn’t know which code they are using. The directed me to a source link which has a 30msec delay. That was a lie! Later they just say they changed it to 500msec and without explaining why they did so.

[jmarcelino (Jose Marcelino)]

Yes I tried explaining to them on twitter that this affects all gateways I’ve tried but they don’t appear to be the most reasonable people on this issue.

I prefer a Hanlon’s razor approach to it but have to say there has been a running rumour that TTN protocol / TTN gateways would have a “preferred” path, even before MatchX took to twitter about it.

So far in my tests TTN protocol has actually had worse delay, likely because of all the overhead (can’t compete with simple JSON over UDP)

[htdvisser (Hylke Visser)]

@Verkehrsrot

But why did the timing suddenly stop to work with the MatchX Gateways, after it worked without problems for several weeks? Was there a change in the TTN backend in the past days/weeks?

There have not been any changes in the scheduling logic since we doubled the time-before-tx on March 10 2016 (1bb19bf818). After reading the discussion here (and on Twitter) I just added another 200ms (c8bd97b135) to increase support for slow connections (or slow gateways). I’ll try to have that change deployed later today.

@jmarcelino

there has been a running rumour that TTN protocol / TTN gateways would have a “preferred” path

This is definitely not true. In the future we may give application owners the option to prefer authenticated/secure gateways (using MQTT or gRPC over TLS) over unauthenticated/insecure gateways (using JSON over UDP), but that won’t be enabled by default.

@jmarcelino

So far in my tests TTN protocol has actually had worse delay

The MQTT server in between indeed adds a bit of delay. We’re working hard on reducing this by optimizing our MQTT server. Right now the extra delay is a few milliseconds at worst.

Of course TCP may also add some extra delay on bad connections (TCP packets are retransmitted, whereas UDP packets would simply get lost)

[jmarcelino (Jose Marcelino)]

@htdvisser

After reading the discussion here (and on Twitter) I just added another 200ms (c8bd97b135) to increase support for slow connections (or slow gateways). I’ll try to have that change deployed later today.

Wow this is like the best Xmas present from TTN I could ever hope (yes even better than the TTN gateway Thank you so much @htdvisser

@htdvisser

Of course TCP may also add some extra delay on bad connections (TCP packets are retransmitted, whereas UDP packets would simply get lost)

Yes I think much of the overhead is on packet generation on the client side. But very good to know you’re on top of the server!

[Verkehrsrot]

I still have the “too late” error with the MatchX Gateway.

Fri Dec 22 12:24:29 2017 daemon.info syslog[1428]:
INFO: Received pkt from mote: D000742B (fcnt=46037)
Fri Dec 22 12:24:29 2017 daemon.info syslog[1428]:
JSON up: {“rxpk”:[{“tmst”:2204454315,“time”:“2017-12-22T12:24:28.961510Z”,“tmms”:1197980687961,“chan”:2,“rfch”:1,“freq”:868.500000,“stat”:1,“modu”:“LORA”,“datr”:“SF7BW125”,“codr”:“4/5”,“lsnr”:7.0,“rssi”:-47,“size”:23,“data”:“ACt0ANB+1bNwA+EbAAujBACLA2K81Hg=”}]}
Fri Dec 22 12:24:30 2017 daemon.info syslog[1428]: INFO: [down] PULL_ACK received in 49 ms
Fri Dec 22 12:24:34 2017 daemon.info syslog[1428]: INFO: [down] PULL_RESP received - token[182:216] : )
Fri Dec 22 12:24:34 2017 daemon.info syslog[1428]:
JSON down: {“txpk”:{“imme”:false,“tmst”:2209454315,“freq”:868.5,“rfch”:0,“powe”:14,“modu”:“LORA”,“datr”:“SF7BW125”,“codr”:“4/5”,“ipol”:true,“size”:33,“ncrc”:true,“data”:“IHrmg/62xh41zRa0XrvdYkBYcgohzAzbMejsCECNSLGH”}}
Fri Dec 22 12:24:34 2017 daemon.debug syslog[1428]: src/jitqueue.c:233:jit_enqueue(): ERROR: Packet REJECTED, already too late to send it (current=2208964645, packet=2209454315, type=0)
Fri Dec 22 12:24:34 2017 daemon.info syslog[1428]: ERROR: Packet REJECTED (jit error=1)
Fri Dec 22 12:24:40 2017 daemon.info syslog[1428]: INFO: [down] PULL_ACK received in 49 ms

[moeterbln (Thomas Graf)]

What firmware version is shown in the MatchX cloud interface for your gateway? I think that runs the old firmware.

[arjanvanb (Arjan)]

@htdvisser

I just added another 200ms (c8bd97b135) to increase support for slow connections (or slow gateways). I’ll try to have that change deployed later today

Is there any way for people to see which backend version is active?

(I don’t think Releases · TheThingsArchive/ttn · GitHub tells us?)

[Verkehrsrot]

Still 0.0.3, and there is no update offered in eux.matchx.io

[moeterbln (Thomas Graf)]

I didn’t choose a update myself. They pushed it to 0.0.4 without notifying me after the long discussions. I think you have to contact them on all channels to make them react.

[arjanvanb (Arjan)]

So, if TTN indeed did not make any changes that caused the problems: any chance MatchX changed the firmware first without all of you knowing that?

[moeterbln (Thomas Graf)]

That is possible and the usual way they seem to act. Unfortunally you don’t have the chance to really know that. You have to believe the firmware version that is displayed.
I won’t expect them to acknowledge any bug that is found to be their fault. By definition it seems that only TTN can be the bad guy.

[htdvisser (Hylke Visser)]

You can get the deployed versions from the discovery server using ttnctl discover [router/broker/handler]:

ID                                   ADDRESS                              VERSION              PUBLIC
==                                   =======                              =======              ======

meshed-router                        thethings.meshed.com.au:1901         v2.9.0-0c455d496d... true
switch-router                        ttn.opennetworkinfrastructure.org... v2.9.0-0c455d496d... true
ttn-router-asia-se                   asia-se.thethings.network:1901       v2.9.1-8477a84c83... true
ttn-router-brazil                    brazil.thethings.network:1901        v2.9.1-8477a84c83... true
ttn-router-eu                        eu.thethings.network:1901            v2.9.1-8477a84c83... true
ttn-router-jp                        router.thethingsnetwork.jp:1901      v2.9.0-dev-e4cb53... true
ttn-router-us-west                   us-west.thethings.network:1901       v2.9.1-8477a84c83... true

v2.9.1 contains the 200ms extra time for slow gateways or gateways on a slow connection

[moeterbln (Thomas Graf)]

@htdvisser

v2.9.1 contains the 200ms extra time for slow gateways or gateways on a slow connection

Does that include earlier delivery of Join Accept messages to the gateway?

[gatewayer]

You have no idea what you are talking about Informed people pushing nonsense, do you even “tech” dude?

[htdvisser (Hylke Visser)]

@gatewayer

The changes were 100% made by TTN, TTN stated they changed it

You can see the history of our scheduling code on github.

@moeterbln

Does that include earlier delivery of Join Accept messages to the gateway?

Yes, all downlink messages should arrive at the gateway earlier now.

[htdvisser (Hylke Visser)]

@gatewayer

You have no idea what you are talking about Informed people pushing nonsense, do you even “tech” dude?

This is not how we talk to our fellow community members here, @gatewayer. Please treat people with respect. Consider this an official warning.

[gatewayer]

Officially considered