Suddenly my MatchX gateway stopped sending. I can’t join anymore. Nothing changed by me. With the same node I can join and get packets on all other gateways I’ve tried. It was all working for 2 Month without changes.
Did anyone have similar problems?
Did you check if txpk packets arrive on port 1700 on the WAN-Port of the gateway?
sudo tcpdump -AUq port 1700
See https://www.thethingsnetwork.org/wiki/Backend/Connect/Gateway
If your MatchX Gateway is from the first batch, you may try to derive the root password from the Gateway’s MAC-address, as described in the MatchX thread in this forum. Then you can take a look at the gateway’s log with journalctl.
Unfortunally that root-password doesn’t work. And I see packets going to the matchx looking in my internet-connection.
TTN console matches those packets. I see the json-data in my wireshark dump. Yesterday i was working 
Did you try to send a packet to node with confirmation flag set (ACK)?
Are those packets displayed as ACKed in the application?
If yes this would proof that TX hardware is up and running. (As long as there is no other TTN gateway in range of node)
Unfortunally the OTAA Join reply is not received by the node. In the UDP traffic I can clearly see the Join Ack that should be sent to the node.
Ah now I see the problem. The gateway generates a error “TOO_LATE” unfortunately that seems to be a problem with the TTN router/backend. My internet connection is fine. I don’t see any delays to various sites.
Using router.eu.thethings.network should be the right one. At least for 2 month that was working.
That is really annoying that I have to look at dumps for simple debugging things. Shouldn’t be there any other way where I can debug such things? It seems to be messaged to the backend so that could be displayed in the ttn console - That would have saved some hours searching every where.
I tried an OTAA join on my TTN connected MatchX Gateway. No problems. Node joins on first attempt.
Comparing to a Laired and a RAK831 Gateway looking at the dumps all looks the same but works. It is totally unclear why the gateway issues a TOO_LATE-Error.
New idea on this: maybe your gateway has no correct time. GPS up und running?
It all looks good. I changed to their could and will test in the evening.
GPS is running fine to their console.
What currently is not working is spectral scan. CPU load is at 17% without any traffic. Maybe that is normal.
Looking at the network packets the timing is the same compared with the other gateways. Clearly I can see in the payload that the packets are totally in time.
Currently I suppose a hardware defect. Maybe something that has to do with the part which is responsible for the spectral scan - That was working in the past.
Ok I compared with the MatchX cloud. The Join Accept from the cloud to the gateway is sent with a delay of 0.26sec while the ttn cloud send that Join Accept after 4.6sec and the MatchX gateway drops the packet with error TOO_LATE. Is there a reason why the packet is sent to the gateway that early? Maybe that can be reduced a bit to help here?
I don’t know any setting to influence that delay. Other gateways seem to be fine with that behavior of ttn.
I just see that I can confirm that the delay is the essential problem in my case. I just got a Join Accept that was sent out. The delay at the gateway for the Join Accept was 4.19sec and that was accepted by the MatchX.
Spectral scan via MatchX cloud ist not working on my MatchX gateway, too, so don’t worry about that.
Sure that this is not a network latency / routing problem?
To check if RX/TX hardware of your MatchX gateway is properly working, you could connect a node by ABP instead OTAA and send some downlink data via the TTN console. If the node receives it, your gateway should be ok.
Using the MatchX cloud works like ttn cloud used to work before. So hardware seems to be OK. There is nothing I can do.
I don’t understand why the packets come that late with TTN to the gateway. Did you look at the traffic? What delays are at your GW for join accept messages?
The Issue has been solved. For the communication regarding MatchX you can read here:
https://matchx.io/community/cloud-software
In short: They changed some constants that where obviously exactly the problem I observed.
Now suddenly my ssh console closed and they updated my firmware and it is working again.
Firmware versioning -> Fail
Parameter changes -> Fail
Accusing TTN of the behaviour -> Fail
No “Sorry for not listening to you” -> Fail
So that is an AWSOME company as they call themselves and keep making other products bad.
Sorry I had to rant but I’m really upset. Just in case you have any problems with that gateway don’t trust what the MatchX people say.
Today i wanted to check the issue in my MatchX gateway and found, that i have the same problem suddenly. OTAA joining is not possible any longer.
TTN Console
Join Accept
21:06:35{
“gw_id”: “eui-40d63cfffe030e78”,
“payload”: “ILL7HyMWilV2GKJhWbx/Q+ketta7e810c0+8Qfw8s3aG”,
“lora”: {
“spreading_factor”: 7,
“bandwidth”: 125,
“air_time”: 71936000
},
“coding_rate”: “4/5”,
“timestamp”: “2017-12-21T20:06:35.754Z”,
“frequency”: 868100000
}Join Request
21:06:31{
“gw_id”: “eui-40d63cfffe030e78”,
“payload”: “ACt0ANB+1bNwA+EbAAujBABYX00bwmg=”,
“dev_eui”: “0004A30B001BE103”,
“lora”: {
“spreading_factor”: 7,
“bandwidth”: 125,
“air_time”: 61696000
},
“coding_rate”: “4/5”,
“timestamp”: “2017-12-21T20:06:31.121Z”,
“rssi”: -59,
“snr”: 10,
“app_eui”: “70B3D57ED000742B”,
“frequency”: 868100000
}Assuming base64-encoded packet
ACt0ANB+1bNwA+EbAAujBABYX00bwmg=Message Type = Join Request
AppEUI = 70B3D57ED000742B
DevEUI = 0004A30B001BE103
DevNonce = 5F58
MIC = 4D1BC268====
MatchX Gateway Syslog:
Thu Dec 21 20:06:31 2017 daemon.info syslog[5145]:
INFO: Received pkt from mote: D000742B (fcnt=46037)Thu Dec 21 20:06:31 2017 daemon.info syslog[5145]:
JSON up: {“rxpk”:[{“tmst”:137200611,“time”:“2017-12-21T20:06:30.078382Z”,“tmms”:1197922009078,“chan”:0,“rfch”:1,“freq”:868.100000,“stat”:1,“modu”:“LORA”,“datr”:“SF7BW125”,“codr”:“4/5”,“lsnr”:10.0,“rssi”:-59,“size”:23,“data”:“ACt0ANB+1bNwA+EbAAujBABYX00bwmg=”}]}Thu Dec 21 20:06:35 2017 daemon.info syslog[5145]: INFO: [down] PULL_RESP received - token[121:8] : )
Thu Dec 21 20:06:35 2017 daemon.info syslog[5145]:
JSON down: {“txpk”:{“imme”:false,“tmst”:142200611,“freq”:868.1,“rfch”:0,“powe”:14,“modu”:“LORA”,“datr”:“SF7BW125”,“codr”:“4/5”,“ipol”:true,“size”:33,“ncrc”:true,“data”:“ILL7HyMWilV2GKJhWbx/Q+ketta7e810c0+8Qfw8s3aG”}}Thu Dec 21 20:06:35 2017 daemon.debug syslog[5145]: src/jitqueue.c:233:jit_enqueue(): ERROR: Packet REJECTED, already too late to send it (current=141910935, packet=142200611, type=0)
Thu Dec 21 20:06:35 2017 daemon.info syslog[5145]: ERROR: Packet REJECTED (jit error=1)
If they are not already are updating your firmware just wait they have full access if you didn’t block the OpenVPN tunnel and trust them.
After i found that the gateway openes up a vpn tunnel to matchX cloud, it isolated it in the network, but left vpn tunnel open.
How can i see that they upgraded my gateway, and when can i expect this to happen?
Strange…