Our problem is different some of our gateways seem to be blocked, and for blocked I mean no data are sent to our application, we use http integration.
Here is a list of some blocked gw
eui-00800000a0005afb
eui-d03972fffe17f926
Sometimes we can see the above gw to ping ttn from console, but no data are showed or sent to our application.
Our infrastructure is based on 29 of the following model https://www.multitech.com/brands/multiconnect-conduit-ip67
About half of them are based on packet forwarder of ttn, the other half is based on legacy semtech packet forwarder. The problems seems to affect much more the gw with the legacy packet forwarder.
Before to write here, we tested and debug different gw in house to ensure no bug is present in the gw firmware.
I personally tested
check if the packet forwarder crash or block for some reasons, nothing found the daemon runs without problem and collect the incoming lorawan data ( the firmware provide an angel for the daemon in case of stuck )
check if the problem it was related to network fault, nothing found
check the udp traffic sent from the packet forwarder to “router.eu.thethings.network” port 1700, nothing found
After the tests, we decided to move the blocked gw to a new eui and magically it started to run again. We moved a working gw to the previous blocked eui and it stopped to work.
From my personal experience when the ttn console stops to work, also the api http://noc.thethingsnetwork.org:8085/api/v2/gateways/ stops to work. The application ttnctl doesn’t give more details, so we based our observations on what our application receive from the http integration.
You’ll need its gateways status command, which is undocumented. Still, it does not help you much, as given its results I would have expected that the second one would have forwarded data to TTN, and hence should have forwarded data to your application just fine, even when nothing showed in TTN Console:
ttnctl gateways status eui-00800000a0005afb
INFO Discovering Router...
INFO Connecting with Router...
INFO Connected to Router
INFO Received status GatewayID=eui-00800000a0005afb
Last seen: 2020-07-03 17:47:20.716732109 +0200 CEST
Timestamp: 0
Reported time: 2020-07-03 17:47:20 +0200 CEST
Frequency Plan: EU_863_870
Bridge: gs.v3.
IP Address: 127.0.0.1
Location: (40.820091, 14.185210; source unknown)
Rtt: not available
Rx: (in: 5; ok: 5)
Tx: (in: 0; ok: 0)
ttnctl gateways status eui-d03972fffe17f926
INFO Discovering Router...
INFO Connecting with Router...
INFO Connected to Router
INFO Received status GatewayID=eui-d03972fffe17f926
Last seen: 2020-07-13 18:26:14.127380524 +0200 CEST
Timestamp: 0
Reported time: 2020-07-13 18:26:14 +0200 CEST
Frequency Plan: EU_863_870
Bridge: gs.v3.
IP Address: 127.0.0.1
Location: (40.820431, 14.185250; source unknown)
Rtt: not available
Rx: (in: 2; ok: 2)
Tx: (in: 0; ok: 0)
@KrishnaIyerEaswaran2 can you see anything special for those gateway EUIs? Even if traffic is not shown in TTN Console, their traffic should still make it to the applications and hence the HTTP Integration?
Aside: I don’t think the gateways are currently connected using those EUIs: both ttnctl commands give the exact same output now, 14 hours later. So I guess the hardware is now using the new EUIs (and other gateways in TTN Console).
As you cannot re-register/re-use a gateway EUI in TTN, just to be sure: you did not edit or delete the troublesome gateway configuration from TTN Console, but added another gateway configuration in TTN Console (and configured the hardware to use the new EUI)?
For the EUIs that work now: can you see anything different for Bridge: gs.v3. in the output of ttnctl? And in TTN Console both configurations are using router-eu?
We managed to use ttnctl to check the status of gateway when the console is unavailable (workaround found on this forum).
The main problem is still there, our application doesn’t receive from certain euis even if the gateway status is online. When we move to a new eui everything starts to work again.
I apologize, I just seen your new message.
Take in account that actually I have a gw on my desk, it has been associated to 2 different euis
eui-00800000a0005afa, this eui is completely blocked, can ping but no message received
.\ttnctl-windows-amd64.exe gateway status eui-00800000a0005afa
INFO Discovering Router…
INFO Connecting with Router…
INFO Connected to Router
INFO Received status GatewayID=eui-00800000a0005afa
Last seen: 2020-07-14 11:37:10.972751387 +0200 CEST
Timestamp: 0
Reported time: 2020-07-14 11:37:10 +0200 CEST
Frequency Plan: EU_863_870
Bridge: gs.v3.
IP Address: 127.0.0.1
Location: (40.820271, 14.184980; source unknown)
Rtt: not available
Rx: (in: 3; ok: 3)
Tx: (in: 0; ok: 0)
if I move the same gw on this eui eui-d03972fffe17f927 the messages are sent to our application
.\ttnctl-windows-amd64.exe gateway status eui-d03972fffe17f927
INFO Discovering Router…
INFO Connecting with Router…
INFO Connected to Router
INFO Received status GatewayID=eui-d03972fffe17f927
Last seen: 2020-07-14 11:24:43.849339311 +0200 CEST
Timestamp: 0
Reported time: 2020-07-14 11:24:25 +0200 CEST
Frequency Plan: EU_863_870
Bridge: gs.v3.
IP Address: 127.0.0.1
Location: (40.820339, 14.185080; source unknown)
Rtt: not available
Rx: (in: 8; ok: 8)
Tx: (in: 0; ok: 0)
As I understand it: not simultaneously. If the hardware uses eui-d03972fffe17f927 all is fine, but if it uses eui-00800000a0005afa traffic is silently lost. (So: two gateway configurations in TTN Console, just one being used at a given time.)
Of course the same gw is not using two different euis simultaneously. Every time I want to change the eui, I edit the configuration and reboot. Actually is on eui-00800000a0005afa.
Yes it uses the legacy packet forwarder on udp.
Nothing relevant in the gw, no error, no warning, I can see the packets outgoing on udp port 1700.
Yes I’m the only one that register and manage the gw, once the gw is working properly I move the ownership to the customer, but I still have full control on the gws because they are bind to my account (as creator I suppose).
Note that this is about receiving (no) data through the HTTP Integration. Even if one does not own the gateway, or cannot even access its details in TTN Console, the application and hence the HTTP Integration should see the data?
Did the problems start when transferring ownership? Or is that unrelated?
(You’re probably still visible as a “collaborator” in the gateway settings?)
Yes this is the main interesting point. As I said we have 29 installed gws, all of them are https://www.multitech.com/brands/multiconnect-conduit-ip67 . About half of them are the first release of hardware with the AEP firmware installed 1.x.x, the other half are the new hardware version with AEP firmware 5.x.x installed. Until the version 5 we installed the ttn packet forwarder. Because the ttn packet forwarder is no more supported and because from the release 5 is quite complicated to install in multitech gw, we decided to move to the legacy packet forwarder that is fully supported also in the web server.
Think about that this gw are installed on pole at 20m of altitude so is quite hard to move them from where they are, it is the main reasons because I choose to use only stable solutions. Fortunately for us the multitech cloud give us almost fully remote control.
At the beginning (1 month and half) we thought the problem was related to the new version of the firmware 5 ftp://ftp.multitech.com/wireless/mtcdt-x-210a/conduit-release-notes_5.1.2.pdf and after some emails to the multitech we decided to update all our gws (new hardware versions) to the last revision.
Actually the application server is in full debug to check that everything is working properly. We decided to log all the incoming messages from ttn before to process them, and when I say “no messages is received from some euis” simple there aren’t in the log files.
Actually only one gateway (first hardware release) with the ttn packet forwarder seems to be affected the other ones are still running with millions of messages without problem.
Has anyone tried to see if the gateway’s packet forwarder gets push/pull acks when using one of the problem eui’s?
Getting acks would at least indicate that the first level of infrastructure is willing to talk to it, in which case a problem would be a level up from there.
I can’t help but notice that’s not a number with a lot of entropy. Presumably it’s one assigned to the chassis by the hardware manufacturer? It does appear to start with an OUI assigned to Multitech.
By my tests, the traffic from the packet forwarder is always sent to the broker, but nothing is received.
Yes is a multitech, like I said we only use gw by multitech.
An other interesting point is that after 2-3 weeks of silence, sometimes an eui come back in live.
Please be more specific. What exactly have you confirmed is occurs, and what does not? Are push and pull acks received? Do you see the push and pull outbound UDP traffic?
The point was that fairly few bits of the given EUI are potentially unique, so the chance of a collision with another gateway seems higher than it would if more of the bits were in use - granted, that would seemingly require the manufacturer to make a mistake in assignments.
Please be more specific. What exactly have you confirmed is occurs, and what does not? Are push and pull acks received? Do you see the push and pull outbound UDP traffic?
I’m gonna to collect some log from the packet forwarder to show you what I mean
The point was that fairly few bits of the given EUI are potentially unique, so the chance of a collision with another gateway seems higher than it would if more of the bits were in use - granted, that would seemingly require the manufacturer to make a mistake in assignments.
What exactly you mean? The gateway eui is global unique from what I know, trying to register an existing eui drive you to an error, so where should be the collision?
A few years back at least two MultiTech gateways used the same EUI. One in the UK and one in the USA. at that time allowed connections with for the same EUI from different IPs, these days a security measure seems to be in place to prevent that. (Probably because some jokers reset all gateway coordinates to a fixed location to demonstrate the Semtech UDP protocol isn’t secure, or may-be just for fun?)