I’ve noticed that my 3 Gateways are “Not Connected” since yesterday for 2 of them and since 3 hours for the third one.
2 of them are on my local home network : 1 Rpi3 and 1 Kona Macro form Tektelic
One is at my office : this one is a Kona Macro.
Thoses gateways have routed millions messages for now and it’s the first time i see them down since the bug of October.
What I can say:
A traceroute to router.eu.thethings.network (NS resolved) fails after 104.44.11.72 :
The logs from packet_forwarder displays a good rythm of uploads but I can see the message : "INFO: [down] for server router.eu.thethings.network PULL_ACK received in 206 ms"
I don’t know if the port TCP/1700 is opened on the servers but I can’t telnet to them.
The TTn Status for Europe is Green.
Does anyone else has the same problem ?
I haven’t noticed any recent thread talking about this.
It’s very strange that there is so few people concerned, I’m even suspecting my provider to filter this UDP1700 port …
But if I’m not the only one concerned, it’s perhaps a problem on the path as you suppose in your reply.
OK I haven’t tested this before but it seems that the packets are still reaching to the applications, despite the gateways are “Down”.
No big problem so, but something is quite strange anyway …
ok, the problem, if problem is, seems stucked to EU.
I have tried to add an OTAA device on an existaing App in my console, but it fails. i can see the join messages but nothing else.
A new ABP device works well.
Can you try a traceroute from your gateway or from your station to see where it stops ?
Here is mine :
traceroute to router.eu.thethings.network (52.169.76.203), 30 hops max, 60 byte packets
Two gateways here are also reported to be down.
Both on completely different internet providers, so it cannot be filtering by the provider.
Also on the TTNmapper I see a lot of gateways offline in the neighborhood.
My gateway: Last heard at 2019-10-23T10:01:53.000Z
Pinging/trace routing through the ttn routers doesnt work as the servers are behind load balancers which are configured to not respond to such traffic.
Its the console that is down vs the routers - traffic is passing through to applications ok, unfortunately the database conection to the console front end looks broken - it has been that way off & on for last 2-3 days (I see 2-5 of my 30+ GW’s connected at any time…frustrating. This is not reflected in the status page.
These pages are not monitored by TTI staff so you need to hook up on Slack via the #ops channel. If you look at the forum threads you will see mutiple threads already calling out the issue so no need to set up & feed another thread
Since yesterday, all our Gw have fallen since the morning, we have reviewed our configurations and everything is fine, someone has any idea if there is a massive failure, it is very strange that all our GW have fallen … I am attentive to any help …
For everyone looking at the gateway console and seeing the gateway seems to be down/not connected, there is another way to check the gateway status.
Go to the ttn cli quick start page and download the appropriate binary for you system.
Use the instructions on the page to get the binary up and running, including ‘Register and Login’.
Once logged in, use ‘ttnctl gateways status < gatewayid >’ to check the status of your gateway. This will provide a ‘Last seen’ value which should be within minutes of the current time for connected gateways. Ignore the Tx/Rx statistics reported, those are not the totals for the lifetime of the gateway.
Example output (redacted):
INFO Discovering Router...
INFO Connecting with Router...
INFO Connected to Router
INFO Received status GatewayID=gatewayid
Last seen: 2019-10-25 01:10:35.388703964 +0200 CEST
Timestamp: 2529939411
Reported time: 2019-10-25 01:10:35 +0200 CEST
Description: Some description
Platform: IMST + Rpi
Contact email: email@some.domain
Frequency Plan: EU_863_870
Bridge: Mystique MQTT Bridge
Location: (53.NNNN, 6.NNNN; source unknown)
Rtt: 31ns
Rx: (in: 37307; ok: 1248)
Tx: (in: 1; ok: 1)
This method is more involved, but does not suffer the many outages the console is experiencing lately.
My 2 gateways are not connected as well since 2 days in two different european countries.
One is a dragino single channel Gateway and an Indoor Gateway from ttn.
But there are other gateways are running in Germany. Weird
Now my two Gateways seems to work again few minutes ago…appears Green in the console but dont forward Messages…weird.
Edit: and now its gone again.
Edit: and its working…cool
