Can't access TTN Console, ttnctl fails, TTN Node-RED library fails, and Kickstarter gateways get stuck after daily reboot

65sc02 · May 17, 2020, 9:10am

Ah! Thanks for sharing this link. Yes, the NOC problem may be what is causing this.

arjanvanb · May 17, 2020, 9:12am

See The Things Network Status Page [HowTo].

IT-Gypsy · May 17, 2020, 9:45am

Yesterday I installed my new gateway and now I wanted to check.

I can’t access the “gateway” page
the system says ÿou don’t have a gateway"
the console page only shows devices, the gateway page keeps saying “initializing”

By the remarks above I assume the issue is with the server and not withme and has not been solved yet?

65sc02 · May 17, 2020, 9:47am

Yup, I think you are correct.

phroc · May 17, 2020, 10:20am

Same here in Spain.

arjanvanb · May 17, 2020, 11:01am

Though TTN Console is great for debugging (especially to debug OTAA Join problems, or to see the gateway’s Traffic page when one does not have access to the gateway’s raw logs), many of us simply don’t need it every day as more reliable alternatives exist:

Unfortunately, right now ttnctl is failing too:

ttnctl gateways status 19711137
INFO Discovering Router...
FATAL Could not get Router from Discovery      error=Internal error: [Unavailable] transport is closing

But often that works when TTN Console does not.

The TTN Node-RED library is reported to be failing too, but it seems it’s time to migrate to a regular Node-RED MQTT Input node anyhow.
Data is being delivered just fine using the MQTT Data API and the HTTP Integration (and likely other integrations as well). An MQTT command line interface might be more reliable than the application/device’s Data page in TTN Console anyhow:
```
mosquitto_sub -h eu.thethings.network -t +/# -u haarlem-app-1 -P ttn-account-v2.ko3<redacted>fyr-A
```
See also A Python program to listen to your devices with MQTT.

andreiasv · May 17, 2020, 12:49pm

Same issue!

pomplesiegel · May 17, 2020, 1:52pm

Same thing in the US. Data is still coming through via MQTT but I can’t access the console for configuration and such

Vaelid · May 17, 2020, 2:51pm

Hi all was up but now down and cant access here in the UK What are the issues if look at fault status show all workig except instability in MQTT

dreimal8 · May 17, 2020, 2:58pm

I can access applications - but gateway settings stuck at “fetching gateways”.
Also some gateways disappered on the map.
(north germany)

jeroen2 · May 17, 2020, 3:02pm

No access to GWs and some of them no traffic at all

jeroen2 · May 17, 2020, 3:02pm

Check thethingsnetwork.slack.com -> ops channel

dreimal8 · May 17, 2020, 3:29pm

console seems to be down complete - getting this now:

502 Bad Gateway

nginx

sweetpants1 · May 17, 2020, 4:39pm

Those are old messages

pablocastillo · May 17, 2020, 4:41pm

The same for me now. 502 bad gateway. I can not enter to the consoel.

wasn · May 17, 2020, 5:21pm

502 bad gateway. the servers behind the nginx instance are down.
please fix it fast.

arjanvanb · May 17, 2020, 9:21pm

This also affects the Kickstarter The Things Gateway, if that does its daily reboot:

[2020-05-17T10:57:25.238Z] MAIN: Rebooting gateway for firmware update check
...
[2020-05-17T10:57:55.889Z] HTTP: Starting connection
[2020-05-17T10:57:55.900Z] HTTPS: Connection Opened: Starting TLS Negotiation
[2020-05-17T10:57:55.903Z] HTTP: Wait for TLS Connect
[2020-05-17T10:57:56.372Z] HTTP: TLS Connection Opened: Starting Clear Text Communication
...
[2020-05-17T10:58:56.383Z] HTTP: Got 341 bytes
[2020-05-17T10:58:58.385Z] HTTP: Connection Closed
[2020-05-17T10:58:58.392Z] HTTP: Close active socket 1
[2020-05-17T10:58:58.395Z] CONF: Parsing response token: HTTP/1.1 504 Gateway Time-out
[2020-05-17T10:58:58.397Z] ACTI: Object expected
[2020-05-17T10:58:58.399Z]
[2020-05-17T10:58:58.402Z] CNFG: Downloading gateway configuration failed
[2020-05-17T10:58:58.409Z]
[2020-05-17T10:58:58.421Z] CNFG: Load online user config state change to 8
[2020-05-17T10:58:58.423Z]
[2020-05-17T10:58:58.424Z] CNFG: Communication ERROR
...
[2020-05-17T10:59:03.424Z] HTTP: Starting connection
...
[2020-05-17T11:00:05.956Z] CONF: Parsing response token: HTTP/1.1 504 Gateway Time-out
[2020-05-17T11:00:05.961Z] ACTI: Object expected

After showing this error a total of 6 times, it did not try again, but after a final HTTP: Starting connection just got stuck with the 3rd LED slowly blinking (“Still activating”) and only logging stack sizes, heap usage and LoRa packets. Only when I rebooted manually a few minutes ago it showed the above errors again.

So: Kickstarter gateways might need a manual reboot when the server problems have been resolved.

Lab3Developer · May 17, 2020, 10:03pm

Something is wrong witht he console this morning… I get a bad gateway error when trying to load the console.

swagner0 · May 18, 2020, 12:42am

Same here in eastern North America … 503 Gateway error. Gateways can’t forward packets.

smartek · May 18, 2020, 2:58am

The same problem in Brazil, since sunday 01h00