TTN-router-US-WEST is having issues as of ~5AM EST, where GWs may have been disconnected/not communicating for an hour, then briefly came back in ~7AM EST, then interrupted again and, as of 10AM EST back online FYI. I will continue to monitor thru out the day, if anyone else can validate this please chime in for the rest of the US / elsewhere community - thank you and all be healthy and good! 
@MicrochipTTN Thank you for the update! didn’t even know TTN had a ‘Status Page’ - much needed. Here’s our 48 hr timestamp of events as reported by one of our sensors FYI:
Thanks!
@MicrochipTTN Any new updates? The problems are still present (spread dots means connection issues >1hr) 
No updates since 2 days ago:
Thanks!
Also seeing issues here …
I have confirmed my gateway is sending data to and from us-west however the gateway
still shows as “disconnected” even though UPD status packets flowing to and from us-west.
I can see lora sensor data being sent up as well yet application page for sensor shows
it has not been seen recently.
10:22:35.846382 IP 192.168.20.104.54193 > 13.66.213.36.1700: UDP, length 187
E…4E@.@.NZ…h.B.$…b…’…yy.{“stat”:{“time”:“2020-03-26 14:22:35 GMT”,“rxnb”:0,“rxok”:0,“rxfw”:0,“ackr”:0.0,“dwnb”:0,“txnb”:0,“pfrm”:“IMST + Rpi",“mail”:"ttn@mydomain.net”,“desc”:“RAK831 TTN gateway”}}
10:22:35.915556 IP 13.66.213.36.1700 > 192.168.20.104.54193: UDP, length 4
E… >.@.3.Q…B.$…h…:…
10:22:37.278764 IP 192.168.20.104.49997 > 13.66.213.36.1700: UDP, length 12
E…(4.@.@.N…h.B.$.M…]g.d…’…yy.
10:22:37.408808 IP 13.66.213.36.1700 > 192.168.20.104.49997: UDP, length 4
E… ?.@.2.Q~.B.$…h…M…z…d…
10:22:42.448738 IP 192.168.20.104.49997 > 13.66.213.36.1700: UDP, length 12
E…(5.@.@.M…h.B.$.M… h.c@…’…yy.
10:22:42.516740 IP 13.66.213.36.1700 > 192.168.20.104.49997: UDP, length 4
E… CX@.2.M…B.$…h…M…=…c@…
10:22:47.548724 IP 192.168.20.104.49997 > 13.66.213.36.1700: UDP, length 12
E…(6p@.@.L…h.B.$.M…’…yy.
10:22:47.617719 IP 13.66.213.36.1700 > 192.168.20.104.49997: UDP, length 4
E… G.@.2.I…B.$…h…M…d…
10:22:52.648754 IP 192.168.20.104.49997 > 13.66.213.36.1700: UDP, length 12
E…(7c@.@.K…h.B.$.M…(x.S8…’…yy.
10:22:52.716839 IP 13.66.213.36.1700 > 192.168.20.104.49997: UDP, length 4
E… J$@.2.G2.B.$…h…M…E…S8…
10:22:57.748785 IP 192.168.20.104.49997 > 13.66.213.36.1700: UDP, length 12
E…(8.@.@.Js…h.B.$.M…9…&…’…yy.
10:22:57.817851 IP 13.66.213.36.1700 > 192.168.20.104.49997: UDP, length 4
E… K.@.2.Ei.B.$…h…M…Vw…&…
10:23:02.351068 IP 192.168.20.104.54193 > 13.66.213.36.1700: UDP, length 235
E…:d@.@.H…h.B.$…’…yy.{“rxpk”:[{“tmst”:3383993300,“time”:“2020-03-26T14:23:02.350784Z”,“chan”:5,“rfch”:1,“freq”:904.900000,“stat”:1,“modu”:“LORA”,“datr”:“SF7BW125”,“codr”:“4/5”,“lsnr”:8.0,“rssi”:-42,“size”:17,“data”:“QP4eAiaA0yIBprSnyQ/kANo=”}]}
10:23:02.858723 IP 192.168.20.104.49997 > 13.66.213.36.1700: UDP, length 12
E…(:.@.@.H…h.B.$.M…-…’…yy.
10:23:02.926655 IP 13.66.213.36.1700 > 192.168.20.104.49997: UDP, length 4
E… Mt@.2.C…B.$…h…M…
10:23:05.843009 IP 192.168.20.104.54193 > 13.66.213.36.1700: UDP, length 187
E…;4@.@.Gk…h.B.$…k…’…yy.{“stat”:{“time”:“2020-03-26 14:23:05 GMT”,“rxnb”:2,“rxok”:1,“rxfw”:1,“ackr”:0.0,“dwnb”:0,“txnb”:0,“pfrm”:“IMST + Rpi",“mail”:"ttn@mydomain.net”,“desc”:“RAK831 TTN gateway”}}
10:23:05.910774 IP 13.66.213.36.1700 > 192.168.20.104.54193: UDP, length 4
E… O.@.3.@u.B.$…h…"…k…
10:23:07.958769 IP 192.168.20.104.49997 > 13.66.213.36.1700: UDP, length 12
1 Like
@tbalon Correct; we’re still seeing ‘network disconnect gaps’ as well within 1,2 and even 3 continuous hours. Doesn’t mean the TTN server was down for 1,2 or 3 hours, but that it was not processing packets correctly / dropping packets within those hours:
Yes, I can see data is occasionally showing up on the TTN application device
data page.
However, when using mosquitto_sub to connect to us-west.thethings.network,
no data is broadcast.
Tom
I can’t register any new devices
Hi everyone,
My gateway was connected for about an hour yesterday but got disconnected and has been disconnected for more than 24 hours now.
My gateway is shown as disconnected, but I can see on the thethingsnetwork site my application node is sending activation requests. Is there any thing we can do to help this issue? My application and gateway have been up a couple of years.
Experimenting the same problems right here in us-west.thethings.network, Gateway seems to be disconected since yesterday, but still receiving packages, devices uplinks are intermitent, this morning packages were lost from 12am to 8am.
Seems that I am not the only one with the problem.
@rlgjr562 Sadly there’s not a lot we can do - this is on TTN’s side.
Something simmilar happened last year around late October. I had a conversation with TTN, who explained to me that TTN “…is for testing and development purposes.”, which I completely understand. But that is precisely the point and why so many of us (including ) only use it for testing and development and NOT for scientific or commercial purposes. Nevertheless, we argue, if you’re a coffee shop with a ‘Free WiFi’ sticker on the door as an invitation to come in and buy your coffee as well and, once inside, I learn the WiFi is not working then we think we’ve earned the right to ask questions about it. Especially when, in our view, TTN is also meant to be the ‘springboard’ to a paid subscription (aka the cup of coffee) to TTI which WILL guarantee SLA, QoS, tech support, etc.
Is understandable that a ‘free’ community-based Network Server (TTN) will not be managed the same way as a PAID commercial-private one (TTI), but with just ONE server in the US (as far as we’re concerned), there’s not a lot the rest of us can do to answer the ‘global community’ call to ‘test and develop’ with our own resources, gateways, backhaul, etc when the Network Server itself goes offline or its capacity is crammed with too many sensor’s request and begins dropping packets. We’ll have to wait for TTN/TTI to figure it out and, hopefully, add more servers in the US Or we can pay TTI (or other NS providers for that matter, aka other ‘coffee shops’) - all of which, of course, are options to us all.
2 Likes
Everything seems to be back to normal since March 30 :
Hi, just for curiosity has anybody switched to the eu router to see if there is connectivity? Does speak anything against using the eu router long term (if it’s more reliable)?
ds
Yes, network delays will kill downlinks for OTAA and regular ones. A gateway won’t be able to send them after the time slot has passed (and even if it did the node would not be listening anyway). So use the closest router with network delay wise.
Is anybody else still having problems with us-west connectivity? We had a couple gateways drop offline on Monday 4/6, and power cycling doesn’t seem to bring them back. Joins from devices also don’t seem to work.
They’re multitech conduit ip67 gateways, and they’re set up to forward packets to TTN using the script detailed here: https://www.thethingsnetwork.org/docs/gateways/multitech/aep.html. Looking at the logs (below), it looks like its failing to connect to bridge.us-west.thethings.network.
Anybody have any ideas?
11:37:31 *** Multi Protocol Packet Forwarder for Lora Gateway ***
Version: 3.0.20
11:37:31 *** Lora concentrator HAL library version info ***
Version: 5.0.1; Options: native;
***
11:37:31 INFO: Little endian host
11:37:31 INFO: found global configuration file /var/config/lora/global_conf.json, parsing it
11:37:31 INFO: /var/config/lora/global_conf.json does contain a JSON object named SX1301_conf, parsing SX1301 parameters
11:37:31 INFO: lorawan_public 1, clksrc 0
lgw_board_setconf:437: Note: board configuration; lorawan_public:1, clksrc:0
11:37:31 INFO: no configuration for LBT
11:37:31 INFO: antenna_gain 0 dBi
11:37:31 INFO: Configuring TX LUT with 16 indexes
11:37:31 INFO: radio 0 enabled (type SX1257), center frequency 904300000, RSSI offset -166.000000, tx enabled 1
lgw_rxrf_setconf:486: WARNING: NOT A VALID TX NOTCH FILTER FREQUENCY [126000..250000]Hz
lgw_rxrf_setconf:498: Note: rf_chain 0 configuration; en:1 freq:904300000 rssi_offset:-166.000000 radio_type:2 tx_enable:1 tx_notch_freq:0
11:37:31 INFO: radio 1 enabled (type SX1257), center frequency 905000000, RSSI offset -166.000000, tx enabled 0
lgw_rxrf_setconf:498: Note: rf_chain 1 configuration; en:1 freq:905000000 rssi_offset:-166.000000 radio_type:2 tx_enable:0 tx_notch_freq:0
11:37:31 INFO: Lora multi-SF channel 0> radio 0, IF -400000 Hz, 125 kHz bw, SF 7 to 12
lgw_rxif_setconf:617: Note: LoRa 'multi' if_chain 0 configuration; en:1 freq:-400000 SF_mask:0x7e
11:37:31 INFO: Lora multi-SF channel 1> radio 0, IF -200000 Hz, 125 kHz bw, SF 7 to 12
lgw_rxif_setconf:617: Note: LoRa 'multi' if_chain 1 configuration; en:1 freq:-200000 SF_mask:0x7e
11:37:31 INFO: Lora multi-SF channel 2> radio 0, IF 0 Hz, 125 kHz bw, SF 7 to 12
lgw_rxif_setconf:617: Note: LoRa 'multi' if_chain 2 configuration; en:1 freq:0 SF_mask:0x7e
11:37:31 INFO: Lora multi-SF channel 3> radio 0, IF 200000 Hz, 125 kHz bw, SF 7 to 12
lgw_rxif_setconf:617: Note: LoRa 'multi' if_chain 3 configuration; en:1 freq:200000 SF_mask:0x7e
11:37:31 INFO: Lora multi-SF channel 4> radio 1, IF -300000 Hz, 125 kHz bw, SF 7 to 12
lgw_rxif_setconf:617: Note: LoRa 'multi' if_chain 4 configuration; en:1 freq:-300000 SF_mask:0x7e
11:37:31 INFO: Lora multi-SF channel 5> radio 1, IF -100000 Hz, 125 kHz bw, SF 7 to 12
lgw_rxif_setconf:617: Note: LoRa 'multi' if_chain 5 configuration; en:1 freq:-100000 SF_mask:0x7e
11:37:31 INFO: Lora multi-SF channel 6> radio 1, IF 100000 Hz, 125 kHz bw, SF 7 to 12
lgw_rxif_setconf:617: Note: LoRa 'multi' if_chain 6 configuration; en:1 freq:100000 SF_mask:0x7e
11:37:31 INFO: Lora multi-SF channel 7> radio 1, IF 300000 Hz, 125 kHz bw, SF 7 to 12
lgw_rxif_setconf:617: Note: LoRa 'multi' if_chain 7 configuration; en:1 freq:300000 SF_mask:0x7e
11:37:31 INFO: Lora std channel> radio 0, IF 300000 Hz, 500000 Hz bw, SF 8
lgw_rxif_setconf:591: Note: LoRa 'std' if_chain 8 configuration; en:1 freq:300000 bw:1 dr:4
11:37:31 INFO: FSK channel 8 disabled
lgw_rxif_setconf:525: Note: if_chain 9 disabled
11:37:31 INFO: /var/config/lora/global_conf.json does contain a JSON object named gateway_conf, parsing gateway parameters
11:37:31 INFO: Found 1 servers in array.
11:37:31 INFO: Server 0 configured to "router.us.thethings.network"
11:37:31 INFO: packets received with a valid CRC will be forwarded
11:37:31 INFO: packets received with a CRC error will NOT be forwarded
11:37:31 INFO: packets received with no CRC will NOT be forwarded
11:37:31 INFO: GPS is disabled
11:37:31 INFO: Upstream data is enabled
11:37:31 INFO: Downstream data is enabled
11:37:31 INFO: Ghoststream data is disabled
11:37:31 INFO: Radiostream data is enabled
11:37:31 INFO: Statusstream data is enabled
11:37:31 INFO: Beacon is disabled
11:37:31 INFO: Packet logger is disabled
11:37:31 INFO: Flush output after statistic is disabled
11:37:31 INFO: Flush after each line of output is disabled
11:37:31 INFO: Watchdog is disabled
11:37:31 INFO: found local configuration file /var/config/lora/local_conf.json, parsing it
11:37:31 INFO: redefined parameters will overwrite global parameters
11:37:31 INFO: /var/config/lora/local_conf.json does not contain a JSON object named SX1301_conf
11:37:31 INFO: /var/config/lora/local_conf.json does contain a JSON object named gateway_conf, parsing gateway parameters
11:37:31 INFO: gateway MAC address is configured to 00800000A000372A
11:37:31 INFO: Found 1 servers in array.
11:37:31 INFO: Server 0 configured to "bridge.us-west.thethings.network"
11:37:31 INFO: packets received with a valid CRC will be forwarded
11:37:31 INFO: packets received with a CRC error will NOT be forwarded
11:37:31 INFO: packets received with no CRC will NOT be forwarded
11:37:31 INFO: GPS is disabled
11:37:31 INFO: Upstream data is enabled
11:37:31 INFO: Downstream data is enabled
11:37:31 INFO: Ghoststream data is disabled
11:37:31 INFO: Radiostream data is enabled
11:37:31 INFO: Statusstream data is enabled
11:37:31 INFO: Beacon is disabled
11:37:31 INFO: Packet logger is disabled
11:37:31 INFO: Flush output after statistic is disabled
11:37:31 INFO: Flush after each line of output is disabled
11:37:31 INFO: Watchdog is disabled
11:37:31 INFO: Contact email configured to "contrail.support@onerain.com"
11:37:31 INFO: Description configured to "Multitech Conduit LoRa gateway"
11:37:31 INFO: [Transports] Initializing protocol for 1 servers
11:37:31 ERROR: [TTN] Connection to server "bridge.us-west.thethings.network" failed, retry in 30 seconds
11:38:01 ERROR: [TTN] Connection to server "bridge.us-west.thethings.network" failed, retry in 60 seconds
I’ve also tried enabling debug_pkt_fwd in my local_conf.json as below, but to no avail:
{
"debug_pkt_fwd": true,
/* Settings defined in global_conf will be overwritten by those in local_conf */
"gateway_conf": {
/* gateway_ID is based on unique hardware ID, do not edit */
"gateway_ID": "00800000A000372A",
/* Email of gateway operator, max 40 chars*/
"contact_email": "redacted",
/* Public description of this device, max 64 chars */
"description": "Multitech Conduit LoRa gateway",
"gps": false,
"fake_gps": false,
"servers": [
{
"serv_type": "ttn",
"server_address": "bridge.us-west.thethings.network",
"serv_gw_id": "or_multitech_conduit",
"serv_gw_key": "ttn-account-redacted-but-looks-correct",
"serv_enabled": true
}
]
}
}There is some sort of connectivity issue to TTN. The debug flags won’t help you because that library does not provide additional debugging output.
Please go to slack and report your issue there so TTN operations can look into it.
Yes, if you have not changed anything there is either an issue at TTN side or in your network (or between you network and TTN)
Thanks for the hint on the slack, i’ll post over there. Hard to keep track of where to go for support! Seems like there 10 places between the forums, the support site, the twitter, the slack, etc…
Support site?