Problem - ttn-router-us-west

TTN Network and Services
1

Hi,
Since approx 15 hours, look like off of my devices (5 applications) are offline. I can see devices uplink to my gateway but no downlink. All worked great before yesterday.

Anyone have the same problem ? I looked status TTN and look working great

Thanks

2

I am in the same boat here. 6 devices, geographically dispersed connecting to 5 different gateways. All went dark approx 15h ago. I can see encrypted traffic from the devices hit the gateway, but no application data for a system that has been running for ~2 years. One device+gateway is sitting here in front of me, if I power cycle it I can see the join messages in the application console, then it joins and starts transmitting data but no data is reaching the application @ TTN.

3

ok and what’s the reason of this problem ? Like you, I have 3 gateways running 22 devices with no problem over 1 year ago but problem since 15 hours ago

4

Logs on everything I can touch from my side look like they are working. LoRaWAN-attached MCU says it’s sending packets, gateway says it’s receiving packets, and gateway says it’s sending packets to TTN. TTN console reports my gateway is sending those packets, but my configured applications are not receiving them. Appears to be something on TTN side.

5

is there an official thread for this problem? We are probably not the only 2 ?

6

is it an integration that’s not working ?
because traffic flows normally (EU)
workssunday

if I see where you all coming from, it looks like the ‘old’ US problem, or are there EU users experiencing the same at this moment ?

7

I have two in Boston, USA. One went offline 17 hours ago. The other reports still being connected but I haven’t tested it.

8

I get nothing in application and is not about integration. All work great since overs 1 year, no problems but now look from 16-17 hours I have nothing to the application

43 54

1 Like
10

Yes same here.

Devices and gateway showing “not connected” since 23 March 2019 ~18:20 EDT.

For the gateway… factory reset, and reactivating did not correct the issue. Factory reset and activating it as a new gateway did work.

Devices are sending traffic to to the “new” gateway, but still showing as “not connected” to TTN.

11

Here is California same issue. I can OTAA sign in new devices, but data does not show up since 20 hrs ago. Multiple integrations do not work. Checking TTN network status shows all systems operational.

12

I really hope to see a fix coming quickly :crossed_fingers:

14

All of my gateways are set to “ttn-router-us-west”. I have not tested to change it

15

Similar issue here in the US also using ttn-handler-us-west

I can see join requests but no data from nodes at the application level.

I also have to query the last 19 hours from storage in order to see data.

We seem to have a lot more issues in the US than the EU :thinking:

16

@KrishnaIyerEaswaran2 wrote in Slack #ops just now:

does anyone here access to the console and can check their devices now?

17

Yes, everything works fine through ttn-handler-eu and doesn’t work with ttn-handler-us-west

18

Same problem for me, from two different gateways. I am using the US setup too.

19

And just now, @KrishnaIyerEaswaran2 fixed things :partying_face:

handler stream was stuck. had to enable deep debug logs to catch that. We usually disable those to prevent cluttering of the logs.

Thanks for your inputs guys. Enjoy your sunday!

3 Likes
20

Well done!
Thanks

21

great ! :partying_face::muscle::clap:

2 Likes
22

thanks