The Australia cluster will undergo an infrastructure upgrade tomorrow, Tue 1-Dec at 4:00pm AEST. If all goes well, outage time will be less than 10 minutes.
I note that after the previous infrastructure upgrade a number of gateways could not reconnect to the server due to stale connections. We’ll be on the lookout for this issue and can address it at the server end if it happens again.
Please reply to this topic on the day if there are any issues, or for more immediate communication join us on the #Australia Slack channel.
3 Likes
Unfortunately we are experiencing issues with the TTN services (even before we started the intended upgrade). Currently waiting on the TTN team to advise.
1 Like
I’ve been pulling my hair out for the last 24 hours thinking I had managed to stuff up every sketch I was working on somehow. 
Thanks for the hardwork. Hopefully we’re back online soon. 
Ah, thanks for the info, well that more than likely explains my MAJOR forwarding issues in the last 2 days. I too have had a sleepless night looking for errors in code and set-up. How is the best way to get updates on the “issues status”?
Apologies, I never closed the loop on this issue. Problems started at 4pm AEST on 1-Dec, before we even got a chance to upgrade. The issue was a DDoS attack on a Netherlands ISP, which caused one of the global TTN services to become unavailable.
The issue was resolved about 3 hours later, we performed the upgrade and brought the services back online by about 8pm AEST on 1-Dec.
We are testing office sensors and people counter sensors (about 3 weeks) and they have been working really well until early this week (1st December)
Since the 1st December no payload packets are passing through from any of our test sensors - Sensors/Gateway all appear to be working as expected however the device status only shows the last connection as 3 days ago.
My nodes are able to join OK and uplink packets are appearing in the gateway console but packets are not appearing in the device console for any of my 2 nodes, nor is data being forwarded to my platform. This started happening about Dec 1. The the frame counters seem to be correct on the gateway console but show ‘0’ on the device console for up and down.
After some research I decided to change the router from eu to asia-se on both my gatway and nodes but still the same result. I have reset my nodes several times and I have have no problems getting a join accept showing on the gateway console but nothing on the device console. As my platform has not seen any new data since Dec 1 I can only assume the router is not forwarding the packets.
1 Like
I can confirm that I am experiencing the exact same issue as outlined by Malcman. Thanks David for your explanation of the issue. Still an issue for me at the moment no new data has been received since Dec 1 - hopefully it will be resolved over the weekend.
1 Like
I can also say I have the same issue across at least 3 gateways. Our gateways and apps are on asia-se so nothing to do with Australian cluster upgrade.
We’ve been working on this throughout the morning. Unfortunately we haven’t found the issue yet. None of our test devices or gateways are affected. It’s affecting some devices on some gateways, but unfortunately it’s inconsistent.
Note that unless your gateway is configured for meshed-router and your application for meshed-handler, we are probably talking about different things.
@Malcman, @EvolvePlus, can you provide a DevEUI and GatewayEUI for an affected device
After hearing from someone else with similar issues I’m starting to think this is a cross-region routing issue.
See
Does anyone have a meshed-handler/meshed-router combination that is failing?
Thanks for your assistance Andrew.
The affected device DevEUI is: A81758FFFE03FD23 and the Gateway EUI is eui-c0ee40ffff297f46. The gateway is configured for meshed-router
Oh sorry Andrew - I forgot to mention: Handler = ttn-handler-asia-se
DevEUI: A8610A34343D8512
GatewayEUI: eui-dca632fffe08e182
I now have my nodes running again! they are uploading and forwarding to the platform.
I changed from “asia-se” to “thethings.meshed.com.au” on both gateway and nodes. Thank you so very much Maj.
I am hoping the au meshed routers are more reliable than the others for us in Australia.
What was the issue with the other routers except for the cross-region routing you mentioned?
The problem is with the code that routes from one region’s gateways to another region’s apps. It’s not a problem with any particular region. Now that you’re on one region only, cross-region routing is not required, so everything works 
1 Like
YES! Thank you so very much Maj. Switching over to “thethings.meshed.com.au” for the devices has solved my problem. Cheers!
In regards to the asia-se issue. The issue solved itself. Nothing was changed on our side it just started working again.