Anyone got any advice/clues at all? Either to fix the application MQTT integration or somehow reset it (I know all this is an open-source freebie but is there any email address I could try?). This permanent failure of an application is really a problem for us, and I’m about to deploy 400 sensors in a research project but really should hold off until this is worked out.
Thanks.
Just as side thought and given the scale of what you are doing and no doubt your reliance on the infrastructure, would it make sense to be working with the The Things Industries - we have dozens of nodes but 400 plus really needs the platform backup support in place I would have thought.
As @jezd says, or run your own backend …
… OR add BOTH data storage AND HTTP Integration to backup the MQTT.
It seems issue is resolved now, MQTT integration working back on all of my applications . Please check if its same on your end too. Meanwhile i have been shifted to http integration and keeping that too as a backup if MQTT stops working in future .
Just run all three if you have a community benefit system critical application.
If it’s commercial and system critical, run your own servers!
I wonder:
Yup, just seen & replied to that!
Thanks for the helpful posts here. Our ‘TTN application’ still fails to send data to new MQTT subscriptions (although oddly the existing connections from before ~Sept 1st continue to run).
I’ve contacted TheThingsIndustries to see what we might do with them. This isn’t as simple as building our own network server and adding a user-friendly front-end (we did that in previous generations of our network) because we’re covering a large region with our gateways and want to continue to support public access - in practice that helps research usage across the university also as TTN is a well-known LoraWAN point-of-access and we’re happy to support it with our gateways.
If we fall back to the HTTP connection (which we used before MQTT) I’ll really need to implement our own “MQTT-HTTP bridge to TTN” as we have messages flowing around on our platform using Mosquitto/MQTT (e.g. messages coming from Wifi and Zigbee sensors) - if anyone’s aware of an existing open-source implementation please let me know. I know the basic idea is simple (a POST arriving from TTN results in a PUB to Mosquitto, and something similar in reverse) but writing all this stuff from scratch adds up a lot of work (e.g. we have an ok decoders system as an alternative to the TTN javascript per-application decoder).
Have you tried to add an additional, new access key to the TTN Application experiencing the MQTT issues and then use the new access key for new connections, just to see if that works?
Are you currently using MQTT bridging?
Using your own MQTT server for your applications makes them less dependent on TTN’s MQTT server. Each message will then have to be transferred from TTN only once - from TTN’s MQTT server to your own (local) MQTT server. Your applications can then communicate with your (local) MQTT server.
But for the bridging to work also requires that new MQTT connections for your TTN Application (from your MQTT server to TTN’s MQTT server) must work successfully.
To prevent naming confusion:
Thanks bluejedi - good suggestion but unfortunately using the new key immediately fails with “Connection Refused: not authorised” and the client (mosquitto) disconnects.
Attempted new MQTT ‘mosquitto_sub’ connections to receive messages from the TTN Application with the original (default) access key continue to hang, with no messages being returned.
I am using Mosquitto MQTT ‘in’ bridging as the preferred method of connecting my Linux servers to the TTN MQTT broker to receive messages, these are (unsurprisingly) failing to connect to TTN in the same way as the Linux terminal ‘mosquitto_sub’.
Existing connections to the TTN MQTT broker are still receiving the data but understandably these are gradually dropping off for local reasons and failing to reconnect.
So the TTN Application MQTT access seems fairly seriously borked, and I’ve deferred my install of the additional sensors until I have an effective migration plan if this happens again. I can run a HTTP client but as the MQTT access seems permanently dead then I have a new dependency on HTTP so I still need a plan B. It would be easy enough to run our own server/platform but we want to continue to support public use of our gateways via the TTN console as that works really well not only for our region but also within the university for the majority of projects which are typically small-scale (like single sensors…).
I’m talking to The Things Industries and they have a proposed solution which I need to look into more carefully.
In any case I’m assuming I should maintain the list of our sensor registration data and implement a program to register all those sensors via a TTN api, so I can re-register them easily to another TTN application if this failure occurs a second time. I’m not sure how the sensors (mainly Elsys, RadioBridge) will behave on a TTN Application swap though, but I can test that.
It sure looks like it! The question is what’s borked?
There is no sign on the forum that this is affecting other users.
I have a test TTN application that has a single GPS tracker device on it that sends its location every 15mins. I use MQTT to subscribe to this feed both from Linux/mosquitto_sub and also from a mosquitto broker/server acting as a bridge so that I can change the topic prefix. Both of these are working correctly and I have rebooted both in the last 24hrs with no issues.
If you want to pm me, I will set up a new API key for you to access the TTN MQTT feed for this test application and you can see if it works.
If it works then the bork looks to be in TTN associated with your user/application.
If it doesn’t work then the bork may be in your infrastructure.
Strange, I’ve been using it for three years without the issues your experiencing.
Have you considered going to a totally different place / lab / office / home with a machine that’s never done anything like MQTT and get someone technical but unrelated to the project to try to subscribe.
Or even better, get the technical someone a number of devices, not just the ones you are using in your project and get them to setup a new TTN account, setup an application, configure the devices and then subscribe to the MQTT feed. Which pretty much describes a day in the life of a new TTNer!
I have a new TTIG just arrived to take apart for fun & profit and some new PCB’s for a new TinyThing variant, so I can do exactly that and document it along the way.
the OP of this thread (not me) had an issue from Sept 2nd, which is when we were affected so there’s at least a couple of us, but probably more.
Important clarification: I don’t mean to suggest the TTN MQTT broker is ‘borked’ for everyone… our issue is limited to a single application (unfortunately our main one) and most people must be unaffected.
We’ve been using LoraWAN for 6 years and TTN for a couple of years - there have been very few issues. The basic idea of receiving the data via an HTTP POST or MQTT subscription is something we’ve implemented many times over, from many different clients.
The simplest test is via a ‘mosquitto_sub’ connection to TTN, and I must have done that successfully hundreds of times by now. From ~Sept 2nd one of our TTN applications stopped accepting new subscriptions to MQTT messages, and the TTN web console shows no messages coming from those sensors either, BUT existing clients with MQTT subscriptions to that application are continuing to receive the messages. If I restart a linux server with a successfully working Mosquitto bridge connection to my TTN application, that machine will then permanently fail to reconnect.
I’m fairly confident it’s the ‘TTN Application’ that’s failing in some way (it would be hard to explain the data not appearing in the console otherwise). But I’m assuming it’s only a few of us affected so we’ll need to move on with our own failover plan to a new TTN Application.
But genuine thanks for the alternative connection tests that have been offered.
Once upon a time, in a business far-far-away, I had 550 customers all using a booking system just fine.
Except for one. Who worked at a company with networking specialists looking after the network. After some under-the-hood diagnosis, we figured out that the company firewall was messing with my load-balancer.
Maybe something is jamming up both MQTT and XHR?
So I would urge you to try someone else network at the very least as it may well uncover something peculiar about the network you are running on that no one has considered an issue.
thanks - given lockdown we have clients connecting from multiple home locations (which were ok before).
I do think we have a common access path via Mosquitto (because we’re mainly using Mosquitto bridges to connect to TTN and also testing with mosquitto_sub), and I’ve tried forcing the ‘mosquitto_sub’ to a given MQTT version, (i.e. -V mqttv31) but no joy. But if a Mosquitto/TTN incompatibility had crept in e.g. as a result of an upgrade and all our clients were up-to-date, that would cause a problem to span our multiple production and testing clients. But I think a lot more people than us would have noticed that.
However, the simplest symptom to report is “no sensor data messages for a given TTN Application displayed in the TTN web console, but being received ok on MQTT clients connected before Sept 2nd”
Is there anything in the data stream that stops someone trying?
nothing at all… so we’ve tried connecting from multiple clients on multiple networks including people’s homes.
See my “simplest symptom” comment above. To me that suggests a broken TTN Application. I don’t mean for every application on TTN, just that one.
From today the TTN MQTT is not working in EU.
Yesterday it working and received data, but today no connection
I am receiving uplinks.
Can you tell us what feedback / error message it is that tells you that MQTT is down?
Then we have something to work on.