Very strange issue we are facing since 2nd September,2020. MQTT integration not working only for one of my application, it has been working without any issue so far. MQTT settings, access key all are same but no data over mqtt integration from this ttn application. Is there any change or outage for ttn mqtt services in europe region. Please share.
Is it because of this ? 
Hi @interravikram, no problem here with my EU region MQTT subscriptions.
I suggest that you try subscribing interactively using the mosquitto_sub utility with the -d [ebug] flag set and observe if the MQTT connect and subscribe works and if there is a flow of uplinks as you expect.
Have you fully stopped and re-started your MQTT subscriber to ensure that it is connecting to the right MQTT server since the incident on Sept. 2nd?
1 Like
Yes i have restarted and its strange because for all other applications mqtt integration is working . Issue is only on one application.
Issue not resolved, hence shifted to http Integration.
This sounds very similar to a serious issue weāre having. Actually existing TTN MQTT connections are still receiving data, but any attempt at a new connection hangs without receiving any new data.
Any hints or help will be much appreciated as currently unable to create any new subscription to the data means we are ultimately going to fail in a serious way.
We have data coming from ~30 sensors and can see the data in the archive from the MQTT subscribers that are still running, but:
-
if we access the console and select āApplications ā appname ā Dataā then no data appears there
-
if we access the console and select āApplications ā appname ā Devices ā devicename ā Dataā then no data appears there.
-
If we subscribe with a new client, we get no messages, e.g. with mosquitto_sub -d ā¦ :
mosquitto_sub -v -h āeu.thethings.networkā -p 1883 -u ācambridge-sensor-networkā -P āttn-account-v2.rest-of-the-keyā -t ā+/devices/+/upā -d
The ā-dā output is a continuous sequence as below with no data and the sequence repeating once-per-second with the āMidā number incrementing (even though the existing connected clients are receiving sensor data messages ok):
Client mosqsub|32618-tfc-app6 sending CONNECT
Client mosqsub|32618-tfc-app6 received CONNACK
Client mosqsub|32618-tfc-app6 sending SUBSCRIBE (Mid: 1, Topic: +/devices/+/up, QoS: 0)
Client mosqsub|32618-tfc-app6 sending CONNECT
Client mosqsub|32618-tfc-app6 received CONNACK
Client mosqsub|32618-tfc-app6 sending SUBSCRIBE (Mid: 2, Topic: +/devices/+/up, QoS: 0)
ā¦ this continues indefinitely
Hi @CambridgeSensorNetwo, I have just tried a new console connection (W10, chrome) and a new MQTT subscribe (mosquitto_sub -dā¦) to an existing app with existing devices on EU.
Both are working normally, showing uplink data as expected.
I guess that the fault that you are seeing is only affecting some users or applications or connections.
thanks - all info is helpful. Iāll test if another TTN āapplicationā for my user also sends MQTT data ok - Iām guessing it will.
Weāve been running sensors on TTN for multiple years without any real issues, so this is new for us.
Iām considering two fallback plans (I have an additional 400 sensors delivered this week, so I have to do something)
- Implement a HTTP POST receiver like the OP .
- Find out if I can download all my sensor registrations into a file and upload them into another application and move on from there.
Of course Iād prefer a way to kick the MQTT feed back into life.
Anyone got any advice/clues at all? Either to fix the application MQTT integration or somehow reset it (I know all this is an open-source freebie but is there any email address I could try?). This permanent failure of an application is really a problem for us, and Iām about to deploy 400 sensors in a research project but really should hold off until this is worked out.
Thanks.
Just as side thought and given the scale of what you are doing and no doubt your reliance on the infrastructure, would it make sense to be working with the The Things Industries - we have dozens of nodes but 400 plus really needs the platform backup support in place I would have thought.
1 Like
As @jezd says, or run your own backend ā¦
ā¦ OR add BOTH data storage AND HTTP Integration to backup the MQTT.
It seems issue is resolved now, MQTT integration working back on all of my applications . Please check if its same on your end too. Meanwhile i have been shifted to http integration and keeping that too as a backup if MQTT stops working in future .
Just run all three if you have a community benefit system critical application.
If itās commercial and system critical, run your own servers!
I wonder:
Yup, just seen & replied to that!
Thanks for the helpful posts here. Our āTTN applicationā still fails to send data to new MQTT subscriptions (although oddly the existing connections from before ~Sept 1st continue to run).
Iāve contacted TheThingsIndustries to see what we might do with them. This isnāt as simple as building our own network server and adding a user-friendly front-end (we did that in previous generations of our network) because weāre covering a large region with our gateways and want to continue to support public access - in practice that helps research usage across the university also as TTN is a well-known LoraWAN point-of-access and weāre happy to support it with our gateways.
If we fall back to the HTTP connection (which we used before MQTT) Iāll really need to implement our own āMQTT-HTTP bridge to TTNā as we have messages flowing around on our platform using Mosquitto/MQTT (e.g. messages coming from Wifi and Zigbee sensors) - if anyoneās aware of an existing open-source implementation please let me know. I know the basic idea is simple (a POST arriving from TTN results in a PUB to Mosquitto, and something similar in reverse) but writing all this stuff from scratch adds up a lot of work (e.g. we have an ok decoders system as an alternative to the TTN javascript per-application decoder).
Have you tried to add an additional, new access key to the TTN Application experiencing the MQTT issues and then use the new access key for new connections, just to see if that works?
Are you currently using MQTT bridging?
Using your own MQTT server for your applications makes them less dependent on TTNās MQTT server. Each message will then have to be transferred from TTN only once - from TTNās MQTT server to your own (local) MQTT server. Your applications can then communicate with your (local) MQTT server.
But for the bridging to work also requires that new MQTT connections for your TTN Application (from your MQTT server to TTNās MQTT server) must work successfully.
To prevent naming confusion:
- Application is any kind of software application.
- TTN Application is the āapplicationā that you define on the TTN console, which serves as ācontainer/parentā for its configured devices.
Thanks bluejedi - good suggestion but unfortunately using the new key immediately fails with āConnection Refused: not authorisedā and the client (mosquitto) disconnects.
Attempted new MQTT āmosquitto_subā connections to receive messages from the TTN Application with the original (default) access key continue to hang, with no messages being returned.
I am using Mosquitto MQTT āinā bridging as the preferred method of connecting my Linux servers to the TTN MQTT broker to receive messages, these are (unsurprisingly) failing to connect to TTN in the same way as the Linux terminal āmosquitto_subā.
Existing connections to the TTN MQTT broker are still receiving the data but understandably these are gradually dropping off for local reasons and failing to reconnect.
So the TTN Application MQTT access seems fairly seriously borked, and Iāve deferred my install of the additional sensors until I have an effective migration plan if this happens again. I can run a HTTP client but as the MQTT access seems permanently dead then I have a new dependency on HTTP so I still need a plan B. It would be easy enough to run our own server/platform but we want to continue to support public use of our gateways via the TTN console as that works really well not only for our region but also within the university for the majority of projects which are typically small-scale (like single sensorsā¦).
Iām talking to The Things Industries and they have a proposed solution which I need to look into more carefully.
In any case Iām assuming I should maintain the list of our sensor registration data and implement a program to register all those sensors via a TTN api, so I can re-register them easily to another TTN application if this failure occurs a second time. Iām not sure how the sensors (mainly Elsys, RadioBridge) will behave on a TTN Application swap though, but I can test that.
Hi @CambridgeSensorNetwoā¦
It sure looks like it! The question is whatās borked?
There is no sign on the forum that this is affecting other users.
I have a test TTN application that has a single GPS tracker device on it that sends its location every 15mins. I use MQTT to subscribe to this feed both from Linux/mosquitto_sub and also from a mosquitto broker/server acting as a bridge so that I can change the topic prefix. Both of these are working correctly and I have rebooted both in the last 24hrs with no issues.
If you want to pm me, I will set up a new API key for you to access the TTN MQTT feed for this test application and you can see if it works.
If it works then the bork looks to be in TTN associated with your user/application.
If it doesnāt work then the bork may be in your infrastructure.
Strange, Iāve been using it for three years without the issues your experiencing.
Have you considered going to a totally different place / lab / office / home with a machine thatās never done anything like MQTT and get someone technical but unrelated to the project to try to subscribe.
Or even better, get the technical someone a number of devices, not just the ones you are using in your project and get them to setup a new TTN account, setup an application, configure the devices and then subscribe to the MQTT feed. Which pretty much describes a day in the life of a new TTNer!
I have a new TTIG just arrived to take apart for fun & profit and some new PCBās for a new TinyThing variant, so I can do exactly that and document it along the way.