We have a tank sensor, and the data is getting recorded against a device in our app.
The tank sensor breaks and we replace it. New TTN device with new deveui, devId, maybe even appId.
We want to data from the new TTN device to go to the same device in our app.
So we need a way to map a TTN device to a device in our app.
ATM we have a database table that does this using the TTN appId and devId. This requires maintenance when we swap devices in the field, and we need to run the database. And we’re moving away from our in-house platform to ubidots.
Today we brainstormed an idea where we could use the TTN device name (which we can carry over to the replacement device and change on the old device) as a more automatic way of doing this mapping.
Then we realised the device name doesn’t come in the message from TTN.
I just looked at webhook variables and I can’t do it that way.
Is there a way to get TTN to include extra device metadata, either standard info like the device name or one of the user-defined properties in the message it sends to a webhook or mqtt topic?
I would prefer not to have to make a REST auth then API call for every message we receive. I don’t think we can cache the auth token due to the runtime environment we’re moving to (an ubidots ubifunction).
More generally I’m puzzled as to how little I see this discussed. It seems to me that if you want to do historical analysis over a time period where a physical device has failed and been replaced you need to be able to stitch the timeseries data from each device into a single stream, or have it recorded in a single stream through some sort of logical device layer as we do in our current system.
There are more requests on the forum for removal of fields than this & a couple of others for adding in fields. It’s huge already.
If an install is big enough to have devices swapped out, then I’d suggest they can figure out remapping on their side - I have a number of integrations that pre-process and pass it on to the destination. Or we all end up with feature creep and eventually we’ll get the veritable kitchen sink being sent.
Or come up with a plan for three levels of payload:
Small - just the payload and device EUI
Medium - as above plus the useful parts of the gateway + signal strength
Large - above plus everything else plus the correlation id’s
I agree - it’s like everyone gets the data to the application server and then … nothing.
Maybe because the dashboards are in someone else’s department.
I do sympathise with your technical challenge but my ingestion databases have ballooned with the new v3 format so adding fields very much feels like going in the wrong direction.
I agree regarding the kitchen sink being added into the standard message.
That is why my first thought was to look at the webhook variables. I think the options there could be extended without too much code complexity in the stack and without impacting the standard message format at all.
A few more standard device values and a way to include a user defined prop value by its key might go a long way.
As it is, the things you can include are all in the message so aren’t that useful IMO.
We have a home grown solution ATM but would rather get rid of that layer of complexity from our plate and give it to someone else
We are changing platforms to reduce the amount of our own software and it would be nice to get rid of this bit and only have to change the device details in Things Stack.
We could make the REST calls from ubidots to get the device name but that adds load to both ubidots and Things Stack for every message.
The Console only provides an interface that maps the backend functionality and I don’t think we currently allow decorating the payload with additional props. So I cannot help with this from the frontend side. @htdvisser Do you have some feedback on this?
I like @DavidTaylor_DPI’s idea about adding additional variables to the path as long as they are in Redis already - and if it’s feasible to reduce the payload down to a reasonable minimum and then add a template in to the path so that we can include them as query variables that can be parsed easily, all good. Given the likelihood that most users will have unique DevEUI’s, we could drop the AppEUI and JoinEUI as this is static data - and put the Device ID in its place.
But overall, I’d rather get my payload with the minimum of stress to the system to ensure Webhooks are the second to last man standing (Data Storage being first as a backup) than increase the workload.
If you were to do anything in this area, it would be to resolve the rather bizarre issue that the SLA for TTI says you’ll process the data 99.9% of the time but provide no SLA on us actually receiving it - so if Webhooks has to be slimmed down and the Data Storage folded in to the SLA as a minimum, that would be sellable to the clients.
Another solution would be for the payload formatter to get a device object in as another entry in the input object.
The formatter code has to reference the bits of the input object by name so it wouldn’t break any current code.
Then the formatter author has the choice of what extra info they want to send. Even better if the return object can have more than the data, warning, and error fields in it that will get sent to the receiver.
But we could get away with knowing the deviceName field we put in the data object is magic and not a piece of timeseries data and handle it in the webhook code.
The difficulty here is that end device information is in two places. All “live” information (basically everything that’s needed for doing LoRaWAN) is in the Application Server, and all “meta” information (name, description, attributes, location, …) is in the Identity Server.
However, the Application Server already fetches (and caches) the location for end devices, so I think it should be possible to also do this for the attributes (which some integrations use to match a device registration in The Things Stack with a device registration in the external platform). Then we could add those attributes to each uplink as well.
If there are concerns about the size of webhooks, I suppose we can apply some field mask before sending them (this would only work for webhooks, not for MQTT).
I’m guessing the payload formatter runtime environment is as stateless as possible by design. Perhaps due to what is explained above about the app/id servers, or so the formatter can be run anywhere, or so users don’t go crazy and write hugely complicated formatters that bog down the servers.
Is that also the case for the webhook URL variable feature? The variables we can use are the ones that are locally available from the application server and the things I’m asking for would have be to fetched from the identity server?
Erm, they already do that and it’s already had lots of work on it and it still does occasionally bog down the servers and times out the JavaScript. Multiple forum posts plus at least one GitHub issue.
Hylke is one of the designers / architects / developers of this part of the stack so he absolutely knows what is available in Redis for immediate access regardless of its origin. His middle paragraph implies that similar things are done for other webhook integrations so could be potentially possible for custom ones.
The fundamentals for me are, get a hold of my payload and do the rest on my own servers, even if they just pre-process and relay on to a third-party service. Payload formatters time out when there is a perfect storm of uplinks arriving, regardless of how small mine are, so I decode on my servers, or at least can if it hasn’t decoded. This timeout occurs sufficiently frequently that I can’t just drop the uplinks that haven’t been processed.
In addition, any form of intermediate server allows the opportunity for the raw data to be stored in a database lest the loverly third-party service have an outage, data loss or go out of business. For solutions where I am providing the end-point, I run two web hooks, one to the primary and one goes to backup raw database on a totally different ISP.
This way seem over the top, but the only reason devices, uplinks, TTS, TTI etc exist is to procure data to report & act on. Every reasonable effort should be made to preserve the data in perpetuity as it came from the device. Anyone who doesn’t will find out why one day when they want to go back and fix a decoding error or re-review raw data to look for patterns that happen annually.
Back on the specific topic, perhaps if we come up with a proposal and then submit to GitHub. It will take me a couple of days to look under the hood and what benefits we accrued with TTS’s implementation of web hooks vs TTN (v2) vs formats I’ve used over the last ~20 years.
What is the symptom of the payload formatters in the stack not working? I’d like to know what to look for.
We have a similar setup to you at present - we get the messages via MQTT and store them in a db, decode the fields, and send the values on to the timeseries data store. It’s a bit simpler but about the same idea.
We’d like to get away from that second part for live data but your comments about payload formatters timing out are a bit of a worry. Is that because you’re in Europe and it’s busy?
I intend to replace the current system that has loads of mosquitto_sub processes with a webhook endpoint to write to the 'every message we ever received" database.
We’re still debating on whether we need to do decode the fields locally or can keep it in the stack app.
If we can’t get the device name in our message then we might end up having to keep a simplified version of our current software that can just map the appId/devId to the ubidots device.
It’s not a “not working” thing, it’s a time out thing. If the component that processes payload formatters is under load - mostly due to a sudden surge in uplinks mostly due to co-incidence, then they are more prone to timeout - currently set to 100ms.
This was an issue with v2 and became an issue with v3 about 200ms after it was made available to the community, some whom have 400KB+ globs of JS!
You don’t get the payload fields ie decoded_payload is empty - or has warnings.
I’d expect the server instances by region to be scaled appropriately - so we may have more powerful ones than you but all regions would be scaled to suit - so I’d expect it to happen anywhere.
I don’t have current stats as I haven’t migrated all of my tools and it sort of doesn’t effect me anyway as I turn it off for production use and for development use I don’t care.
Cool - MQTT is harder on the servers than Webhooks and confers no real benefits as QOS is set to 0 and a web server is very scalable all by itself.
I use server side whilst coding but deploy using my side decoder.
As I mentioned previously, having an intermediary comes with the benefit of being able to store “everything for ever” in a data store you can backup yourself so if a third party has a bad server day, you’re not entirely lost.
One other possibility is that Ubidots provides a mapping function - you could ask them for that so you have two options running at once - one here & one with them - it’s not an unusual issue so others may have had to swap out devices. And you can then send your data over to Ubidots directly but still have a Webhooks for your backup data store.
We’ve found various problems in our timeseries data, including but not limited to decoder bugs.
So we’d like to recreate it for use in analyses and luckily the last guy set up a backup process that means we’ve been able to recreate the raw message set and can now hunt down the dups, re-run fixed up decoders, and see if we get a better data set.
We were always going to keep a local webhook to update this raw message table but it’s sounding like we need to consider keeping the decoders running locally as we do now too.
We’ve kept thinking about this and the latest idea was to use the device location and map that to our logical location-based device. We set the lat/long on some devices using the location tab in the console on a few devices.
We’ve noticed the location info may be dropped from the JSON messages sent to the webhook. We added it 5 days ago and for 4 days all the messages had it, then yesterday it started getting patchy, sometimes dropped every few messages.
If the location info is getting cached in the app server, why would it be dropped from the message so often?
For our own webhook we could try caching the lat/long values mapped to a deveui to handle the messages where it’s missing, but we can’t do that on ubidots because I think every message received is run in a random instance of a VM so they have to be stateless.
I view TTN as the transport layer for my data. TTN knows about gateways and devices, and gets my data to my application.
TTN does not need to know that a particular node is measuring temperature in a calf rearing shed in New Zealand, or that a particular water tank is only half full and another 200m away is empty. I’m happy with the device id and the measurement data. The extra metadata that TTN provides about snr, rssi, gateway are a bonus feature.
I keep my devices and measures separate. Outdoor temperature and shed temperature are measures. Temp sensor nodes 1, 2 and 3 are devices. If devices are equivalent then they are interchangeable and can be used for any appropriate measure.
I’m already using Node Red and influx to decode and record my data. My influx tables record measurements not device data. I have a static table in influx to link devices and measures. Node Red uses this table to associate the device with the measure. I can also add other static metadata from the table before the measurement is recorded in influx.
Once a device is placed, measure name, location, sensor type etc. are all static for more than a year. If I need to, can take a sensor node out of service, update the table in influx, and bring a new sensor node into service without disturbing the measurement.
I don’t need accurate calibration, but if the sensors had individual calibrations, this could be part of the device metadata. Calibration could then be applied to the measurement data in Node Red before recording the adjusted values in the database.
BTW. I also record the raw data from each device in influx just in case I need it.
The same answer keeps coming back so I guess we need to take it on
We have essentially the same mechanism in our current system but were hoping to simplify and ditch as much in-house software as possible, but it seems we’ll need to keep this device mapping function around.
Thanks everyone, it’s interesting to see how other people do it, and we’ve all arrived at about the same place.
To be fair, if we add other transports we probably wouldn’t have a lot of context from those devices either so if we ever did that the problem would show up then.
The only way I found to do it is to use “javascript payload formatter” (one different for each device), that "add the device name into the json message ":
function decodeUplink(input) {
message = {
nickName: "my device name here !!!"
}
return {
"data": message,
}
}
That’s works, but it is a trick! I would appreciate a more efficient and scalable solution.
