Convert custom payload to lpp payload in TTN console

Refer to the “Data Types” section here
https://mydevices.com/cayenne/docs/lora/#lora

The uplink Payload Structure is as follows

1 Byte 	        1 Byte 	        N Bytes 	1 Byte 	        1 Byte 	        M Bytes 	…
Data1 Ch. 	Data1 Type 	Data1 	        Data2 Ch. 	Data2 Type 	Data2 	

Each data type can use 1 or more bytes to send the data according to the following table.

Type 	                IPSO 	LPP 	Hex 	D.Size 	Data Resolution per bit
Digital Input 	        3200 	0 	0 	1 	1
Digital Output 	        3201 	1 	1 	1 	1
Analog Input 	        3202 	2 	2 	2 	0.01 Signed
Analog Output 	        3203 	3 	3 	2 	0.01 Signed
Illuminance Sensor 	3301 	101 	65 	2 	1 Lux Unsigned MSB
Presence Sensor 	3302 	102 	66 	1 	1
Temperature Sensor 	3303 	103 	67 	2 	0.1 °C Signed MSB
Humidity Sensor 	3304 	104 	68 	1 	0.5 % Unsigned
Accelerometer 	        3313 	113 	71 	6 	0.001 G Signed MSB per axis
Barometer 	        3315 	115 	73 	2 	0.1 hPa Unsigned MSB
Gyrometer 	        3334 	134 	86 	6 	0.01 °/s Signed MSB per axis
GPS Location 	        3336 	136 	88 	9 	Latitude : 0.0001 ° Signed MSB
                                                        Longitude : 0.0001 ° Signed MSB
                                                        Altitude : 0.01 meter Signed MSB

Check out how @kersing sends this data in his code here

Here is the code bit, you can see the first channel has 9 bytes for GPS( 0x88), and the 2nd channel has 2 bytes for the analog (battery voltage) value

   if (fix) {
     buffer[i++] = 0x01;
     buffer[i++] = 0x88;
     h = lat / 1000L;
     buffer[i++] = (h >> 16) & 0xff;
     buffer[i++] = (h >> 8) & 0xff;
     buffer[i++] = h & 0xff;
     h = lon / 1000L;
     buffer[i++] = (h >> 16) & 0xff;
     buffer[i++] = (h >> 8) & 0xff;
     buffer[i++] = h & 0xff;
     buffer[i++] = (alt >> 16) & 0xff;
     buffer[i++] = (alt >> 8) & 0xff;
     buffer[i++] = alt & 0xff;
   }

   measuredvbat *= 2;    // we divided by 2, so multiply back
   measuredvbat *= 3.3;  // Multiply by 3.3V, our reference voltage
   measuredvbat /= 1024; // convert to voltage
   h = measuredvbat * 100;
   Serial.print("VBat: " ); Serial.println(h);
   buffer[i++] = 0x02;
   buffer[i++] = 0x02;
   buffer[i++] = (h >> 8) & 0xff;
   buffer[i++] = h & 0xff;

This is how he sets up Cayenne here

When you are at the Cayenne end, you can represent the data channel as you want e.g. an analog value can be represented as “Battery Voltage”

The whole GPS tracker works nicely!
Well done @kersing

1 Like

That code runs on the node, it would be nice if we could transform data to lpp format in the console as well. I have some nodes where changing the code is hard (nodes are deployed, 40 minute drive away), being able to reformat the data and push it to cayenne would allow skipping the additional server now required to do the same thing.

3 Likes

Exactly what we need, change payload content on fly, and we all are coder, we know it’s do able easily since decoder and transformer funct already exists :wink:

1 Like

you have seen version V2 of lpp … bi directional

Yeah, but did not saw what I need, “Just A counter” I asked 6 month ago on this thread

1 Like

Indeed, the idea of a decoder, converter and validator is that you can separate decoding binary data (i.e. bytes to voltage), converting units (voltage to temperature) and validating measurements (remove outliers).

We’re dropping the converter and validator in V3 as they’re not being used; you can put conversion in the decoder, and it makes more sense to drop/filter out invalid values higher upstream.

The requested feature is really being able to transform binary data to another binary format, right? You don’t really want to manually convert your binary payload to CayenneLPP in JavaScript right, just to update the Cayenne dashboard?

What is the goal? Let’s see if we can find a sensible solution that leads to that!

1 Like

@johan
Thanks for your answer. Indeed it’s correct our final objective is to see data on Cayenne dashboard.
And for this, on nodes where we have hand, we can change the code and it’s fine
But sometimes, on other nodes where we do not have the source code and possibility to change payloads (when we buy devices) we need to convert known payload to CayenneLPP format, whatever solution is fine as soon we have hand to transform binary to another binary. JS is cool but another idea is fine, we would adapt I think.

I see. I would completely decouple the payload from Cayenne in that case.

Instead of using the TTN-Cayenne integration, consider using “Bring your own thing” and route data to Cayenne with MQTT. You can easily do this with Node-RED, taking data from TTN and routing it to Cayenne, without bothering with CayenneLPP.

Absolutely, the drawback of this is the need of another clouded server running MQTT client and some JS scripts (node red or native).
I’m fan of “less streams, less servers” => less things to check and less problems :wink:

But I will do that, got an NodeRed Instance running on my server
Thanks for the tip.

@johan
I tried your solution, unfortunately I’m not happy how cayenne handle that. While everyone is using user/pass in the MQTT broker and then ThingsID in the topic, cayenne NEEDS broker clientID set as ThingsID in the broker (and also in the topic).
This means if you have 100 devices, you need 100 different broker connexion (each with it’s own clientID). despite the fact than configuring 100 brokers in node red will be a nightmare, even if it could works, I can’t (don’t want sorry) use as many connexion as things, painfull
I know it’s not TTN fault but I’m back to my first step. I think adding the possibility to return a byte buffer from the converter will be really simple for everyone here to convert any payload.
But, I know simple for us does not mean simple for you.
If you could just think about the idea it would be very kind.

1 Like

I see. It’s not the first IoT platform with that requirement. That MQTT is really designed for end device to cloud, not cloud to cloud.

What you can do is posting with TTN’s JSON format, i.e. the one you see in MQTT and HTTP integration, to: https://lora.mydevices.com/v1/networks/ttn/uplink. Cayenne looks at the payload_raw, in which you can put anything that you like. But, it has to be CayenneLPP.

Now, I would say this is the hacky solution and there’s no guarantee this API will stay available this way.

Exactly what I wanted to do, I thought if TTN can send LPP I should be able to do also just needed the undocumented interface
Thanks for sharing, since I’m doing from node red should be easy just to transform payload_raw between TTN mqtt and cayenne HTTPS post :wink:
Hope there is no auth or source ip filtering for this one
Thanks

There is indeed no auth nor IP filtering, and that is indeed what’s is wrong with this interface :wink:

1 Like

@johan So

won’t be implemented? Currently I don’t have any Node RED running, so I can do what @Charles plans to do… :confused:

In V2 we don’t allow altering payload_raw. It will, technically, be supported in V3 but not sure if it will be GA in the public Consoles

@johan I tried your solution with no luck, may be I’m missing something

I convert the received TTN payload_raw to cayenne LPP payload_raw => OK
image

Then I send the full msg object (converted to JSON) as new msg.payload (so http request node has all original message in it’s msg.payload)

The recode function (after decoding incoming payload) is

var buf = Buffer.alloc(4+4+11);

// Channel 1, Analog In (Battery)
buf.writeUInt16BE(0x0102,  0);
buf.writeUInt16BE(batt*100, 2);

// Channel 2, Temp sensor
buf.writeInt16BE(0x0267, 4);
buf.writeInt16BE(temp*10, 6);

// Channel 3, GPS
buf.writeUInt16BE(0x0388,  8);
buf.writeIntBE(lat*10000, 10,3);
buf.writeIntBE(lon*10000, 13,3);
buf.writeIntBE(  0*10000, 16,3);

// Save new payload format to current msg payload_raw
msg.payload_raw = buf.toString('hex');

var retmsg = {};
// Get full current message as JSON
retmsg.payload = JSON.stringify(msg);

return retmsg;

Looks fine, output of recode (going to http request to url you mentioned) looks like OK
image

and output of http request (cayenne) says ok with HTTP 200
image

But I have noting on cayenne dashboard, and my node is created of course
image

I tried just sending payload_raw and hardware_serial but in this case, cayenne return 500 error so my 1st output seems good.

We’re so close, that’s it’s a pity it does not work. Do you know what field cayenne needs? Did I missed something?

Thanks for your help

1 Like

I’ve not got time atm - but I would try to emulate the HTTP using postman first (just quicker to test really) This could help?

At first sight the payload_raw is hex and it should be base64

I got your code to work as an “integration” between a TTN node device to Cayenne - well done.
In my case, I started with a TTN node device running the “Cayenne_lpp” sketch with the integration “Cayenne” setup and running.

The first thing I did, was remove the “Cayenne” integration in the TTN console.
Next I set up a node-red as you have shown.
The only thing I did differently. was change a few things in your “Recode” function
i.e. assign the correct channels to data and use this line to encode to base64
msg.payload_raw = buf.toString('Base64');
After this, I could see the values change in Cayenne
N.B. Even if the data code isn’t mapped correctly to channels, I still saw that the RSSI and SNR change in Cayenne - so that proves that it is receiving data


*edit - I also got postman to work using as follows: (the trick is not to include Payload)

POST           https://lora.mydevices.com/v1/networks/ttn/uplink
Authorization  No Auth       
Headers        Content-Type          application/json
Body           the json after payload e.g.   {"app_id":"xxx.....  ...."temperature_5":23.1},"_msgid":"cc993c47.19999"}

For any node, this is the JSON, you need to emulate - no secret needed and you have to publicise the DEVEUI. You also “Base 64” the payload_raw from the byte array (that was set up as per the “Cayenne lpp” format)
N.B. It doesn’t seem to matter what the “metadata” values are, as long as they are there

{"app_id":"YOUR_APPLICATION_ID","dev_id":"YOUR_DEV_ID","hardware_serial":"YOUR_8_BYTE_DEVEUI","payload_raw":"BAIAKAAAAAAAAAAAAAAAAAAAAA==","metadata":{"gateways":[{"gtw_id":"TEST_1","rssi":-73,"snr":6.75}]}}


*edit - added curl command (windows)

curl -v -X POST --data "{\"app_id\":\"YOUR_APP_ID\",\"dev_id\":\"YOUR_DEV_ID\",\"hardware_serial\":\"1122334455667788\",\"payload_raw\":\"BAIAKAAAAAAAAAAAAAAAAAAAAA==\",\"metadata\":{\"gateways\":[{\"gtw_id\":\"TEST_1\",\"rssi\":-73,\"snr\":6.75}]}}" https://lora.mydevices.com/v1/networks/ttn/uplink -H "Content-Type: application/json"

That was the trick, noticed some strange things .

  • If I’m not on the device tree selected widged (in my case ls18002) on cayenne browser, then nothing appears on dashboard, looks like 1st time device integration, you need to be on the device on cayenne dashboard
  • as said @CurlyWurly the 1st widget that came, was only RSSI and SNR
  • i need to check why gps coord are wrong
  • after that all went fine, great

I used node base64 and json but @CurlyWurly just showed that base64 can be native I’ll integrate this in Recode function. I’m doing some check and post new Recode source code
image

@johan you saved my day, @CurlyWurly thanks also, I just saw your post after got it working :wink:

1 Like