Decoding a float value in a payload function

kersing · December 19, 2016, 10:38pm

@erikHouten
The default function does not decode data, it simply passes the data on to the client which means you can do your own decoding in the client.

douglarue · December 20, 2016, 3:40am

a 16bit integer is 2 bytes and a 2 byte array is 16bits. Ie you don’t need to create the 2 byte array. Just send the 16bit int.

Batilan · December 20, 2016, 5:59pm

You might also want to have a look at https://github.com/thesolarnomad/lora-serialization (as mentioned before in Approaches for efficient payload encoding). It should take care of any endiannes issues.

brandonbyrne · December 20, 2016, 10:03pm

This is how I am decoding the ttn payload via node-red. This assumes that the payload I am getting from ttn via mqtt is an 8 byte payload, coming out of the node-red mqtt node in base64, with the first four bytes being one float, and the second four bytes being the second float.

If you are trying to decode a raw 8 byte value, it’s even easier:

var lat = new Buffer(msg.payload).readFloatLE(0);
var lon = new Buffer(msg.payload).readFloatLE(4);
msg.payload= “[{“lat”:” + lat + “,“lng”:” + lon + “}]”;//"{“lat”:lat,“lng”:lon};
return msg;

I hate javascript and am terrible at doing anything with it, so there is probably a more efficient way to do this.

cBashTn · December 21, 2016, 11:20am

@erikHouten
On staging.thethingsnetwork.org/applications I use this function:

function(bytes) {

   var Bytes2Float32 = {
       decodeFloat: function(bytes, signBits, exponentBits, fractionBits, eMin, eMax, littleEndian) {
           var totalBits = (signBits + exponentBits + fractionBits);
           var binary = "";
           for (var i = 0, l = bytes.length; i < l; i++) {
               var bits = bytes[i].toString(2);
               while (bits.length < 8)
                   bits = "0" + bits;
               if (littleEndian)
                   binary = bits + binary;
               else
                   binary += bits;
           }
           var sign = (binary.charAt(0) == '1') ? -1 : 1;
           var exponent = parseInt(binary.substr(signBits, exponentBits), 2) - eMax;
           var significandBase = binary.substr(signBits + exponentBits, fractionBits);
           var significandBin = '1' + significandBase;
           var i = 0;
           var val = 1;
           var significand = 0;
           if (exponent == -eMax) {
               if (significandBase.indexOf('1') == -1)
                   return 0;
               else {
                   exponent = eMin;
                   significandBin = '0' + significandBase;
               }
           }
           while (i < significandBin.length) {
               significand += val * parseInt(significandBin.charAt(i));
               val = val / 2;
               i++;
           }
           return sign * significand * Math.pow(2, exponent);
       }
   }


   var value1 = [bytes[4], bytes[5], bytes[6], bytes[7]];
   var value2 = [bytes[10], bytes[11], bytes[12], bytes[13]];
   var value3 = [bytes[14], bytes[15], bytes[16], bytes[17]];
   var value1float = Bytes2Float32.decodeFloat(value1, 1, 8, 23, -126, 127, false);
   var value2float = Bytes2Float32.decodeFloat(value2, 1, 8, 23, -126, 127, false);
   var value3float = Bytes2Float32.decodeFloat(value3, 1, 8, 23, -126, 127, false);

   return {
       payload : bytes,
       batteryVoltage: value1float,
       pressure: value2float,
       temperature: value3float
   };
}

It works for me. The decoded json is sent to the Azure IoTHub (with webjobs), there it goes to a SQL-server and a demo-web app shows the data (http://cloud.keller-druck.com).

Be aware that a Test doesn’t work (

TypeError(“Cannot access member ‘toString’ of undefined”) )

On console.thethingsnetwork.org/applications on the other hand, you must have a successful Test before saving the payload function. Therefore, I couldn’t get it working there.

csabagod · March 2, 2017, 7:25pm

Hi,

Can someone send me a code for HX711 loadcell amplifier and DHT22 library?

I have tried this:

uint16_t weight = scale.get_units() * 100; 

// Read sensor values and multiply by 100 to effictively have 2 decimals
uint16_t humidity = dht.readHumidity(false) * 100;

// false: Celsius (default)
// true: Farenheit
uint16_t temperature = dht.readTemperature(false) * 100;


// Split both words (16 bits) into 6 bytes of 8
byte message[6];

message[0] = highByte(temperature);
message[1] = lowByte(temperature);
message[2] = highByte(humidity);
message[3] = lowByte(humidity);
message[4] = highByte(weight);
message[5] = lowByte(weight);

LMIC_setTxData2(1, message, sizeof(message), 0);
}

On TTN side this is the payload function:

function (bytes) {

var decodedValue1 = (bytes[4] << 8) | bytes[5];
var decodedValue2 = (bytes[2] << 8) | bytes[3];
var decodedValue3 = (bytes[0] << 8) | bytes[1];

return { Weight: decodedValue1/100, Temp: decodedValue2/100, Hum: decodedValue3/100};
}

The result is this. It should be 4 decimal each:

Hum 20.3
Temp 1
Weight 654.36

arjanvanb · March 2, 2017, 8:21pm

You’re multiplying by 100, and storing that in memory as an “unsigned 16 bits integer” (uint16), hence losing all remaining decimals. For example: 12.34567 is stored and sent as 1234, which is probably just fine as most sensors really don’t have an accuracy of 4 decimals (or even 2 decimals). It even says this in your code:

// Read sensor values and multiply by 100 to effictively have 2 decimals
uint16_t humidity = dht.readHumidity(false) * 100;

As 2 bytes unsigned integers can only hold values 0 to 65535, you don’t really have room to multiply by, say, 1000 to get 3 decimals as then any value larger than 65.5 would cause an overflow with unexpected results.

Things are even more complicated if the values might be negative, in which case the JavaScript in the payload functions expects 4 bytes to do proper calculations for negative values:

// Signed 16 bits integer, -32767 up to +32767
int16_t temperature = dht.readTemperature(false) * 100;

…with in the payload function:

decodedValue3 = (bytes[0] & 0x80 ? 0xFFFF<<16 : 0) | bytes[0]<<8 | bytes[1];

More details on the above “sign-extension” in Decrypting messages for dummies (and more links to examples in Is there any documentation on payload functions?).

You’re also mixing the values. You’re sending 2 bytes for temperature, humidity and weight. But you’re are showing that as humidity, temperate and weight. So, 20.3 is your temperature. I’d clean up and use the same order in your code:

// For staging-v1
function(bytes) {
    // temperature might be negative; sign-extend to get 4 bytes:
    var t = (bytes[0] & 0x80 ? 0xFFFF<<16 : 0) | bytes[0]<<8 | bytes[1];
    var h = bytes[2]<<8 | bytes[3];
    var w = bytes[4]<<8 | bytes[5];

    return {
      temperature: t / 100,
      humidity: h / 100,
      weight: w / 100
    }
}

I don’t understand why you’re getting a humidity of exactly 1. What does the false in the formula do? I guess the result of dht.readHumidity(false) is 1 or 1.00, which multiplied by 100 is sent as 100 and then decoded back to 1.

By the way, on the production environment the first line of a Decoder payload function needs to read:

function Decoder(bytes, port) {

And as Staging will be taken offline at the end of March, I strongly advise to use production.

arjanvanb · March 5, 2017, 1:16pm

2 posts were split to a new topic: How to use the HX711 24 bit ADC for weight scales?

Jonathanmf · June 2, 2017, 9:10pm

Hi !

I have some difficult with the payload function.
Here is my Payload: 32352E3736302E32
This is for a DHT22 sensor. When i use python I can read values:

>>> binascii.unhexlify('32352E37')
b'25.7'
>>> binascii.unhexlify('36302E32')
b'60.2'

But I can’t make readable on the decoder. What can I do I tried … a lot !

thx (and sorry for my english)

arjanvanb · June 2, 2017, 9:56pm

That’s decoding text (where the hexadecimal 0x32 is the character 2, 0x35 is 5, 0x2E is the dot, and 0x37 is 7). But your node should not send text as that needs 8 bytes for just two values with a very limited range.

Still then, while you’ve not changed your node’s code to use a better encoding, and assuming that always 4 characters per value are used, so if:

no negative values are sent
zero is sent as 00.0, or as 0.0 with a leading space
values below 10 are sent with a leading space or zero, like 08.1
integer values are sent with a fractional part, like 12.0

…then you could decode the text using:

function Decoder(bytes, port) {
  // test with 32352E3736302E32
  var text1 = String.fromCharCode.apply(null, bytes.slice(0, 4));
  var text2 = String.fromCharCode.apply(null, bytes.slice(4, 8));
    
  return {
    text1: text1,
    float1: parseFloat(text1),
    text2: text2,
    float2: parseFloat(text2)
  }
}

…which will get you:

{
  "float1": 25.7,
  "float2": 60.2,
  "text1": "25.7",
  "text2": "60.2"
}

Jonathanmf · June 3, 2017, 10:30am

Oh I understand, so I need to optimise the code. I’m totally “noob” I need to learn. Thanks for your explanation.

BTW, here is the code

void onEvent (ev_t ev) {
    Serial.print(os_getTime());
    Serial.print(": ");
    switch(ev) {
        case EV_SCAN_TIMEOUT:
            Serial.println(F("EV_SCAN_TIMEOUT"));
            break;
        case EV_BEACON_FOUND:
            Serial.println(F("EV_BEACON_FOUND"));
            break;
        case EV_BEACON_MISSED:
            Serial.println(F("EV_BEACON_MISSED"));
            break;
        case EV_BEACON_TRACKED:
            Serial.println(F("EV_BEACON_TRACKED"));
            break;
        case EV_JOINING:
            Serial.println(F("EV_JOINING"));
            break;
        case EV_JOINED:
            Serial.println(F("EV_JOINED"));
            break;
        case EV_RFU1:
            Serial.println(F("EV_RFU1"));
            break;
        case EV_JOIN_FAILED:
            Serial.println(F("EV_JOIN_FAILED"));
            break;
        case EV_REJOIN_FAILED:
            Serial.println(F("EV_REJOIN_FAILED"));
            break;
        case EV_TXCOMPLETE:
            Serial.println(F("EV_TXCOMPLETE (includes waiting for RX windows)"));
            if (LMIC.txrxFlags & TXRX_ACK)
              Serial.println(F("Received ack"));
            if (LMIC.dataLen) {
              Serial.println(F("Received "));
              Serial.println(LMIC.dataLen);
              Serial.println(F(" bytes of payload"));
            }
            // Schedule next transmission
            os_setTimedCallback(&sendjob, os_getTime()+sec2osticks(TX_INTERVAL), do_send);
            break;
        case EV_LOST_TSYNC:
            Serial.println(F("EV_LOST_TSYNC"));
            break;
        case EV_RESET:
            Serial.println(F("EV_RESET"));
            break;
        case EV_RXCOMPLETE:
            // data received in ping slot
            Serial.println(F("EV_RXCOMPLETE"));
            break;
        case EV_LINK_DEAD:
            Serial.println(F("EV_LINK_DEAD"));
            break;
        case EV_LINK_ALIVE:
            Serial.println(F("EV_LINK_ALIVE"));
            break;
         default:
            Serial.println(F("Unknown event"));
            break;
    }
}

void do_send(osjob_t* j){
    byte buffer[8];
    float t = dht.readTemperature();
    float h = dht.readHumidity();
    dtostrf(t, 2, 1, buffer);
    dtostrf(h, 2, 1, &buffer[4]);
    String res = buffer;
    res.getBytes(buffer, res.length() + 1);
    Serial.println("");
    Serial.print("Sending - temperature: ");
    Serial.print(t);
    Serial.print(", humidity: ");
    Serial.print(h);
    Serial.println("");

    // Check if there is not a current TX/RX job running
    if (LMIC.opmode & OP_TXRXPEND) {
        Serial.println(F("OP_TXRXPEND, not sending"));
    } else {
        // Prepare upstream data transmission at the next possible time.
        LMIC_setTxData2(1, (uint8_t*) buffer, res.length(), 0);
        Serial.println(F("Packet queued"));
    }
    // Next TX is scheduled after TX_COMPLETE event.
}

arjanvanb · June 3, 2017, 11:04am

This goes horribly wrong for any value that does not convert to exactly four characters. (If you found that code elsewhere, please refer the author to this answer…)

Instead, use something like explained above:

// Read sensor values and multiply by 100 to effectively keep 2 decimals
// Signed 16 bits integer, -32767 up to +32767
int16_t t = dht.readTemperature() * 100;
// Unsigned 16 bits integer, 0 up to 65535
uint16_t h = dht.readHumidity() * 100;

This can be encoded into just 4 bytes for the 2 values, using some “shifting” as explained in more detail in Decrypting messages for dummies:

byte buffer[4];
buffer[0] = t >> 8;
buffer[1] = t;
buffer[2] = h >> 8;
buffer[3] = h;
LMIC_setTxData2(1, buffer, sizeof(buffer), 0);

And decoded like also explained in more detail in the link above:

function Decoder(bytes, port) {
  var t = (bytes[0] & 0x80 ? 0xFFFF<<16 : 0) | bytes[0]<<8 | bytes[1];
  var h = bytes[2]<<8 | bytes[3];
  return {
    temperature: t / 100,
    humidity: h / 100
  }
}

You could probably even send humidity without any decimals, saving yet another byte:

// Unsigned 8 bits integer, 0 up to 255
uint8_t h = dht.readHumidity();

…with:

byte buffer[3];
buffer[0] = t >> 8;
buffer[1] = t;
buffer[2] = h;
LMIC_setTxData2(1, buffer, sizeof(buffer), 0);

…and:

function Decoder(bytes, port) {
  var t = (bytes[0] & 0x80 ? 0xFFFF<<16 : 0) | bytes[0]<<8 | bytes[1];
  var h = bytes[2];
  return {
    temperature: t / 100,
    humidity: h
  }
}

Jonathanmf · June 3, 2017, 12:30pm

Work Perfectly (with the optimized version for humidity) ! Big thank for all your explanations ! Clear and working. I’m getting smarter today

kots · August 9, 2017, 1:17pm

Any ideas i could do what you are describing?
Let me first tell you that i use loriot server instead of ttn.
I tried to connect loriot backend with node red and ibm bluemix but i am struggling with javascript.
So any ideas about decoding with python and then connecting to an application server?

BoRRoZ · August 9, 2017, 1:24pm

and you asked questions on the TTN forum ? … strange

BoRRoZ · August 9, 2017, 2:03pm

your welcome

arjanvanb · September 22, 2017, 8:32am

Just for future reference, another option to decode an IEEE-754 floating point in Decode float sent by Lopy as Node - #2 by arjanvanb.

brady_aiello · January 4, 2018, 7:15am

I pulled my answer from here:

So my payload function, making up for the ATSAMD21G18’s opposite endianness looks like this:

github.com

brady-aiello/Seeeduino_LoRaWAN_for_hybrid_gateways/blob/master/Seeeduino-LoRaWAN-GPS-app/payload_function.js

function Bytes2Float32(bytes) {
    var sign = (bytes & 0x80000000) ? -1 : 1;
    var exponent = ((bytes >> 23) & 0xFF) - 127;
    var significand = (bytes & ~(-1 << 23));

    if (exponent == 128) 
        return sign * ((significand) ? Number.NaN : Number.POSITIVE_INFINITY);

    if (exponent == -127) {
        if (significand == 0) return sign * 0.0;
        exponent = -126;
        significand /= (1 << 22);
    } else significand = (significand | (1 << 23)) / (1 << 23);

    return sign * significand * Math.pow(2, exponent);
}

function Decoder(bytes, port) {
	var lat = bytes[3] << 24 | bytes[2] << 16 | bytes[1] << 8 | bytes[0];
	var lon = bytes[7] << 24 | bytes[6] << 16 | bytes[5] << 8 | bytes[4];

This file has been truncated. show original

Mat_Lynx · November 17, 2018, 7:01pm

This was very helpful! Thanks.

cBashTn · November 30, 2018, 11:45am

Thanks to @arjanvanb we updated our payload decoder code:
https://github.com/KELLERAGfuerDruckmesstechnik/KellerAgTheThingsNetworkPayloadDecoder