Decoder Help with bit shifting

End Devices (Nodes)
1

I have a sensor that is one of my end devices. It sends the GPS location, altitude, time and the feedback of a geophone (ground sensor) to my gateway. The end device is a Heltec ESP32 LoRaWAN microcontroller. I’m really struggling with the decoder. I’ve seen a lot of documentation using the decoder for temperature and humidity sensors but I’m not really sure on how to apply it to my project.

Arduino Sketch



#include <ESP32_LoRaWAN.h>
#include "Arduino.h"
//Include the needed libraries for the ADS and the GPS module
#include <Wire.h>//I2C library
#include <Adafruit_ADS1X15.h>//ADS library
#include "TinyGPS++.h"//Gps module library

//Define the BAND frequency
#define BAND    915E6//set BAND to US which is 915E6 or 915MHz

//Declare your objects which are the GPS module and the ADS1115 in which we can read the geophone inputs
TinyGPSPlus gps;//This is the GPS object that will pretty much do all the grunt work with the NMEA data
Adafruit_ADS1115 ads;/* Use this for the 16-bit version */

//Declare the global variables
int value = analogRead(A0);
float Voltage;
float Perc;
int16_t SensorRead;
float LatRead;
float LonRead;
float AltRead;
int HourRead;
int MinRead;
int SecRead;





/*license for Heltec ESP32 LoRaWan, quary your ChipID relevant license: http://resource.heltec.cn/search */
uint32_t  license[4] = {0xD5397DF0, 0x8573F814, 0x7A38C73D, 0x48E68607};

/* OTAA para*/
uint8_t DevEui[] = { 0x70, 0xB3, 0xD5, 0x7E, 0xD0, 0x04, 0xC7, 0x25 };
uint8_t AppEui[] = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 };
uint8_t AppKey[] = { 0x00, 0x5C, 0xC0, 0x9F, 0x5D, 0x71, 0x4F, 0xE1, 0x0D, 0xAC, 0x21, 0x7D, 0xC4, 0xB3, 0x43, 0x40};

/* ABP para*/
uint8_t NwkSKey[] = { 0x15, 0xb1, 0xd0, 0xef, 0xa4, 0x63, 0xdf, 0xbe, 0x3d, 0x11, 0x18, 0x1e, 0x1e, 0xc7, 0xda, 0x85 };
uint8_t AppSKey[] = { 0x00, 0x5C, 0xC0, 0x9F, 0x5D, 0x71, 0x4F, 0xE1, 0x0D, 0xAC, 0x21, 0x7D, 0xC4, 0xB3, 0x43, 0x40 };
uint32_t DevAddr =  ( uint32_t )0x007e6ae1;

/*LoraWan channelsmask, default channels 0-7*/
uint16_t userChannelsMask[6] = { 0x00FF, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000 };

/*LoraWan Class, Class A and Class C are supported*/
DeviceClass_t  loraWanClass = CLASS_A;

/*the application data transmission duty cycle.  value in [ms].*/
uint32_t appTxDutyCycle = 15000;

/*OTAA or ABP*/
bool overTheAirActivation = true;

/*ADR enable*/
bool loraWanAdr = true;

/* Indicates if the node is sending confirmed or unconfirmed messages */
bool isTxConfirmed = true;

/* Application port */
uint8_t appPort = 2;

/*!
  Number of trials to transmit the frame, if the LoRaMAC layer did not
  receive an acknowledgment. The MAC performs a datarate adaptation,
  according to the LoRaWAN Specification V1.0.2, chapter 18.4, according
  to the following table:

  Transmission nb | Data Rate
  ----------------|-----------
  1 (first)       | DR
  2               | DR
  3               | max(DR-1,0)
  4               | max(DR-1,0)
  5               | max(DR-2,0)
  6               | max(DR-2,0)
  7               | max(DR-3,0)
  8               | max(DR-3,0)

  Note, that if NbTrials is set to 1 or 2, the MAC will not decrease
  the datarate, in case the LoRaMAC layer did not receive an acknowledgment
*/
uint8_t confirmedNbTrials = 8;

/*LoraWan debug level, select in arduino IDE tools.
  None : print basic info.
  Freq : print Tx and Rx freq, DR info.
  Freq && DIO : print Tx and Rx freq, DR, DIO0 interrupt and DIO1 interrupt info.
  Freq && DIO && PW: print Tx and Rx freq, DR, DIO0 interrupt, DIO1 interrupt and MCU deepsleep info.
*/
uint8_t debugLevel = LoRaWAN_DEBUG_LEVEL;

/*LoraWan region, select in arduino IDE tools*/
LoRaMacRegion_t loraWanRegion = ACTIVE_REGION;

static void prepareTxFrame( uint8_t port )
{
  /*appData size is LORAWAN_APP_DATA_MAX_SIZE which is defined in "commissioning.h".
    appDataSize max value is LORAWAN_APP_DATA_MAX_SIZE.
    if enabled AT, don't modify LORAWAN_APP_DATA_MAX_SIZE, it may cause system hanging or failure.
    if disabled AT, LORAWAN_APP_DATA_MAX_SIZE can be modified, the max value is reference to lorawan region and SF.
    for example, if use REGION_CN470,
    the max value for different DR can be found in MaxPayloadOfDatarateCN470 refer to DataratesCN470 and BandwidthsCN470 in "RegionCN470.h".
  */
  pinMode(Vext, OUTPUT);
  digitalWrite(Vext, LOW);
  float Voltage = value * 5.0 / 1023;
  float Perc = map(Voltage, 3.6, 4.2, 0, 100);
  float LatRead = gps.location.lat();
  float LonRead = gps.location.lng();
  float AltRead = gps.altitude.feet();
  int HourRead = gps.time.hour();
  int MinRead = gps.time.minute();
  int SecRead = gps.time.second();
  int16_t SensorRead = ads.getLastConversionResults();
  digitalWrite(Vext, HIGH);
  unsigned char *puc;

  puc = (unsigned char *)(&Voltage);
  appDataSize = 34;
  appData[0] = puc[0];
  appData[1] = puc[1];
  appData[2] = puc[2];
  appData[3] = puc[3];

  puc = (unsigned char *)(&Perc);
  appData[4] = puc[0];
  appData[5] = puc[1];
  appData[6] = puc[2];
  appData[7] = puc[3];

  puc = (unsigned char *)(&LatRead);
  appData[8] = puc[0];
  appData[9] = puc[1];
  appData[10] = puc[2];
  appData[11] = puc[3];

  puc = (unsigned char *)(&LonRead);
  appData[12] = puc[0];
  appData[13] = puc[1];
  appData[14] = puc[2];
  appData[15] = puc[3];

  puc = (unsigned char *)(&AltRead);
  appData[16] = puc[0];
  appData[17] = puc[1];
  appData[18] = puc[2];
  appData[19] = puc[3];

  puc = (unsigned char *)(&HourRead);
  appData[20] = puc[0];
  appData[21] = puc[1];
  appData[22] = puc[2];
  appData[23] = puc[3];

  puc = (unsigned char *)(&MinRead);
  appData[24] = puc[0];
  appData[25] = puc[1];
  appData[26] = puc[2];
  appData[27] = puc[3];

  puc = (unsigned char *)(&SecRead);
  appData[28] = puc[0];
  appData[29] = puc[1];
  appData[30] = puc[2];
  appData[31] = puc[3];

  puc = (unsigned char *)(&SensorRead);
  appData[32] = puc[0];
  appData[33] = puc[1];
  appData[34] = puc[2];
  appData[35] = puc[3];


  Serial.print("Lat=");
  Serial.print(LatRead);
  Serial.print(", Lon=");
  Serial.print(LonRead);
  Serial.print(", Alt=");
  Serial.println(AltRead);
  Serial.print("Time: ");
  Serial.print(HourRead);
  Serial.print(":");
  Serial.print(MinRead);
  Serial.print(":");
  Serial.println(SecRead);
  Serial.print("Voltage: ");
  Serial.print(Voltage);
  Serial.print(", Percentage: ");  
  Serial.println(Perc);
  Serial.print("Sensor Reading: ");
  Serial.println(SensorRead);
}

// Add your initialization code here
void setup()
{
  Serial.begin(115200);
  while (!Serial);
  SPI.begin(SCK, MISO, MOSI, SS);
  Mcu.init(SS, RST_LoRa, DIO0, DIO1, license);
  deviceState = DEVICE_STATE_INIT;
  Serial2.begin(115200, SERIAL_8N1, 2, 17);
  //serial_connection.begin(115200);//This opens up communications to the GPS
  Serial.println("Hello!");
  Serial.println("Single-ended readings from AIN0 with >3.0V comparator");
  Serial.println("ADC Range: +/- 6.144V (1 bit = 3mV/ADS1015, 0.1875mV/ADS1115)");
  Serial.println("Comparator Threshold: 1000 (3.000V)");
  //                                                                ADS1015  ADS1115
  //                                                                -------  -------
  // ads.setGain(GAIN_TWOTHIRDS);  // 2/3x gain +/- 6.144V  1 bit = 3mV      0.1875mV (default)
  //ads.setGain(GAIN_ONE);        // 1x gain   +/- 4.096V  1 bit = 2mV      0.125mV
  // ads.setGain(GAIN_TWO);        // 2x gain   +/- 2.048V  1 bit = 1mV      0.0625mV
  // ads.setGain(GAIN_FOUR);       // 4x gain   +/- 1.024V  1 bit = 0.5mV    0.03125mV
  // ads.setGain(GAIN_EIGHT);      // 8x gain   +/- 0.512V  1 bit = 0.25mV   0.015625mV
  //ads.setGain(GAIN_SIXTEEN);    // 16x gain  +/- 0.256V  1 bit = 0.125mV  0.0078125mV
  ads.begin();
  ads.setGain(GAIN_FOUR);
  if (!ads.begin()) {
    Serial.println("Failed to initialize ADS.");
    while (1);
  }
  // Setup 3V comparator on channel 0
  ads.startComparator_SingleEnded(0, 1000);
}

// The loop function is called in an endless loop
void loop()
{
  switch ( deviceState )
  {
    case DEVICE_STATE_INIT:
      {
#if(LORAWAN_DEVEUI_AUTO)
        LoRaWAN.generateDeveuiByChipID();
#endif
        LoRaWAN.init(loraWanClass, loraWanRegion);
        break;
      }
    case DEVICE_STATE_JOIN:
      {
        LoRaWAN.join();
        break;
      }
    case DEVICE_STATE_SEND:
      {
        prepareTxFrame( appPort );
        LoRaWAN.send(loraWanClass);
        deviceState = DEVICE_STATE_CYCLE;
        break;
      }
    case DEVICE_STATE_CYCLE:
      {
        // Schedule next packet transmission
        txDutyCycleTime = appTxDutyCycle + randr( -APP_TX_DUTYCYCLE_RND, APP_TX_DUTYCYCLE_RND );
        LoRaWAN.cycle(txDutyCycleTime);
        deviceState = DEVICE_STATE_SLEEP;
        break;
      }
    case DEVICE_STATE_SLEEP:
      {
        LoRaWAN.sleep(loraWanClass, debugLevel);
        break;
      }
    default:
      {
        deviceState = DEVICE_STATE_INIT;
        break;
      }
  }
}

and below is the decoder I’m using

Decoder

function Decoder(bytes, port) {
  var voltage = (bytes[0] << 8) | bytes[1];
  var perc = (bytes[1] << 8) | bytes[3];
  var lat = (bytes[4] << 8) | bytes[5];
  var lon = (bytes[4] << 8) | bytes[7];
  var alt = (bytes[8] << 8) | bytes[9];
  var hour = (bytes[0] << 8) | bytes[1];
  var min = (bytes[2] << 8) | bytes[3];
  var sec = (bytes[4] << 8) | bytes[5];
  var sensor1 = (bytes[4] << 8) | bytes[7];

  
  return {
    voltage: voltage,
    perc: perc,
    lat: lat,
    lon: lon,
    alt: alt,
    hour: hour,
    min: min,
    sec: sec,
    sensor1: sensor1,

  }
  
}

Please help, thank you!

2

Let’s start at the beginning.

How many bytes did you pack in?
How many bytes did you pack out?

What precision of values do you need?
…

Here is a great thread to similar query.

2 Likes
3

Hello, I really appreciate the response, and I apologize for my lack of understanding, I’ve been trying to research the link you sent me. I’m not really sure how many bytes I’m packing in or out. I saw that my Latitude, Longitude, and Altitude is using 9 bytes total. and I read that float and int use 4 bytes a piece. I’m very new to LoRa and Am trying to understand how to properly allocate bytes, send data and decode that data on my gateway. I am looking for the Longitude and Latitude to give me an accuracy of 6 decimal places. The other values only need to be two decimal places if they are floats.

4

This is the function i use for decoding multiple bytes of information to a long integer.

byteArrayToLong = function(/*byte[]*/byteArray) {
    var value = 0;
    for ( var i = byteArray.length - 1; i >= 0; i--) {
        value = (value * 256) + byteArray[i];
    }

    return value;
};

First declare the function. Then further on in the decoder:

function Decoder(bytes, port) {
  // Decode an uplink message from a buffer
  // (array) of bytes to an object of fields.
  
  var decoded = {};
  if (port == 2){
    decoded.time = byteArrayToLong(bytes.slice(0,4)); //enter the position in the bytes array which you need decoded. It selects from (0) to (4). So location 0, 1, 2 and 3 enter the function. (the first 4 bytes)
  }else{
    return bytes;
  }
}
1 Like
5

Hi, The approach I take is to try to convert everything to unsigned integer wherever possible in my sensor node code and convert it back to the real value in the decoder. This does mean that you have to have some knowledge or expectation of the range of values that you will be sending.

Where I live I know that -10C outdoor temperature is extremely unlikely. To send temperature to 2 decimal places I would use a formula like:

  // encode float as int
  uint16_t tempInt = round((temperature +10.0) * 100);

  // encode int as bytes
 
  appData[0] = highByte(tempInt);
  appData[1] = lowByte(tempInt);

Without clever bit packing, this has already reduced the 4 byte float down to a 2 byte unsigned integer.

Then the code fragment I would use to convert this back to floating point in my decoder would be:

     // temperature 
      rawTemp = bytes[1] + bytes[0] * 256;
      decoded.degreesC = rawTemp / 100 - 10.0;

If you also know the maximum values of your measurements you can calculate how many bits each measurement needs and bit pack your data to get a smaller transmission.

Next time I rebuild the GPS node that I use for TTN Mapper I will apply the same principle to the latitude and longitude values. If I’m only mapping the coverage in my own neighbourhood I can significantly reduce the size of the data transmitted. If I want to be able to collect data anywhere in the country I can still achieve quite a saving without sacrificing accuracy.

1 Like
6

This is extremely helpful. So, if I have an appData size of 34, would it still look like this?

"var sec = (bytes[24] << 8) | bytes[27]; "

my arduino sketch looks likes

  puc = (unsigned char *)(&MinRead);
  appData[24] = puc[0];
  appData[25] = puc[1];
  appData[26] = puc[2];
  appData[27] = puc[3];

I have a lot of values I’m sending, and your method would significantly reduce that number of bytes packed. But I’m SUPER new to all of this, so does the byte allocation go in the brackets?

7

Hi,

I think the first thing to consider is the number of bytes you are transmitting. 34 bytes is quite a large message and will limit the number of times you can send each 24 hours. Search the forum and read the documentation about TTN’s Fair Use Policy. Then have a look at the amount of air time you will use. Here is a link for an air time calculator https://avbentem.github.io/airtime-calculator/ttn/eu868

If you are sending a value in 4 bytes you will need to decode all 4 bytes. The most significant byte would need to be shifted 24 bits, the next 16 bits, the next 8 bits, and the last one left as it is. So something like
var sec = (bytes[24]<<24) | (bytes[25]<<16) | (bytes[26]<<8) | bytes[27]
Assuming byte[24] is the most significant. If it doesn’t decode correctly it’s probably the other way round.

Have a think about whether you need to transmit the GPS time. Is the time that is recorded in the uplink data from TTN accurate enough for your application?

If you are sending the battery voltage for your node. Depending on the battery type it can probably be sent in a single byte. If you set a minimum voltage of 2.5V and the maximum never more than 5V. This more than covers the range you should see with a LiPo cell. Then the code for that becomes

 uint8_t battInt = round((volts - 2.5) * 100);
1 Like
8

would it be more efficient if I were to send my values as one string?
or should I just try to get that amount of data condensed by limiting the bytes per value using the method you proposed?

9

I am only at the stage where I think about reducing the size of the individual values to bytes or integers and building my appdata array for transmission. For what I want to do, that makes my messages small enough that I can transmit every 15 minutes without getting anywhere near the Fair Use Policy limit.

As I said earlier you can go as far as bit packing your appdata array. On/Off can be represented by a single bit, something that only changes by up to 16 units will fit in 4 bits, anything in the range 0 to 511 will need 9 bits. This of course makes building the appdata array quite complicated, and the decoder equally complicated. As yet I haven’t needed to go that far.

I have also seen discussion on this Forum about applying lossless compression algorithms, but that is way beyond my level of knowledge.

1 Like
10

No, never - it makes things even larger.

The conversion of values to an unsigned integer as demonstrated by @AndyG is the standard recommended technique. Taking things further by reducing from, say 16 bits to 12 and then sharing the other four bits is entirely feasible but does make for code that needs to be well tested & understood. Using a byte for on’s & off’s of digital devices is relatively straightforward coding and if you are writing firmware should be doable.

But if you have a float value of temp at 21.7 degC for instance, that’s either 4 bytes as the float which can be decoded at the back end, or with an appropriate offset could be just one byte of if it had two decimal places then it would be two bytes using the scheme above. But as soon as you start using strings you make the payload much bigger. So 21.75 deg C would need at least 6 bytes, four for the digits, one for the decimal and you will need a delimiter so you can tell where one reading ends & the next one starts.

Try reading https://www.thethingsnetwork.org/docs/devices/bytes/ - the pink warning box does not apply to this general document.

2 Likes
11

Everyone thank you so much! I finally am getting the grasp of it and my decoder is showing the correct data that I am sending.

2 Likes
12

This topic was automatically closed 24 hours after the last reply. New replies are no longer allowed.