CayenneLPP and FIWARE IoT Agent for LoRaWAN

[electronics1920]

Hi everyone again!

Well, of course this topic hasn’t much to do with TTN, but I’d like to discard a few things. Firstly … I’ve published a question in StackOverflow and an issue in GitHub (where I think is most appropriate to treat this topic), however I’m not sure if those will be answered any soon time.

The point is … that I’m trying to build a local application (based on FIWARE) to manage the TTN traffic; however and despite I’m trying to follow their docs (regarding to LoRaWAN agent) I’m getting the following errors in the docker logs

fiware-iot-agent | {"timestamp":"2020-06-23T11:45:53.689Z","level":"info","message":"New message in topic"}
fiware-iot-agent | {"timestamp":"2020-06-23T11:45:53.690Z","level":"info","message":"IOTA provisioned devices:"}
fiware-iot-agent | {"timestamp":"2020-06-23T11:45:53.691Z","level":"info","message":"Decoding CaynneLPP message:+XQ="}
fiware-iot-agent | {"timestamp":"2020-06-23T11:45:53.691Z","level":"error","message":"Error decoding CaynneLPP message:Error: Invalid CayennLpp buffer size"}
fiware-iot-agent | {"timestamp":"2020-06-23T11:45:53.691Z","level":"error","message":"Could not cast message to NGSI"}

I think there must be something wrong with IoTAgent-LoRaWAN itself because in those error messages are a few typos, it’s not CaynneLPP but CayenneLPP so maybe it’s not my fault. Also … as you can see in the StackOverflow question I’ve provisioned the IoTAgent-LoRaWAN following their docs at this respect but I’m not much confident on the used parameters:

{
  "devices": [
    {
      "device_id": "{{node}}",
      "entity_name": "LORA-N-0",
      "entity_type": "LoraDevice",
      "timezone": "Europe/Madrid",
      "attributes": [
        {
          "object_id": "potVal",
          "name": "Pot_Value",
          "type": "Number"
        }
      ],
      "internal_attributes": {
        "lorawan": {
          "application_server": {
            "host": "eu.thethings.network",
            "username": "{{TTN_app_id}}",
            "password": "{{TTN_app_pw}}",
            "provider": "TTN"
          },
          "dev_eui": "{{TTN_dev_eui}}",
          "app_eui": "{{TTN_app_eui}}",
          "application_id": "{{TTN_app_id}}",
          "application_key": "{{TTN_app_skey}}"
        }
      }
    }
  ]
}

Because I’ve used as "password": "{{TTN_app_pw}}", the password provided in TTN application overview

The {{}} variables are stored in the Postman environment in order to make easier the HTTP requests.

And I’m not sure whether this is the required password because the FIWARE docs doesn’t specify such things.

Thank you and I hope this doesn’t violate the forum rules (because this post hasn’t much to do with TTN, as I’ve said.)

[arjanvanb (Arjan)]

@electronics1920

Decoding CaynneLPP message:+XQ=
Error decoding CaynneLPP message:Error: Invalid CayennLpp buffer size

So, what is the payload you’re sending? Assuming +XQ= is the Base64 encoded payload, then in hexadecimal the payload would be the two bytes 0xF974. Did you try to decode that yourself? Just two bytes seems way too short for Cayenne LPP, to hold channel, type and value. Even more, I don’t think that LPP defines the type 0x74?

(Aside, I am not a fan of people posting the same thing in 3 places. It’s up to you to keep all 3 in sync now.)

[electronics1920]

Yes, in fact TTN is decoding it as you can see here

It is not the same payload because the sensor value has changed from yesterday but I’m providing the following decoder to TTN (in Payload Formats tab)

// TTN Decoder for TTN ABP dendrometer EU868
function Decoder(bytes, port) {
  // Decode an uplink message from a buffer
  // (array) of bytes to an object of fields.
  var decoded = {};
  
  // Potmeter
  rawPot = bytes[0] + bytes[1] * 256;
  //decoded.potVal = sflt162f(rawPot) * 1023;
  //decoded.potVal = parseFloat((sflt162f(rawPot) * 1023).toPrecision(2));
  decoded.potVal = parseFloat((sflt162f(rawPot) * 1023).toFixed(2));
  
  return decoded;
}

function sflt162f(rawSflt16)
	{
	// rawSflt16 is the 2-byte number decoded from wherever;
	// it's in range 0..0xFFFF
	// bit 15 is the sign bit
	// bits 14..11 are the exponent
	// bits 10..0 are the the mantissa. Unlike IEEE format, 
	// 	the msb is transmitted; this means that numbers
	//	might not be normalized, but makes coding for
	//	underflow easier.
	// As with IEEE format, negative zero is possible, so
	// we special-case that in hopes that JavaScript will
	// also cooperate.
	//
	// The result is a number in the open interval (-1.0, 1.0);
	// 
	
	// throw away high bits for repeatability.
	rawSflt16 &= 0xFFFF;

	// special case minus zero:
	if (rawSflt16 == 0x8000)
		return -0.0;

	// extract the sign.
	var sSign = ((rawSflt16 & 0x8000) !== 0) ? -1 : 1;
	
	// extract the exponent
	var exp1 = (rawSflt16 >> 11) & 0xF;

	// extract the "mantissa" (the fractional part)
	var mant1 = (rawSflt16 & 0x7FF) / 2048.0;

	// convert back to a floating point number. We hope 
	// that Math.pow(2, k) is handled efficiently by
	// the JS interpreter! If this is time critical code,
	// you can replace by a suitable shift and divide.
	var f_unscaled = sSign * mant1 * Math.pow(2, exp1 - 15);

	return f_unscaled;
	}

And it’s perfectly decoding the value as the picture shows. But I thought the IoTAgent-LoRaWAN was retrieving that decoded info from TTN.

Also … yes, I don’t like also post the same thing in several sites but in some of them (as in GitHub) I don’t hope an answer soon. Of course I’ll update all of them when I got a solution, don’t worry

Thank you for your answer again @arjanvanb.

PS: The decoder, as you know, is provided by lmic library

[arjanvanb (Arjan)]

So, you’re using your own decoding, and are not using LPP at all. If you want some third party to decode LPP, then you need to make your device send LPP. (Note that you cannot make a Payload Format Decoder output valid LPP.)

@electronics1920

Also … yes, I don’t like also post the same thing in several sites but in some of them (as in GitHub) I don’t hope an answer soon. Of course I’ll update all of them when I got a solution, don’t worry

No, I don’t think you can wait for a solution. When posting in multiple places, you need to keep all of them in sync to avoid wasting people’s time in posting something that was already posted by someone else in one of the other places. So, no sleep for you, and 24/7 monitoring of all occurences, to keep them in sync, until the end of time. Really, I find it so extremely rude to do this! (But: kudos for warning us.)

Anyway, now that you got your solution, please quickly remove the other two occurences of your question, to avoid people wasting any time on that.

[electronics1920]

Of course, I’m answering myself in StackOverflow and closing the issue in GitHub after a few more clarifications

So … The point is I must switch the encoder to the LPP because FIWARE doesn’t allow third party encoding? This is my Arduino sketch and that’s how I’m encoding the float read from the analog pin, so … I gues the point is I have to use CayenneLPP to encode that float in the sketch itself?

[electronics1920]

I appreciate so much your help I still would like a bit of help in order to know if I could switch the part of my sketch which is encoding the float for the corresponding CayenneLPP encoding

        // Prepare upstream data transmission at the next possible time.
        
        // Read the analog value of the potmeter (0-1023)
        potVal = analogRead(potPin);
        
        // Write the value to the serial monitor
        Serial.println(potVal);

        // adjust for the f2sflt16 range (-1 to 1)
        potVal = potVal/1023;

        // Write AGAIN the value to the serial monitor
        Serial.println(potVal);

        // float -> int
        // note: this uses the sflt16 datum (https://github.com/mcci-catena/arduino-lmic#sflt16)
        uint16_t payloadPot = LMIC_f2sflt16(potVal);

        Serial.println(payloadPot);
        
        // int -> bytes
        byte potLow = lowByte(payloadPot);
        byte potHigh = highByte(payloadPot);

        // place the bytes into the payload
        payload[0] = potLow;
        payload[1] = potHigh;
        
        // prepare upstream data transmission at the next possible time.
        // transmit on port 1 (the first parameter); you can use any value from 1 to 223 (others are reserved).
        // don't request an ack (the last parameter, if not zero, requests an ack from the network).
        // Remember, acks consume a lot of network resources; don't ask for an ack unless you really need it.
        LMIC_setTxData2(1, payload, sizeof(payload)-1, 0);

Or should I use just the CayenneLPP library and drop the arduino-lmic? Because I’m watching the suggested example in here and I’m afraid I cannot combine both (CayenneLPP and arduino-lmic) because that CayenneLPP example is using also your
TheThingsNetwork.h library, is that right? I mean, I would have to rewrite the whole sketch? Could I fix my sketch just to use CayenneLPP and encode the payload in another different way?

[electronics1920]

@arjanvanb I was thinking in replace this part with something like this

void do_send(osjob_t* j){
    // Check if there is not a current TX/RX job running
    if (LMIC.opmode & OP_TXRXPEND) {
        Serial.println(F("OP_TXRXPEND, not sending"));
    } else {
        // Prepare upstream data transmission at the next possible time.
        // Read the analog value of the potmeter (0-1023)
        potVal = analogRead(potPin);
        
        // Write the value to the serial monitor
        Serial.println(potVal);

        // prepare upstream data transmission at the next possible time.
        // transmit on port 1 (the first parameter); you can use any value from 1 to 223 (others are reserved).
        // don't request an ack (the last parameter, if not zero, requests an ack from the network).
        // Remember, acks consume a lot of network resources; don't ask for an ack unless you really need it.
        
        lpp.reset();
        lpp.addAnalogInput(1, potVal);

        LMIC_setTxData2(1, lpp.getBuffer(), lpp.getSize(), 0);
        Serial.println(F("Packet queued"));
    }
    // Next TX is scheduled after TX_COMPLETE event.
}

But I’m not sure if this is going to work because I’m using the LMIC_setTxData2() to send the data.

[electronics1920]

I gave a try and it’s apparently working but I don’t know why:

• The decoded value is not in the [0, 1023] range and why sometimes is negative
  
• Now the IoTAgent-LoRaWAN is decoding the payload but not casting the message to NGSI

fiware-iot-agent | {"timestamp":"2020-06-24T15:36:15.263Z","level":"info","message":"New message in topic"}
fiware-iot-agent | {"timestamp":"2020-06-24T15:36:15.264Z","level":"info","message":"IOTA provisioned devices:"}
fiware-iot-agent | {"timestamp":"2020-06-24T15:36:15.265Z","level":"info","message":"Decoding CaynneLPP message:AQIVWA=="}
fiware-iot-agent | {"timestamp":"2020-06-24T15:36:15.265Z","level":"error","message":"Could not cast message to NGSI"}

What do you think? I’d say the problem is not anymore in Arduino Sketch or TTN but the IoTAgent-LoRaWAN or OrionCB (FIWARE)