Storage Integration - Not showing payload fields when integers or floats

1

Hello everyone,

I am having some issues trying to recover the values from my devices saved in the storage integration. When the payload is decoded into a JSON format, where all the different values are strings, everything works like charm. However, if the values are floats or integers, when I try to get the last X minutes of data, the response is a 204, saying that there is nothing stored.

I wonder if this is normal or if I am doing something wrong when decoding my payload. When the values of the different objects of my JSON are floats, integers, strings (all mixed), I can recover the payload with MQTT, I can see the the payload in JSON format when I explore the uplink packet in the Data screen… everything is perfect, except that the packets are not stored in the integration. For that I have realised that I need to set each one of the values of the objects of the JSON as strings. Also, that storage only happens if ALL the values are passed as strings.

E.g.

If my payload is decoded with this format, the storage integration let me recover the data (response 200):

{
  "PA": "15.09",
  "frameCounter": "229",
  "sensorID": "123123123"
}

If my payload is decoded with this other format, the storage integration says there is no data (response 204):

{
  "PA": 15.09,
  "frameCounter": 229,
  "sensorID": 123123123
}

Any help? I need the data as floats and integers, not as strings. I have seen the video of the set-up of the Storage Integration and the values recovered when running the test button are mixed, with integers etc. I don’t get why this is not working for me.

Thank you!

1 Like
2 
// TTN Decoder for TTN OTAA Feather US915 DHT22 Sketch
// Link: https://github.com/mcci-catena/arduino-lmic/blob/master/examples/ttn-otaa-feather-us915-dht22/ttn-otaa-feather-us915-dht22.ino
function Decoder(bytes, port) {
  // Decode an uplink message from a buffer
  // (array) of bytes to an object of fields.
  var decoded = {};
  
  // temperature 
 
  rawTemp = bytes[0] + bytes[1] * 256;
  
  decoded.degreesC = sflt162f(rawTemp) * 100;
  
  // humidity 
  rawHumid = bytes[2] + bytes[3] * 256;
  decoded.humidity = sflt162f(rawHumid) * 100;
  
  //motion
  if (bytes[4] == 0x11){
    decoded.motion = true;
  }else{
    decoded.motion = false;
  }
  
  //Wake Count
  decoded.wakecount = bytes[5];
  
  return decoded;
}
 
function sflt162f(rawSflt16)
	{
	// rawSflt16 is the 2-byte number decoded from wherever;
	// it's in range 0..0xFFFF
	// bit 15 is the sign bit
	// bits 14..11 are the exponent
	// bits 10..0 are the the mantissa. Unlike IEEE format, 
	// 	the msb is transmitted; this means that numbers
	//	might not be normalized, but makes coding for
	//	underflow easier.
	// As with IEEE format, negative zero is possible, so
	// we special-case that in hopes that JavaScript will
	// also cooperate.
	//
	// The result is a number in the open interval (-1.0, 1.0);
	// 
	// throw away high bits for repeatability.
	rawSflt16 &= 0xFFFF;
 
	// special case minus zero:
	if (rawSflt16 == 0x8000)
		return -0.0;
 
	// extract the sign.
	var sSign = ((rawSflt16 & 0x8000) !== 0) ? -1 : 1;
	
	// extract the exponent
	var exp1 = (rawSflt16 >> 11) & 0xF;
 
	// extract the "mantissa" (the fractional part)
	var mant1 = (rawSflt16 & 0x7FF) / 2048.0;
 
	// convert back to a floating point number. We hope 
	// that Math.pow(2, k) is handled efficiently by
	// the JS interpreter! If this is time critical code,
	// you can replace by a suitable shift and divide.
	var f_unscaled = sSign * mant1 * Math.pow(2, exp1 - 15);

	return f_unscaled;
	}

I uses this for decoding. This works for me.

3

That’s a Decoder for a very specific device, but indeed it outputs numbers, just like the OP’s Decoder does, and booleans. Are you saying you’re using this with the Storage Integration, and that it shows data when fetching that?

No. It works for me, at least when using Swagger UI (as linked from “go to platform” from the integration’s settings).

Are you testing with Swagger UI? What if you do? What if you don’t set a time in your request (which then defaults to 1 hour)? What if you add a single dummy string value? (You already wrote “storage only happens if ALL the values are passed as strings”.)

You might want to post the Payload Format code.

Did you really test the all-numbers Decoder output? Or is there always some string value? (I don’t have any decoders at hand that output some string along with numbers. Maybe having at least one string value changes things.)

Just for your information: I do have problems fetching nested objects, which yield a string value "map[...]". But numbers are okay.

For example, when TTN Console shows that my Decoder returns some nested object:

{
  "battery": 93,
  "no2": {
    "max": 0.26,
    "median": 0.22,
    "min": 0.19
  },
  "pressure": {
    "max": 107.47,
    "median": 106.49,
    "min": 106
  },
  "temperature": {
    "max": 20.96,
    "median": 20.96,
    "min": 20.96
  }
}

…then Swagger UI gets me:

[
  {
    "battery": 93,
    "device_id": "haarlem-waspmote-1-24-139",
    "no2": "map[median:0.22 min:0.19 max:0.26]",
    "pressure": "map[min:106 max:107.47 median:106.49]",
    "raw": "CDAIMAgwKWgpmSn7ABMAFgAaXQ==",
    "temperature": "map[max:20.96 median:20.96 min:20.96]",
    "time": "2019-11-28T22:36:36.665667086Z"
  }
]

This is the same format that TTN Console shows for “historical data”, when the Storage Integration is enabled.

But above, the "battery": 93 is fine.

Same good results for another application, which shows the following in TTN Console, hence does not include any string values (but does contain some null values, which are not shown in TTN Console):

{
  "humidity": 60.01,
  "temperature": 23.91
}

For the above the integration does include the null values, which TTN Console does not show above:

[
  {
    "device_id": "haarlem-bb-1",
    "humidity": 60.01,
    "raw": "CVcXcQ==",
    "temperature": 23.91,
    "time": "2019-11-28T22:47:11.63668127Z",
    "vcc": null,
    "vdd": null
  },
  ...
]
1 Like
4

@arjanvanb yes this could special. But if I fechting the data with swagger f.e. Yes the decoded data is also in the historical from the data storrege integration.
I get an json object. All data is in side. Also decoded. So practical and easy.

5

Thank you very much for your replies @arjanvanb @MOS-FET

I have check this morning in the Swagger UI and the data are back. I have extracted them by using the REST API with a python script and I got my data stored in my DB. I have no idea about which was the problem (I made no changes form yesterday evening to this morning) but it is self-solved.

I am thinking about the fact of having objects in the JSON returned by the decoder function with the same name but different data types at different moments. Could be the storage integration messing up because of that?

If I detect any other weird behavior I will write again in this thread, but everything seems to work :smiley: !

PS: thank you for the info about the nested objects, @arjanvanb. I will try to avoid them from now on.

6

I doubt that. I assume the integration stores the result of the Decoder once the message arrives. (So the Decoder is not invoked by the integration, like each time when data is fetched, or when the Decoder is changed.) But that’s easily tested. :slight_smile:

7

Yes the integration does also store the decoded data. I thought that was clear for all. For me it was clear.

8

Well, it also stores the raw data (and a timestamp), which allows for re-decoding at any later time as well.

9

Sorry some word wrong in my post before just changed. It stores all! Raw and decoded.