Parsing payload in Python to send from RPi based node

End Devices (Nodes)
1

Hello how it is possible to parse and count from a JSON file and direct the print of it into the lora.send() command in python. I used to parse it but im getting errors sent as im getting a python list instead of a HEX. My JSON sample below. For that example i want to count the object names i get. Eg Vase =2 Book = 4 etc.
Any help on this

[
{
“frame_id”:1,
“filename”:“sample_imgs/1.jpg”,
“objects”: [
{“class_id”:75, “name”:“vase”, “relative_coordinates”:{“left_x”: 457, “top_y”: 119, “width”: 17, “height”: 36}, “confidence”:0.309923},
{“class_id”:75, “name”:“vase”, “relative_coordinates”:{“left_x”: 475, “top_y”: 138, “width”: 10, “height”: 15}, “confidence”:0.289516},
{“class_id”:73, “name”:“book”, “relative_coordinates”:{“left_x”: 291, “top_y”: 276, “width”: 53, “height”: 14}, “confidence”:0.778392},
{“class_id”:73, “name”:“book”, “relative_coordinates”:{“left_x”: 242, “top_y”: 282, “width”: 57, “height”: 15}, “confidence”:0.401355},
{“class_id”:73, “name”:“book”, “relative_coordinates”:{“left_x”: 292, “top_y”: 284, “width”: 54, “height”: 11}, “confidence”:0.382199},
{“class_id”:73, “name”:“book”, “relative_coordinates”:{“left_x”: 306, “top_y”: 268, “width”: 21, “height”: 12}, “confidence”:0.259332},
{“class_id”:58, “name”:“pottedplant”, “relative_coordinates”:{“left_x”: 233, “top_y”: 111, “width”: 106, “height”: 171}, “confidence”:0.668581},
{“class_id”:57, “name”:“sofa”, “relative_coordinates”:{“left_x”: 1, “top_y”: 225, “width”: 290, “height”: 133}, “confidence”:0.982571},
{“class_id”:56, “name”:“chair”, “relative_coordinates”:{“left_x”: 347, “top_y”: 234, “width”: 234, “height”: 190}, “confidence”:0.971144},
{“class_id”:56, “name”:“chair”, “relative_coordinates”:{“left_x”: 558, “top_y”: 229, “width”: 53, “height”: 136}, “confidence”:0.462609},
{“class_id”:45, “name”:“bowl”, “relative_coordinates”:{“left_x”: 261, “top_y”: 271, “width”: 25, “height”: 15}, “confidence”:0.853924}
]
}
]

2

Curious how there is a similar post on the RAK forum - same collection count, same use of lora.send function. I wonder if this is a student assignment?

Have you tried a Google search such as “python count number of occurrences in list”.

I’m not sure why you would expect a response in hex.

And if you look on the RAK forum, you’ll see my response about the expectations of the lora.send() function.

1 Like
3

I tried on the RAK forum and what i got (not from you) was some nosense out of topic for the most part of what i was asking. Thing is i either expect a HEX which i later decode in my application in plain text or a text string directly to lora.send(). What i tried so far i couldn’t relay the message as its not supported by lora (python list)

4

Sounds like you have a basic python issue, may-be ask the question in a generic python forum?

5

Hi @edukavala, it seems you need to do 2 things. Both use the python3 json module to encode/decode JSON and python strings/dictionaries.

https://docs.python.org/3/library/json.html

Send data that is currently in a JSON file. Python pseudo code is:
import json
your-string = json.dumps(your-JSON)
lora.send(your-string)

Count occurrences in a JSON file. Python pseudo code is:
import json
your-dictionary = json.loads(your-JSON)
for your-item in your-dictionary
… check for match, do stuff, etc.

However, your JSON file looks to be way too large for a single send and you should NOT be sending ascii/string/JSON data over a constrained bandwidth network like LoRaWAN.

6

I have already parsed, count and print my JSON file, though i cant pass this
{‘book’: 4, ‘chair’: 2, ‘vase’: 2, ‘bowl’: 1, ‘pottedplant’: 1, ‘sofa’: 1}
but i can this if i use an asterisk to my print(*output)
[book, chair, vase, bowl, pottedplant, sofa].
What i want though is to pass book = 4, chair = 2, etc. Method above just eliminates the numbers associated with counts.
Yes it is a python issue.

@cultsdotelecomatgmai
Im sending only limited things, not all JSON as i know this limitation. But send in ASCII eg book = 4, chair = 2 is 19 bytes and its equivalent in HEX is more

7

If there is a limited range of objects you detect you can change for instance book to be the number 1, in binary format that is just 1 byte in stead of 4 ascii characters. You can represent 256 objects that way. Then the next byte could be the count (if the range is 1 to 255, otherwise use two bytes)
That way your book/chair string could be reduced to 4 bytes in stead of 19. Far more efficient…
By transforming the data that way you end up with a binary array that you can transform to whatever format the Lora.send call requires. (That depends on whatever Lora implement in python you are using and you didn’t mention that)

8

Just because i wont know in any given instance on first element what object would be for the lora to convey that would not work. If i gathered correctly you imply to map a list of objects from 1 to 255 and then just decode in my application?
Sorry im using a Pi4 with a pHat on top if that’s what you ask about my implementation.
For example as the extracted JSON gives me the following list of data
{‘book’: 4, ‘vase’: 2, ‘chair’: 2, ‘bowl’: 1, ‘pottedplant’: 1, ‘sofa’: 1}
I should simplify to something without bracket or colons. What i gathered so far it would only accept basic strings.

9

How many objects can your code recognize? Is that a limited (countable) set like table, chair, sofa, potted plant and a few more? Or could the objects be literally anything?

10

The requirement for recognition are limited, what i can think now is about 5 objects at a time. But my basic requirement is how many times the same object appear. Eg persons 10, dogs 2. its just a python tweak for now to run unquoted something like the following (‘book’, 4) (‘vase’, 2) (‘chair’, 2) (‘pottedplant’, 1) (‘sofa’, 1) (‘bowl’, 1)

11

You must not send strings.

Identify the maximum number of object types and how many bits that takes to represent as a type number (type “enum” if you like). Identify the maximum count and how many bits that takes. Send some number of [type][count] binary conglomerations that fit in packet.

Recognize that LoRaWAN may not be (realistically almost certainly is not) a fit for your application. You haven’t even begun to think about surrounding issues such as the fact that the traffic is not, and realistically cannot, be specifically confirmed as received.

12

So now you assign a number to each possible object you can recognize. For instance book = 1, vase = 2, chair = 3, potted plant = 4 and so on. This mapping is static for all your messages.
Now you want to report 2 vases and one potted plant so you create an array with the numbers 2 2 4 1. Send that 4 byte message and in the backend use an array with object names in the same order as on your node. Then use the first and third number in the received message as array index resulting in vase and potted plant and elements 2 as the number of vases and element 4 as the number of potted plants.

13

I’m not sure the bit about doing an OTAA join every time you run the script could strictly be described as nonsense. It would be much more efficient for testing and kinder to the other devices on the airwaves around you if you used ABP whilst developing - just remember to turn off Frame Counters.

For the rest of the dear readers on here, that would be a RAK811 HAT and you appear to be using the GitHub - AmedeeBulle/pyrak811: RAK811 Python 3 library for Raspberry Pi library or fork thereof.

Your Pi is unlikely to run too many classifiers otherwise you’ll be moving to a Jetson real soon now, so you must know what it’s been trained on and therefore be able to give them id numbers, so you end up with the scheme suggested here & on the RAK forum.

At which point you have an array of 8 bit integers like 03051702 where 03 = tree and 05 = five of them and 17 = aardvark and 02 means two of them. This is not a string.

You’d then turn this ‘python byte array to hex string’, which has multiple hits on Mistress Google, the first one having several solutions.

15

Thanks Jac, i’ll see it that way or thinking message concatenation for my pre-payload

16

Regarding the above and no text send, in what way will you send the final value of 0x080F out of your node to TTN based on the example?
i mean within your lora.send(). Give me one example in python, because how much i try in encoding something it seems i sent text with the representation of binary. I’ve sent something just before, converted to string and had no payload port 0 as in photo. In my code i get a result while printing.cropped

17

As that depends on the Python library you are using no one will be able to help you if you do not supply details. Details on what software you are running on your RPi to interface with the Lora HAT.

18

Is this not the same question from before & on the RAK forum?

You populate a binary array and then convert it to a hex string for the lora.send parameter.

It will arrive at the application server as the Base64 encoded binary array.

19

Hi @edukavala, you request…

Herewith one example. This uses python running on an RPi to drive a RAK811 module using “AT commands”. The RAK811 is connected to the RPi via USB/serial and is /dev/ttyUSB0.

First I declare a prototype of the AT command to send. This is done in a module to improve performance:

c3 = 'at+send=0,fp,h1h2\x0D'

Then I use string substitution to replace the fp, h1 and h2 values with the ones that I want to use. The payload is in hex:

    r811.c3 = r811.c3.replace("fp", str(lwtxq.tagtxq_prt), 1)
    r811.c3 = r811.c3.replace("h1", binascii.hexlify(struct.pack('>I', lwtxq.tagtxq_rowid))[4:], 1)
    r811.c3 = r811.c3.replace("h2", lwtxq.tagtxq_hbs, 1)

Then I send the AT command to the RAK811 module and it works, all nicely done as terse hex/binary.

YMMV

3 Likes
20

Jac im using the pyrak811 library for a pHat that sits on top the Pi. Sorry i didn’t mentioned it before.
What i did so far was to print my output to bytes. i get the output in this form b’\x04\x02\x02\x01\x01\x01’
Dunno if that’s the right way. What i was thinking as you suggested it before for some short 16 bit signed integer and map this to my application. I dunno this what what would be the overhead cost and airtime

21

On the current application i had created a tuple in my backend in the form of (0,222,1,40) and i successfully send as a byte string with minimal payload to TTN as 4 bytes total.
Example:
def convertTuple (tup):
str = functools.reduce(operator.add, (tup))
return str
tuple = (a)
str = convertTuple(tuple)
lora.send(bytes(str))

As my application grew i had countable objects above the 256 value which it triggered an error (range must be…). I decided to create a numpy array instead from the tuple and again this array to bytes string. Although now i am able to sent my message from python values above 256 seem to not be displaying correctly to my application(node-red).
npArray = np.array(str, dtype=np.uint16)
print(npArray)
is it a specific decoder function to send a value > 256 as if i use the Cayenne LPP i get correct values up until 256. Following is the printed values of my console.
[ 0 326 1 42]
b’\x00\x00F\x01\x01\x00*\x00’

One note to consider if i “lora.send((1024).to_bytes(2, byteorder =‘big’))” as a test i get 0400 in TTN and [4,0] in my front end. Isn’t it suppose the default decoder decodes the above 1024 value as it does up to 256?