LMIC-node on ESP32 not working when Heltec does

End Devices (Nodes) LMIC
1

tl;dr
What is a good common off the shelf device to get in the US to test the configuration and operation of The Things Internal Gateway configured on The Things Industry Cloud.

[details]
Hi,
I am new to LoRaWAN and TTI and am interested in finding out more about it. The application domain of interest is the stabilization of time-series of residential resource consumption data (e.g. water, electricity) transmitted via LoRa to a number of specially provisioned LoRaWAN gateways in densely populated urban areas. I have been programming professionally for over 30 years, initially as what used to be referred to as ‘Systems programming’ and now work mostly in Python and databases.

For this experiment, I am pretty agnostic to everything other than LoRaWAN as I want to gauge the level of complexity with setting up (getting one application, gateway and single TTI tenant) and scaling out (getting many such things (applications X gateways X tenants) working).

@descartes advised me via Slack that The golden rule with LoRaWAN is get a gateway, a COTS device & an LMIC based device going before starting to design anything
To this end, I have:

  1. TTIG gateway (in the Boston suburbs in the USA) and https://tgrid4all.nam1.cloud.thethings.industries/ instance.
  2. LMIC-node as a “LMIC-based device” with a wifi-lora-32-v2
  3. I do not have anything picked out as a COTS device but am open to suggestions for something like a temperature/humidity sensor that is known to work in the US

I have been unable to get LMIC-node working on a Heltec board with OTAA and a perusal of the forum posts suggests that this may be due to node being too close to the gateway (the messages I see in the Live Data for the gateway report a SNR of 10.25 and RSSI of -60 to -70. There is also a suggestion that Rx1 needs to be set to 5 seconds for TTNv3. Once I get a COTS device working, I will return to the LMIC-based test.

thanks in advance,
pjjH

LMIC-node

Device-id:     wifi-lora-32-v2
LMIC library:  MCCI
Activation:    OTAA
LMIC debug:    2
Interval:      60 seconds

RXMODE_RSSI
000000029349:  Event: EV_JOINING
31392: engineUpdate, opmode=0x4
000000031455:  Event: EV_TXSTART
33571: TXMODE, freq=904700000, len=23, SF=10, BW=125, CR=4/5, IH=0

On the console, I see the uplink messages like this (main bulk of the message removed for readability)

  "data": {
    "@type": "type.googleapis.com/ttn.lorawan.v3.UplinkMessage",
    "raw_payload": "AFoulALF1Mh6cLPVftgABPIZdr9ptto=",
    "payload": {
      "m_hdr": {},
      "mic": "v2m22g==",
      "join_request_payload": {
        "join_eui": "7AC8D4C502942E5A",
        "dev_eui": "F20400D87ED5B370",
        "dev_nonce": "7619"
      }
    },
2

Looks like some of the important things are already working, the LMCI stack runs, and sends a join request, the join request is received by the gateway.

An RSSI of -60 to -70 is just loud and clear, doesn’t look “excessive” to me.
(I think I get -47 dBm across the room with messages arriving just fine).

I think the next part is checking if the network is sending a join accept back.
It’s very easy to get something wrong in the device EUI, app/join EUI and app key.
In that case, you’ll probably see something like “checksum mismatch” in the live TTN console and the network will simply not send anything back to the device.
Most likely cause is that have to enter the bytes for one of these in reverse order.
If you look in the live part of the TTN console for the gateway, you can determine if the bytes are in the order you expect, i.e. the same order as you entered them in the end device properties.

If you use OTAA (and it looks like that), things like the receive window delays will be configured automatically, that’s the beauty of OTAA (over-the-air-activation). It does not only set up the device address and encryption keys but the procedure also sets up some network parameters.

3

Excepting that in LMIC for OTAA the bytes are little endian (lsb) as @bluejedi highlights in the key file both in the description and for the Dev + App EUI.

The console displays as big endian but when you use the hex array / clipboard tool there is also a byte swap tool to make it easier to transfer over.

When you click the <> icon, it will then show the byte swap arrows to the left.

Screenshot 2022-01-02 at 11.38.08

If you are not seeing anything in the device console, I’d suspect this byte swap as being the issue.

There is a lot of bike shedding about RSSI, SNR and gateway/device proximity which can go around the houses for some time - soooooo much quicker to try it (should take about 2 minutes to move the device on a battery pack), so 5m & a brick wall please.

As for the Golden Rule of debugging your LoRaWAN, it’s mostly ignored if you have something working, but the acid test for any setup is what we call the Canary - a known good device that transmits and you see the uplink on your gateway (internal) log, the gateway console, the device (or application) console and then out via your integration of choice. For first builds we all help with the device and then it turns out something silly was up with the gateway.

If I was to buy a COTS (I sort of make them and I already have quite a few anyway), I’d get either a Dragino Door / Water sensor on the grounds they are very inexpensive or a Dragino LT-22222-L which has inputs & outputs to play with.

I’m suggesting Dragino as they are a no nonsense manufacturer that has resisted gold-plating everything. Otherwise, pretty much any other small device you find that is stated as being for US bands will suffice - it’s non-trivial to create a device & not something even Walmart stock, so anyone with a solid looking website selling devices isn’t doing it randomly.

But we are so close to getting your ESP working, so don’t rush out to buy anything just yet.

PS, going large will be about a year long journey - and tenants aren’t really a thing unless you rent an enterprise edition of TTS to a disparate range of clients - mostly organisations is more than sufficient for a solution provider & requires far less admin.

4

PS, as you’ve said the Heltec + TTIG are working which means your TTIG and your instance is working which contradicts the title and that it’s the ESP32 that isn’t, I’ve adjusted the title & category accordingly.

5

Thanks for the help. I see a variety of messages relating to LMIC-node in the TTI console.

I have ordered a Dragino ( LT-22222-L LoRa I/O Controller) which I will use for development. This looks pretty perfect for my sensor experiments needs (as a ‘cuckoo canary’!)

regards,
pjjH

Details

I have at least one issue remaining which looks like Link ADR rejection received - #2 by Hulkco

If you are not seeing anything in the device console, I’d suspect this byte swap as being the issue.

That was the issue. I was aware from the comments in the lorawan-keys.h file about 2 or the 3 inputs being lsb but I had not noticed the byte order toggle and I incorrectly assumed that the TTI UI was emitting the array-initializer in the form usable by LMIC. This was an eyesight error on my behalf (I have a strong preference for text-based interfaces. see below) and I now realize I had an expectation that TTI console would complain to me about Dev and App EUIs that were unknown.

From the LMIC-node documentation:

Registration is not further described here

I think I may submit a PR for this! I note that the end-device key data is retrievable via JSON from the command-line (here is for the end-device; I assume the app key can be retrieved similarly). I would rather codegen the file from this rather than go through n clicks on the TTU web UI. Similarly, it is a bit of a pain to copy paste stuff from the device definition repository via the web UI. It would be nice to have a script which probed the MCU and looked up its characteristics in the repo and added the device via the API (either command-line or HTTP)

C:\packages\ttn>.\ttn-lw-cli.exe end-devices list --application-id tgrid-lmic-test
[{
  "ids": {
    "device_id": "eui-70b3d57ed80004f2",
    "application_ids": {
      "application_id": "tgrid-lmic-test"
    },
    "dev_eui": "70B3D57ED80004F2",
    "join_eui": "5A2E9402C5D4C87A"
  },
  "created_at": "2022-01-01T22:32:27.796Z",
  "updated_at": "2022-01-02T15:41:05.100Z"
}]

Please find attached a snippet of the log from startup

LMIC-node

Device-id:     wifi-lora-32-v2
LMIC library:  MCCI
Activation:    OTAA
Interval:      60 seconds

000000029681:  Event: EV_JOINING
000000031742:  Event: EV_TXSTART

000000033857:  doWork job started
000000374814:  Event: EV_JOINED
               Network Id: 19
               Device Address: 26092A4E
               Application Session Key: 21-1A-6D-FD-F1-1A-A8-DC-FD-7C-50-23-48-2C-C2-BE
               Network Session Key:     A8-4E-0B-C0-04-67-6E-44-E7-1E-49-93-82-E6-3C-64

000000377732:  doWork job started
000000381059:  Input data collected
               COUNTER value: 1
000000381944:  Packet queued
000000384171:  Event: EV_TXSTART
000000725377:  Event: EV_TXCOMPLETE
               Up: 1,  Down: 1
               Downlink received
000001469086:  Event: EV_TXCOMPLETE
               Up: 3,  Down: 3
               Downlink received
               RSSI: -61 dBm,  SNR: 9.0 dB
               Port: 0
000001499731:  Event: EV_TXSTART
000001840963:  Event: EV_TXCOMPLETE
               Up: 4,  Down: 4
               Downlink received
               RSSI: -63 dBm,  SNR: 8.7 dB
               Port: 0
000001871608:  Event: EV_TXSTART
000002212842:  Event: EV_TXCOMPLETE
               Up: 5,  Down: 5
               Downlink received
               RSSI: -60 dBm,  SNR: 9.0 dB
               Port: 0
6

LMIC is the open-source MAC that has only really just got it’s big boy pants on in the last few months. LoRaMAC-node is the official reference source and it doesn’t need to have things to wrong way round. The grown-ups have a secure element with the EUI’s & key pre-provisioned and would take a completely different path to registration.

How would it know - the airwaves are full of LoRaWAN signals - the Network Server takes incoming uplinks from gateways and has to process them in stages to get to the ones that are on it’s Join Server. To put it in context, TTN gateways deliver ~425 uplinks per second, only ~90 are for TTN devices.

I will look forward to Leonel’s robust reply to this - particularly as the very next sentence is:

For more information see The Things Network documentation, The Things Stack documentation and The Things Network Forum.

LMIC-node is all about the device firmware whose author happens to use TTN rather a lot - but it can be used against any LoRaWAN network and it’s a bit much to ask the firmware person to write a manual on the SysOps job when there is already plenty of documentation around.

All of these exist, but you can’t run until you can walk. Otherwise we would have been adding more tools - the CLI and, most likely Python as that’s what I tend to use for admin tools - before you’d used the core facilities - and you have dived in to using a private instance which means none of what we type will be directly applicable to the vast majority of users on the forum as the URL’s will be different.

And the exact tool you are asking about is exactly the sort of thing that the community produces, so, congratulations, we look forward to your up & coming GitHub repro with a command line tool that you enter your app & dev id and it creates the lorawan-keys.h for you.

It’s great you have the device working, it’s about par for the course that you missed some of the details along the way, you’ve uncovered level 1 of LoRaWAN, there a good few levels plus many nuances to go, so expect there to still be some road humps along the way. LoRaWAN hasn’t been around so long & the servers & firmware haven’t existed in stable enough form that detailed documentation has evolved - so best not to take 2 & 3 and end up with 5, think that it may be -1 or 4.99.

But now you have uplinks going, here’s a present to help start getting the data out:

It is expected you will consume the documentation - both the overviews plus the reference, but here’s about the closest that’s available as instructions: GitHub - descartes/TheThingsSummerAcademy2021-Integrations: The Things Summer Academy 2021 - MQTT & HTTP Integrations workshop materials and if you want to stick pins in my image, this: https://www.youtube.com/watch?v=yMXlTDljz_E which the Summer Academy material was developed from.

All we’d ask is that you read many many forum articles before asking questions about downlinks.

1 Like
7

My meaning of PR == github Pull Request (from your comment, I think you may have assumed I meant “Problem Report”?). It looks like the header-file generation is rather straightforward so I was either going to contribute the code or a patch to the example file to include a link to the code.

Once again, thanks for all the help.

8

I’m au fait with a GitHub PR - what I very much doubt is the appropriateness of providing a script that needs the CLI or a web API to remotely reach out and access the key - I’d not want it if I was the author as it starts mission creep and issues with separation of concerns.

Having a set of CLI wrappers or web API scripts for development & management of devices that simplify’s the interface would be useful. So once it has your login / API key setup it can give you menus - for instance a list of applications and then the option to do things with the devices, register a new one etc etc, which is the CLI equivalent of the web interface but without the CLI in all it’s glory (see http://descartes.co.uk/TTSv3CLI.html as a helper - it’s not up to date and TTI have now effectively replicated the same UI on their documentation website) would be useful. But those that can have but there is a limit to what you can release before you start finding your offering generates support issues - the starter integrations generate questions from people that start out with “I don’t know how to program, but how do I …”

The reality is that for the occasionally device setup, hacking away at the web console is fine for most. If you need to register 1,000 devices, then you are in a different ball park and scripting makes sense. Or a device management database.

Please do not confuse what some of us do on here with what pays our bills, we have created the tools but we can’t afford the time to support them if we released them, so it’s better that we direct people to the tools that are known to work, however inefficient for larger scale deployments & management, than make all our lives miserable.

But you can still publish your “lorawan-keys.h” creator - many will find it useful.

9

I did some thinking about what kind of tool would have been useful to me in getting LMIC-node set up. As per @descartes advice, LMIC-node does seem a great way to start to experiment with LoRaWAN. My misgivings about a C-based system (specifically, having to install a huge toolchain) were completely obviated by the use of platformIO (since it appears to do all the installations for you behind the scenes). However, I found I made a large number of errors, misunderstanding and misassumptions while attempting to follow the instructions and related documentation (I spent about 6-10 hours on it, all told).

I present a sketch of the general approach I plan to follow for TTI-related experiments and the specifics of dealing with transformation of keys to representation usable (after some manual steps) for lorawan-keys.h. I will start a repo on this once if the technique seems generally fit for use. I spent about 2 hours reading the jq manual, fiddling with the ttn-lw-cli command-line and googling/writing the transforms.

I welcome opinions, criticisms of the approach and suggestions. One of the things I plan to do immediately is to scrape my TTI definition on a regular basis to git (or immediately post change) so that I can figure out what was different.

pjjH

I gave a high positive weighting to being a data-oriented, “programming is my day job” programmer with a bias towards scriptable, text-oriented interfaces (revision control friendly for the scripts and their output). I also want something that is composable wrt other tools and is pretty agnostic. I have very little domain knowledge on TTI, LoRaWAN and need as much help as I can get from my tools. I want to do experimentation with a wide variety of hardware and have some kind of testing/regression prevention capability.

The TTN cli tool seems to be be code-generated or metadata-driven from the Web API. For purposes of reading TTI configuration and operational data without much programming, there does not seem to be anything comparable.

Somewhat reluctantly (but, guided by the principals I outlined, consistently) I ended up with jq. Despite me having to read the manual each time I use it, I find that for dealing with large clumps of JSON, it is much less bad than anything else I have used thus far.

This combination of choices yields a somewhat clunky but very precise command-line (for Windows. I developed on Windows and wsl2):

(venv) C:\packages\ttn>.\ttn-lw-cli.exe  end-devices get tgrid-lmic-test eui-70b3d57ed80004f2 --root-keys.app-key | jq -f ttn_to_lmic_defines.jq
{
  "OTAA_DEVEUI": "0xF2, 0x04, 0x00, 0xD8, 0x7E, 0xD5, 0xB3, 0x70",
  "OTAA_APPEUI": "0x7A, 0xC8, 0xD4, 0xC5, 0x02, 0x94, 0x2E, 0x5A",
  "OTAA_APPKEY": "0xBD,  <redacted>  0xBE"
}

This is the main ‘script’: it plucks 3 items from the JSON blob and reformats them (as array initializers) suitable for copying and pasting or templating, There is no other logic. It is a blob of JSON in and a carefully prepared blob of JSON out. I don’t have an editing mode for jq. I like very much that the transforms can be worked on incrementally and interactively from the command-line. There are also interactive ‘playgrounds’ online but I prefer using bash (and command history especially), diff etc.

include "ttn";
# the correspondance between the source and target is concise
{OTAA_DEVEUI:(.ids.dev_eui | lsb), OTAA_APPEUI:(.ids.join_eui | lsb),OTAA_APPKEY:(.root_keys.app_key.key | msb)}

The jq helper functions are concise and composable (because that is generally how jq is)

# helper functions to 'do stuff' with JSON result payloads from the TTN
# V3 API and the ttn-lw-cli executable
# Wrote this in jq since the input is JSON and want this to be a minimal requirement footprint
# jq is a bit quirky but it is very composable, very widely used and very powerful.

# from https://stackoverflow.com/a/31757661/40387
def pairs: . as $arr| [ range(0; length/2) * 2 | $arr[.:.+2] ];

def hexprefix: "0x"+.;

def lsb: pairs | reverse | map (hexprefix) | join(", ");
def msb: pairs | map(hexprefix) | join(", ");
10

Having looked at jq some time ago I found it a whole lot easier to write the equivalent code in python. I found the result a lot more maintainable as well (without giving me headaches every time I needed to change something).
The only time I really used jq was on an embedded Linux system where there was no python but jq was present.

11

Thank you for your comments.

I too find it much easier to write the equivalent code in Python. There is a catch though!

I frequently have had the opposite experience wrt maintenance. While the declarative languages I use are a massive pain in the neck in many ways (SQL, jq, sed) , they end up compensating for that by being very composable and helping prevent me from succumbing to the temptation of mixing in side-effectful code. Perhaps if I were more disciplined about it, I could avoid stuff getting inappropriately entangled. I have come to realize that while I love code and APIs, I love data and schemas more!

in any case, I have addressed my original impetus use-case and will continue on with the others.

I think the original topic can now be closed: I am the very happy and grateful possessor of a working LMIC-node instance and I have learned a great deal in the process. I am regretful of the amount of volunteer support time I took up as a result of my comprehension errors and I hope to be able to contribute back over the next couple of months.

warm regards,
pjjH

techincal detail describing my bias

I used to be an absolute database and hater until I accidently became a DBA 20+ years ago for a short while (about 4 years). I am back to being a programmer again and can’t ‘unsee’ the advantages (to me) of declarative techniques. However, I frequently employ hybrid approaches such as the use of a path-expression library (jsonpath_ng) within Python. I would not dream of trying to do this in something like jq alone.

# This is the query for a (recursive) search an object property 'target'
q = jsonpath_ng.parse('$..target')

# Now find all the matches
match = q.find(data)

resources=[]
for m in match:
    u = urlparse(m.value)
    p = Path(u.path)
    socrata_resource_id = p.name
    if not (len(socrata_resource_id)==9 and socrata_resource_id[4]=='-'):
        continue

    print(socrata_resource_id, m.value, m.context.value['children'][0]['content'])
    resources.append(socrata_resource_id)

I also am prepared to go in the other direction and use Python to craft command-line invocations that can do what I need to do way more effectively than coding it myself. For example, curl is another program that I end up using frequently and yet has such a cumbersome (to me) user-interface that I have to look up the manual every single time I use it.

# here is how to generate the curl command that you can copy and paste to the command-line.
resource_ids=','.join(resources)
print('curl  "https://data.cityofnewyork.us/api/views/{%s}/rows.csv?accessType=DOWNLOAD&api_foundry=true" -o "#1.csv"'
12

@phrrngtn

LMIC-node is:

  • An easy to use example application to help you get a LoRaWAN node quickly up and running with The Things Network. It demonstrates how to use and implement basic LoRaWAN features using the LMIC library (end device LoRaWAN stack).

LMIC-node is not:

  • a ‘LoRaWAN and The Things Network step by step’ tutorial.
  • a course for how to use The Things Network.
  • a course for how to use the Arduino framework and upload firmware to your device.

Before availability of LMIC-node, getting a DIY (LMIC-based) LoRaWAN node up and running was not a simple task and required a steep learning curve with multiple pitfalls waiting to be stepped in to.

With availability of LMIC-node that is no longer the case. With LMIC-node (and a supported board) getting a LoRaWAN node up and running has become very simple.
And while LMIC-node makes things simple and easy, there is still some homework to do yourself (e.g. registering a device with a network provider like The Things Network).

You don’t need a tool for setting up LMIC-node. LMIC-node already makes things easy. LMIC-node already provides things on a silver platter (for free), the gold-plated version will cost €2000 more.

As @descartes already mentioned: you will first have to learn how to walk before you can run.
Learning means that you will have to learn and master the basics yourself. The standard admin tool for The Things Stack is the Console (not more advanced automated scripting that has to be learned and installed separately).

13

I am grateful for it. I shared some observations I drew from my experience and recorded them here as anecdotal information from a single user. I appreciate your enumeration of some of things that LMIC-node is not. I did not find the time spent in the web UI(*) as rewarding in comparison to the time spent reading the things stack API documentation and using the command-line tool.

pjjH
(*) I found the use of the web UI Console to be confusing and/or buggy e.g. I had to follow the tooltips on the website to see which items have been renamed e.g. OTAA_APPEUI comes from the JoinEUI column of the table in the ‘End Devices’ part of the UI (which offers copy and paste but not the serialization as an array initializer and the byte order control) and then found out that that corresponds to AppEUI by mousing over AppEUI and seeing a mention of JoinEUI.

My experience with the command-line tool was quite different: the installation (on Windows) was a simple unpack; I ran the program with --help and was able to find the option I needed. I followed the online instructions for logging in and authenticating via OAuth. Very nice. I was able to dump out the configuration of the application and peruse it in a text editor and search for the various magic strings. I could run it from the command line and grep for particular strings. I was able to change things in the web console, re-run the script and then use diff to compare the outputs. I could make multiple device registrations in the UI and see how the output changed.