Retiring "matthijskooijman" version of LMIC

End Devices (Nodes) LMIC
1

(repost of https://github.com/matthijskooijman/arduino-lmic/issues/297)

TL;DR: Today, I have retired my version of the LMIC library. As an alternative, I’m now recommending the MCCI version of LMIC, which is based on my port, but has seen much improvements over the years and much better documentation. If you need support for the SX126x, you can consider Lacuna’s port of BasicMAC, which is based on a parallel development based on the original LMIC. Only when you have very tight constraints on RAM or flash should this version still be considered.

Over the years, this version of the LMIC library has not received the attention and time from me that it deserved. I’m afraid that my neglect of this repository and the resulting fragmentation and unclarity around LMIC has not been helpful for the LMIC and LoRaWAN / Arduino community and ecosystem as a whole. I’m sorry that people have run into problems with their projects without proper response, or have invested time in providing pull requests that have not been merged here. However, I hope that taking this step will provide the clarity and focus for users and developers and allows the ecosystem to become stronger again.

To better understand how and why things have happened as they did, let me walk you through some history.

In 2015, I needed to do some LoRa experiments in an Arduino environment, so I looked for existing code that could talk to the SX127x radios. I wasn’t particularly interested in LoRaWAN yet at the time, but LMIC seemed promising and easy to port to Arduino, so I did just that. A few months later, The Things Network launched and I also had a usecase of my own that could make good use of LoRaWAN, so I polished up things to allow using LoRaWAN with The Things Network and a lot of people started using it for this as well.

However, to make the library work on Arduino, I had selectively imported files from the upstream LMIC repository, and moved files around to accomodate Arduino, losing the ability to use this repository outside of Arduino. This meant that importing changes from the upstream LMIC repository was hard and that any improvements made to the core code in this repository would not benefit any other platforms. For this reason, I was immediately quite hesitant with making signficant changes, to this repo to prevent deviating from the upstream version to the point of changes being unmergable between both. The fact that upstream development happened privately, only publishing zips of release versions, also complicated things.

A year or so later, IBM stopped developing LMIC and released their last 1.6 release (under a more liberal license, on my request). For me, this was the incentive to restart LMIC-for-Arduino from scratch, based on the 1.6 version and new license and this time keeping the original file layout and build system intact and adding Arduino support on top of that. My intention was to try and make this the standard LMIC version (by this time, a few people had already forked my repo and applied some improvements they needed), hosted under a github organisation where more people could collaborate on the library. I started this new repo in 2016, ported over more changes from this repo in 2017 and was almost ready to announce it as a rally point for all LMIC development, but for various reasons (mostly a chronic surplus of other projects to work on, but also as the announcement of the LoRaWANMiniMouse and BasicMAC LoRaWAN stacks that cast some doubt on the future of LMIC), I never got around to do so unfortunately (which, looking back, seems like a bad choice, but well, here we are).

Early 2019, the BasicMAC LoRaWAN stack was also published, which is a privately developed continuation of LMIC, which AFAIU was later acquired and released by Semtech. Initially, this seemed like it could maybe serve as a better starting point for renewed LMIC-based development. Conveniently, Lacuna was in need of a LoRaWAN stack that supported the newer SX126x chip series and was willing to sponsor porting BasicMAC to Arduino, which I undertook late 2019 and early 2020. This time, I used the repository setup and scripts I had created for the “new” version of LMIC before, to allow the Arduino support to co-exist with the existing platforms and build system. However, since then Semtech has again stopped BasicMAC development, there is still a lot of things in BasicMAC that would need to be further improved, I found that the memory usage is significantly larger (too big for 328p as it is now) and porting over the improvements separately made to LMIC over the years (especially in the MCCI version) would be quite a large, if not impossible task. There does seem to be one other active fork by one of the Semtech developers, but that already deviates from the Lacuna version. Maybe someone will still pick up the work of unifying all these developments, or maybe some of the BasicMAC improvements can be ported back to LMIC instead, but realistically I will not be the one to pull that cart.

In the meanwhile, I mostly neglected this repository, leaving pull requests unreviewed and unmerged and leaving support requests unanswered most of the time. I didn’t want to mark this repository as officially unmaintained until I had the new repo published as an alternative to recommend, but that never happened.

Over the years, the MCCI fork of this repo did see additional development, including some significant structural changes, making my original plan of restarting development with a clean repo less and less feasible. But keeping this repository in this “limbo” state of maintenance is also bad, so the next best plan seemed to just retire my repository and recommend people to use the MCCI version instead. But I didn’t want to do that blindly, without trying the MCCI version in my own projects, which I also did not get around to in the last months.

Recently I finally gave the MCCI version a whirl and found it looks very good. Documentation is a lot more complete, it contains a lot of extra fixes and features and still works pretty much as a drop-in replacement. I did find that it uses a bit more flash and RAM (which makes sense given extra features), but still small enough to run in our 328p-based project. For very-resource-constrained projects, maybe this LMIC version can still be useful, but in general, I would really recommend people to go with the MCCI version instead. There are still a few small functional improvements made to this repository after the MCCI version was forked, which I plan to review and submit to the MCCI repository where appropriate.

16 Likes
2

Matt, firstly, thanks for all your work and totally sympathetic with how projects can be influenced by outside sources that eventually make them rather lost in the heady whirl of changes.

However I would like to sound a note of caution about the MCCI library for use with the Arduino’s based on the ATmega328P.

I’ve been answering many questions on this forum over the last six months about DIY Arduino Pro Mini + RFM95 (or similar) builds, I recently had to migrate away from the MCCI repro as not only was I originally using an older version that appeared more stable, but the Arduino compiler update in v1.8.10 ballooned the older code size so it would no longer fit in a 328P even before adding any actual sensor code. To support answering the maker community about getting a DIY device working, I published my work in progress, along with an account of how I arrived at the setup & my findings on its reliability: https://github.com/descartes/Arduino-LMIC-example

Whilst supporting someone getting a LoRaWAN tracker going, I realised that the combo I was using didn’t actually compile on Windows for what seemed like any reasonable version. So I spent a long weekend with a huge matrix of library, IDE & OS and finally arrived at your 1.5.0+arduino-3 working just fine AND compiling using the latest Arduino IDE AND on Windows as well as macOS.

So as a supporter of the concept of the affordable, starter, maker, DIY type device, right now I’d still be recommending your library - you are saying it’s not had much love of late and you are definitely not going to do any more work on it - but it doesn’t mean it has stopped working and for me for the last few months, I’ve seen others use it to good effect so it definitely currently has value.

I’ll redo the testing matrix in a slightly more scientific & publishable way so that people can make informed choices. A good percentage of issues is hardware (soldering!) related or just understanding the pin allocations. Another chunk is documentation of the event loop when people start adding in their own code. These are all variables outside of our control, so having a reliable library helps eliminate one big area of debugging a DIY device.

It is interesting reading about the evolution of the various offerings and how they have ended up where they are. I wonder if there has been too much evolution and if perhaps looking at some of the non-LMIC work, like Ideetron’s implementation, and having a group effort from a clean sheet would stand the smaller device creator in good stead. The LoRaWAN spec has some nicely delimited sections making it very suitable for such a project.

I am acutely aware that there are those that would have us run on “Big Iron” as the cost of the smaller ARM MCUs are of the same order as the ATmega & ATtiny 1-series parts, but the complexity of the ARM peripherals, the blizzard of supporting code just to do a Blink & having the Semtech code base as pretty much the only choice isn’t educational or maker friendly.

Now you have freed up some time & brain capacity, good luck with your future endeavours.

3

Hm, interesting. I actually tested MCCI on a 328p, since that has been my primary platform for LMIC, and found that it was tight, but it did fit my existing sketch that includes a GPS parser, and Si7021 sensor reading (IIRC switching from my version to MCCI added 3k flash and 100bytes ram or so. I suspect some of that RAM can still be saved using PROGMEM, but haven’t checked yet). I did need to apply one more optimization that saves 160 bytes of unused space in the AES implementation (PR submitted), but I’m surprised to hear you couldn’t make it it work even without additional sensor-reading code.

perhaps looking at some of the non-LMIC work, like Ideetron’s implementation, and having a group effort from a clean sheet would stand the smaller device creator in good stead.

It might indeed be interesting to not try and support both tiny devices and full features in the same stack, but instead have a separate, feature-limited but tiny stack. I believe that the GitHub - LoRaWanMiniMouse/Mini-Mouse: LoRaWan Stack was originally intended to be something like that, but to be honest I haven’t really looked at that closely yet.

Another related avenue is the new “Softmodem” stack by Semtech. I’m not sure how public it is yet, and I suspect that it might be coded too generic to become small enough for e.g. the 328p, but I am hoping to see if that can be ported to Arduino too at some point (but so far, haven’t been able to really look at it, time is always a limiting factor…).

1 Like
4

Don’t be surprised - this device design is one of convenience, to provide community build options and makes for a good under-graduate training platform. I may have got something wrong as it wasn’t a very controlled scientific solution hunt!

I only started originally with an ATmega328 because I could send things to the edge of space & get them back and do commercial security / data-logging work using SD card, low power radio, WiFi or Ethernet. So that’s what I used for my initial foray in to LoRaWAN.

The current minimum I use for co-development with a client is an Arduino Nano Every + radio = 48K flash + 6K RAM or a pre-built RAK module based on STM or SAMD, with or without supplementary MCU. So there was no optimisation and not much in the way of documenting results when I ran in to the Windows compile issue in Aug and looked for a solution.

As I said above, I can set aside a bit of time to rerun the tests, this time with the results formally documented rather than a back-of-envelope testing grid.

5

That would be interesting as long as the small stack is LoRaWAN compliant. Having a non compliant stack (for instance no ADR) would be detrimental to the network and should be avoided imho.

3 Likes
6

Where is that last comment coming from?

The primary targets of MCCI LMiC are ARM chips with Arduino Ports - the SAMD21 on a Feather, and the STM32L073 in the Murata module on MCCI’s own board.

To say “having the Semtech code base as pretty much the only choice” is to completely miss the point of MCCI LMiC for the Arduino IDE.

7

Thank you!

While this did not come as a surprise, it’s good that it has now been formalized that your version of the library will not be further maintained.

@matthijskooijman Thanks for all your efforts that you have put into porting IBM’s original LMIC implementation to the Arduino framework. It has served as the defacto LMIC library for the Arduino platform since the start and for many years.
Your LMIC implementation for Arduino has made available LoRaWAN (end device) development to a broad community of non-professional-embedded-developers and is a major contribution to the The Things Network community.

While the Arduino framework initially only targeted Atmel ATmega328 (and alike) microcontrollers, the Arduino framework is now available for many microcontroller families, including SAMD, ESP32, STM32, nRF52 and more. As a result LMIC Arduino can now be used with all these microcontroller families.

While your version of the library will no longer be maintained, the community is lucky that Terry from MCCI has continued its further development and maintenance in the form of the MCCI LMIC implementation, which currently is the defacto LMIC implementation for the Arduino platform.

1 Like
8

SX1262 Support

Observations after a brief scan of mkuyper’s BasicMac fork:

  • This fork has no documentation whatsoever.
  • It’s goal is unclear.
  • It appears to NOT support the Arduino framework.

So for the Arduino community mkuyper’s fork is no alternative for Lacuna’s port of BasicMac.

Conclusions

For SX1262 support for the Arduino framework the only (open) solution currently available appears to be Lacuna’s port of BasicMAC

Above statement “there is still a lot of things in BasicMAC that would need to be further improved” will therefore also hold for Lacuna’s port.

MCCI’s fork of LMIC currently still does not support the SX1262. It is on their wishlist but a timeframe for (possible) implementation is unknown.

Details about Semtech’s new ‘SoftModem’ project are still unclear and IIRC Arduino support is not on Semtech’s list so any SoftModem support for Arduino will have to come from ‘the community’.

Except for Lacuna’s port of BasicMac there currently appears to exist no other open SX1262 support for Arduino. Your above statements give the impression that this will not be much further developed and maintained however.

Questions

@matthijskooijman Can you elaborate on the status of Lacuna’s port of BasicMac?

  • How does it compare to your ‘classic’ LMIC Arduino implementation?
  • How does it compare to MCCI LMIC?
  • What functionality of above libraries is or is not implemented in Lacuna’s port of BasicMac.
  • Is it suitable for LoRaWAN v1.0.x only? What about LoRaWAN v1.1.x (and further).
  • Is it usable for TTN V2 only or will it also be suitable to be used with TTN V3?
9

You say that like it’s a bad thing…

Realistically, people wanting to use the newer and less well understood SX1262 should probably bite the bullet and switch to a traditional embedded development paradigm.

Arduino tends to make it easy to get a project started, but then adds countless complications and plays so poorly with modern software development practices that it tends to encourage continuing to do things in bad ways, particularly with its antiquated idea of how libraries are managed, and where it comes to trying to have a codebase which supports multiple hardware options and multiple regions.

The Arduino approach is probably best left to well supported beginner boards with the older, well-understood radios on them.

If you really want to try to support something that way, it’s of course your choice - but spend some time first looking at the struggles which the existing Arduino-compatible projects have had in trying to force-fit sane software organization into the picky and primitive build flow.

10

Yes that is a bad thing from the following perspective / aspects:

  • Accessibility and learning curve for those that are not professional embedded developers.
  • Arduino framework is available for many microcontroller families. For ARM based solutions each microcontroller manufacturer seems to use/require its own specific platform, framework and tooling. For non-ARM native tooling (e.g. ESP32) the same holds.
  • Often these are commercial tools costing much money (out of reach for non-professionals).
  • Unlike the Arduino framework there appears to be no other ‘one size fits all’ solution.

I understand that Arduino has its lacks but due to broad community support and availability of libraries (whatever their quality), Arduino is what many non-professionals (i.e. not embedded developers) use.

In all these years, I have not seen any good advice like “Use the following framework and tools instead of Arduino. They support cross-platform development for different microcontroller families, have wide community support and make that you can leverage your existing knowledge”.

For professional embedded developers things will be different. The Things Network is community based however and many people (users, supporters) in that community are not professional embedded developers.

2 Likes
11

Those are precisely the people who should be sticking with the older, well-understood and well-supported radio chips.

If you look through the number of platform-driven issues in something like the MCCI LMiC issue tracker, it’s pretty hard not to wonder if by the time things actually start working Arduino has really yielded any net benefit to the approachability of the project, vs the alternative of having a good turn-key build system (eg, install this compiler, then type make REGION=EU868 TARGET=feather)

You can try to build an SX1262 system compatible with Arduino, but the two issues you are likely to face are that LMiC as it exists today heavily assumes the interface “style” of the older radios, and that the Arduino IDE makes it especially hard to support configuration diversity in the sorts of ways which modern software projects typically rely on.

Go ahead and do it if you like, but please remember you were warned it would be difficult and awkward.

12

Better SX1262 Arduino support is currently lacking and needs change in order to be able to use newer hardware. The problem is who is able to provide it and make it happen?

The Arduino IDE and the Arduino Framwework are two separate entities.
We don’t need the Arduino IDE (and its limitations) for using the Arduino Framework and there are much better alternatives for the IDE (e.g. PlatformIO).
Proper SX1262 support does not depend on limitations of the Arduino IDE.

13

You seem to be persistently ignoring the reason why that is the case.

The problem is who is able to provide it and make it happen?

I nominate bluejedi for this task.

We don’t need the Arduino IDE (and its limitations) for using the Arduino Framework and there are much better alternatives for the IDE (e.g. PlatformIO).

It’s unclear what sticking with the very odd and bad-habit-encouraging APIs of the Arduino framework gets you at that point.

By not using the Arduino IDE you’ve already lost much of the audience, those you can convince to follow you to this tool could likely use another, too. There are of course other turn-key solutions with somewhat goofy APIs which could be used - I think LoRaMAC works under mbed’s online compiler for example. And RAK has their own cheesy idea for some of their node boards.

If you want to support SX1262 on Arduino, no one is stopping you.

But expecting someone else to do so, would require that there first be someone who thinks its enough of a good idea, to pour their time/money into the task.

Maybe there’s a middle way such that the LoRaWAN solution isn’t really “Arduino” but can be used from an Arduino sketch. If platformio is as flexible as you claim, that should be possible. But investing effort in an Arduino-only codebase (which is exactly what actual Arduino-LMiCs are, requiring hours of manual and script editing to turn back into ordinary code) seems unwise.

14

Maybe you seem to be persistently ignoring that change is possible.
I have not yet seen any proof that it is technically impossible to support SX1262 for the Arduino framework. It may not be the most evident from perspective of current LMIC implementations but:

  • If you would have said some 8+ years ago that the Arduino framework was going to run on multiple different microprocessor families, many might have laughed and not believed you. But nowadays this is reality.
  • SX1262 support is still on ‘MCCI’s wishlist’. They haven’t said (not yet at least) that this is never going to happen.
  • Heltec appears to have ported LoRaMac-node to Arduino for ESP32. Unfortunately their implementation is partially closed source. For that reason I never had a good look at it. LoRaMac-node supports SX1262 so this sounds like an opportunity.
    Heltec has also ported LoRaMac-node to Arduino for their CubeCell products which are ARM + SX1262 based. Unfortunately this again is (partially) closed source. But it shows that SX1262 support for the Arduino framework is technically possible.

Thanks for your nomination. :slightly_smiling_face:
I wish that I had the knowledge and experience to do that.

That was already explained above.

Let’s stop discussing about whether combining SX1262 and Arduino framework is a good or bad thing. That’s off-topic here.

15

No one ever said it was impossible, just pointed out why it was difficult and likely not worthwhile.

Everybody agreed that Anybody could do it and allowed that maybe someday Somebody would, but so far Nobody has seriously tried.

Actually it already did eight years ago. But the way it does is still problematic, in that the third party ports are forever getting broken by changes in how things are expected to work.

They’ve made it pretty clear they aren’t going to spend time or money on it anytime soon.

16

@bluejedi, I’ll respond to your questions about BasicMAC below, to the best of my knowledge.

BasicMac is a continued development of LMIC, so the code structure is largely the same, but they added some features. I don’t have a complete list, but from looking at the code I’ve seen hints of LoRaWAN 1.1 support, multi-region (i.e. runtime rather than compiletime region selection), some improvements in the scheduler that could make sleeping easier to implement, and SX1262 support (that I actually used and tested).

I can’t really answer that, since I haven’t seen all of BasicMAC nor MCCI LMIC yet. Both have been independently developed, so probably have their own features and advantages. Both have implemented multiregion mode, which requires signifcant changes and I doubt they did this in exactly the same way, so that probably makes it hard to move code between both.

Furthermore, MCCI is primarily for Arduino (though I’ve heard Terry say people were using it in other environments too), while BasicMAC is originally not for Arduino (examples are for STM32 with a Makefile-based build system), the Lacuna fork of BasicMAC adds Arduino support using a generator script (that generates an Arduino library from the original layout, allowing the original Makefile-based build system to be preserved).

The entire codebase of BasicMAC is included in the Lacuna port, but I cut some corners in implementing the HAL for Arduino (see the TODOs in hal.cpp in target/arduino somewhere). From memory, some of the long-term timekeeping is still flaky, saving of nonces in EEPROM (needed for LW 1.1) is not implemented and probably some more things.

The code has evidence of 1.1 support, but I’m not sure if it is complete.

I assume TTN v3 will just implement the LoRaWAN protocol? Nothing TTN specific in any of this LMIC or BasicMAC code?

For another project, I’m planning to use the SX1262 with Arduino, so I’ll probably make sure that the Lacuna version stays working. I can’t promise any real ongoing development, though.

Switching to replying to some of @cslorabox’s post.

This is a real problem indeed, at least if you’re talking about making it possible to do compile-time configuration of code and libraries. IMHO, this should really be implemented in Arduino sometime, but indeed, the low entry-level that Arduino aims for makes it more tricky to do this than a simple Makefile-or-whatever-based system where you can just add some -D options to the commandline.

However, I do think you’re underestimating the convenience of Arduino for more novice users. Also, it’s fine if you don’t want to use Arduino, but then maybe you shouldn’t be telling Arduino users what hardware they should or should not want to use. I’m probably cutting a few corners here, but no time right now (likely not later either) to really respond to all of your points in detail.

This is one of the things I regret happened with Arduino-LMIC, that it became Arduino-only and no longer usable outside of Arduino (as also indicated in the initial post). For the Lacuna BasicMAC port, I fixed this by keeping the original build system intact, so this point seems not to apply there at least.

1 Like
17

The script generating an Arduino library is an interesting idea. However, it also further complicates one of the problems of the Arduino approach, which is that it’s very awkward to make fixes in library code and then get them adopted into the build.

When I speak in terms of recommending that beginners stick to well understood radios and platforms, its precisely because debugging and fixing things in the library code is so painful. Arduino even makes it challenging to do something so simply as get a debug printout in C-language library code, since all of the usual I/O routines are C++ methods requiring that output be proxied back into the sketch or a wrapper.

As everyone who has actually gotten elbow deep in the code (or simply reviewed the issue trackers) knows, this code has had many bugs and invalid assumptions tracing back to its IBM origins, some addressed, some yet to be found. MCCI’s efforts to pass LoRaWAN cerification with regard to MAC command decoding and the like have uncovered some fairly challenging deep-rooted design issues in things like paths that are theoretically possible but rarely exercised, while timing quirks of particular platforms have highlighted many others. I don’t even blame the authors - there’s a lot about the LoRaWAN specification that makes one wonder if the people who wrote it gave sufficient thought to what it would be like to practically implement everything it calls for.

If someone want to use Arduino on a LoRaWAN node, they should probably use it in the context of a well proven hardware and software starting point that isn’t going to require debugging and making fixes to the starting point code. And right now, that means a traditional radio on a couple of hardware platforms that see active maintenance and regression testing.

18

There’s also an option to generate symlinks rather than copies, which makes this nearly painless, though.

Yeah, that’s what I meant with plenty of room for improvement (both in LMIC and BasicMAC). I guess this is partly just bad code, and partly a side effect of the code being small (I’m not saying it couldn’t be better and still small, but it is hard to make code clean and small). None of this is really related to Arduino, though.

Phrased like this, I think this is indeed valuable advice. I guess this is also not specific to the Arduino environment, though outside of the Arduino environment there might be more choice of stacks that support the SX1262 (though I can’t comment on their quality).

19

The core of my point is that Arduino is only advantageous if the behind-the-scenes library code entirely works as is and doesn’t need touching.

If it doesn’t, then not only is the target market user of an Arduino approach in over their head, they then face a greater challenge in trying to debug and fix the library code, than they would in a conventional MCU project where such sketch-vs-library distinctions are not so as large.

It becomes a specific concern with the SX1262 in that the latter is newer, so code that supports it is newer, too.

1 Like
20

Many thanks Matthijs; you were one of the pioneers to bring Lorawan to the early adopters (still soldering together chips to get functional nodes).

To be honest I’ve always used your library up until very recently when I did a review of alternatives. It’s great that there are has been done some great development but you were one of the people making this possible in the early days. A big thank you and bravo!

3 Likes