Building the TTN Gateway firmware from the GitHub code

arjanvanb · February 19, 2018, 8:48pm

As a start: see the instructions at https://github.com/TheThingsProducts/gateway/tree/master/firmware

To enable debug logging, use SYS_ERROR_DEBUG in system_init.c:

SYS_DEBUG_INIT debugInit =
{
    .moduleInit = {0},
    .errorLevel = SYS_ERROR_DEBUG
};

To increase the logging baud rate for the serial cable, which might lower the chances for weird/incomplete output when using a lot of logging, use the following in system_config.h:
```
#define DRV_USART_BAUD_RATE_IDX0 921600
```
…and use the same baud rate in your serial monitor software.
Consider using "%02x" rather than "%#x" to print hexadecimal values with leading zeroes.

cyberjunky · February 1, 2018, 9:25pm

These are the steps I took, all at your own risk:

Compile TTN Gateway Firmware on Ubuntu

Install 64 Bit dependencies

Resource: http://microchipdeveloper.com/install:mplabx-lin64

sudo apt-get install libc6:i386 libx11-6:i386 libxext6:i386 \
libstdc++6:i386 libexpat1:i386

Download and install MPLABX

Resource: http://microchipdeveloper.com/mplabx:installation

tar -xvf MPLABX-v4.05-linux-installer.tar
chmod u+x MPLABX-v4.05-linux-installer.sh
sudo ./MPLABX-v4.05-linux-installer.sh

Next, Accept Agreement, Next
/opt/microchip/mplabx/v4.05
Next, Finish

To prevent error java not found:

sudo mkdir /opt/microchip/mplab_ide
sudo ln -s /opt/microchip/mplabx/v4.05/sys /opt/microchip/mplab_ide/sys

Install XS32 compiler

Download
Resource: http://www.microchip.com/mplabxc32linux

NOTE: we need older version v1.42 to prevent errors like these:
error: '_CNCONB_SIDL_MASK' undeclared (first use in this function..)
You get them when you try to compile with 1.43 or higher.

https://www.webeaver.com/1199815/08f0d806ca3a4d6a7dc32c4344237875/download-xc32-v1-42-full-install-linux-installer-run
If anyone find a safer download source, please let me know.

chmod +x ./xc32-v1.42-full-install-linux-installer.run 
sudo ./xc32-v1.42-full-install-linux-installer.run

Next, Accept, Next, Next
/opt/microchip/xc32/v1.42
Next, Add path, Next, Next, Get Coffee, Next, Finish

sudo apt install build-essential cppcheck

mkdir ~/ttn-gw-dev && cd ~/ttn-gw-dev

git clone --recursive https://github.com/TheThingsProducts/gateway.git
cd gateway/firmware

cppcheck --enable=all --inconclusive --force --xml --xml-version=2 src 2> cppcheck.xml

./generate_version_header.sh
./fix_mplab.sh

vi compile.sh and correct prjMakefilesGenerator path:

elif [ "$(uname -s)" = "Linux" ]; then
  prjMakefilesGenerator='/opt/microchip/mplabx/v4.05/mplab_ide/bin/prjMakefilesGenerator.sh'

./compile.sh

[Get Coffee here]

./generate_hex_with_checksum.sh

ls -l firmware.hex 
-rw-r--r-- 1 ron ron 2767956 feb  1 21:22 firmware.hex

cat checksums 
337950176cab81e5715b16c86af424cca57d0de088e5a24dcec7fef5c24e432b  firmware.hex

Write them to SD Card (I used a 2GB FAT32 formatted one)

mkdir /media/<your path>/update

cp firmware.hex checksums /media/<your path>/update

ls -l /media/<your path>/update/
total 2708
-rw-r--r-- 1 ron ron      79 feb  1 21:52 checksums
-rw-r--r-- 1 ron ron 2767956 feb  1 21:52 firmware.hex

Insert in gateway and boot
Notice the intermittent outside and center leds blinking until finished.

Check you info page at http://things-gateway.local/info

rmemory · February 19, 2018, 3:05am

In the following file, I am wondering if someone could kindly fill in a few details about this statement from the firmware readme:

After building the project using the MPLabX GUI

My goal is to build my own firmware from the “develop” branch, with logging enabled as mentioned by @Arjan on this issue. Essentially, I want to see if I get the same behavior which I will be happy to report.

When I cd to the firmware directory, I see errors like this when I run the compile.sh command:

[Project Loading Error]  Error: Project "TTN_Gateway" refers to file "certs_test.h" which does not exist in the disk. The project failed to load.
[Project Loading Error]  Error: Project "TTN_Gateway" refers to file "certs_test.h" which does not exist in the disk. The project failed to load.

I suspect I don’t have all of the required tools installed. After successfully installing the 32 bit compatibility libs on my Linux host, and successfully running “MPLABX-v3.45-linux-installer.sh”, I believe I have the Microchip IDE installed here: /opt/microchip/mplabx/v3.45/mplab_ide

But, I suspect I am still missing some of their tools. I looked around on their web site, but didn’t immediately find what I was looking for. Any hints are most welcome.

arjanvanb · February 19, 2018, 6:00am

On a Mac, I get the same errors in the IDE, but the project builds just fine.

I could not even get the command line build to work when the IDE is not open (running into some Java headless error). Also, I’ve ignored the “After building the project using the MPLabX GUI” part (though I have run the script once or twice). Instead, after building in the IDE, I just run generate_hex_with_checksum.sh now.

rmemory · February 19, 2018, 5:14pm

I launched the MPLAB X IDE, and opened the TTN_Gateway.X project in the IDE. I hit compile errors of not being able to find constants such as _CNCONJ_SIDL_LENGTH, _CNCONB_SIDL_POSITION, and others.

Just for fun, I tried it from the command line again using compile.sh but experienced the same errors.

I grep’d around in the tree for those constants, but came up empty. Seems possibly they might be somehow generated elsewhere. But that is just a guess on my part. The error on the command line ends up looking like the following I poked around a bit in the makefiles, but haven’t yet determined the source of the error.

 nbproject/Makefile-TTN_Gateway_v1.mk:1714: recipe for target 'build/TTN_Gateway_v1/production/_ext/1263322388/MQTTHarmony.o' failed
make[1]: *** [build/TTN_Gateway_v1/production/_ext/1263322388/MQTTHarmony.o] Error 1
make[1]: Leaving directory '/home/rmemory/ttn-gateway/gateway/firmware/TTN_Gateway.X'
nbproject/Makefile-TTN_Gateway_v1.mk:97: recipe for target '.build-conf' failed
make: *** [.build-conf] Error 2
make: Leaving directory '/home/rmemory/ttn-gateway/gateway/firmware/TTN_Gateway.X'

arjanvanb · February 19, 2018, 5:20pm

I’ve opened firmware/TTN_Gateway.X (from the February 14th develop branch) and am using menu Run, Build (or the hammer button).

But I’m on a Mac. As you’re on Linux, did you see @cyberjunky’s post?

cyberjunky · February 19, 2018, 6:02pm

I believe you get those errors if you installed a xc32 version higher than the recommended v1.42, so you have to downgrade that one.

rmemory · February 19, 2018, 6:12pm

Thank you very much for the pointer to the post from @cyberjunky . His instructions worked perfectly.

And thanks very much @cyberjunky!!

weradis · February 19, 2018, 7:30pm

Do you have a PRO license for the compiler or did you go for the 60 day trial version.

rmemory · February 19, 2018, 7:38pm

FWIW, I used the 60 day eval

arjanvanb · February 19, 2018, 7:47pm

Oh, I thought I just enabled some free version. But apparently it might be a trial then? Bummer, then there are not many days left to fix this thing

(Ah, that might be the IDE; the compiler might be different?)

rmemory · February 19, 2018, 9:16pm

I wasn’t entirely clear on the licensing either (IDE vs compiler).

Edit: I have been poking around a little bit on the command line to see if I can find out how the command line xc32 utils are licensed and how much time remains on my current, presumably free, license. I haven’t yet found the magic cli flag to provide that information.

Apparently there is a way to build the tools at the command line as well? Although, I am not sure that changes anything.

In any case, FWIW, it does matter that you use the V1.42 version to build the TTN gateway firmware. The newer versions such as 2.05 will not build the firmware source correctly.

pe1mew · March 21, 2018, 4:46pm

Has anyone used Windows to compile and program the gateway?

ScottM · June 14, 2018, 1:54am

I have been lurking here for the last week and thought I would share my experience building/debugging the gateway firmware…

Tools:

OSX (10.13.5)
MPLAB X (4.20) need this for PICKIT 4
XC32 (2.10 and 1.42)
PICKIT 4 (MPLAB IPE programming works but MPLAB X IDE debug is still… “Super Crunchy”.

Code:

TTNGW development branch (1.0.4 and the bootloader.hex it contains.)
Harmony (1.08.01 & 2.05.01 but I am not regenerating code from either at the moment…)
Configurator in 1.08.01 gives “[Error] Value (MICREL_KSZ8061RND) not valid…”
Configurator in 2.05.01 allows replacing MICREL_KSZ8061 with KSZ8061 but merging generated code looked… painful…

Building/programming functional “production” image with current tools:

A simple workaround for building with a recent compiler is to add back the defs referenced by the old Harmony framework which were removed from the compiler’s cpu include file because the SIDL bit is not present on the pic32mz2048efm144 device.

I extracted the necessary defs from an old/v1.42 XC32 install:
grep _CNCON._SIDL_ /Applications/microchip/xc32/v1.42/pic32-libs/include/proc/p32mz2048efm144.h
Then jammed them into the top of the offending old Harmony generated file:
gateway/firmware/vendor/harmony/framework/peripheral/ports/processor/ports_p32mz2048efm144.h

I am able to build a functional production image, combine it with the supplied bootloader, and program a functional gateway.

Building/debugging crunchy “debug” image with current tools:

I can also build a (partially) functional debug image which I can load with the MPLAB X IDE debugger (which also loads the bootloader) and can preform quite a bit of debug. I have even successfully debugged a flaw in the generated debug image which resulted in the wireless ssid string not being properly linked and causing “Things-Gateway-############” to be written on top of unrelated (and fairly important) data.

A few questions:

Where is the project that was used to generate the bootloader.hex? (I have seen at least one reference to a v2.x bootloader…)
Has anyone had more success using the MPLAB X IDE debugger with a debug build of the firmware? (I get same problems with my debug image with both new and old XC32.)
Has anyone had more success regenerating Harmony code from existing harmony config file?
Has anyone tried migrating to the current Harmony? (Any guess on probability this could address reported WiFi stability issues?)
Is everyone (especially the developers) aware of the pic32mz2048efm144 rev.A3 device errata? (No offense intended… but I find the results below a bit disconcerting:)

iMac5k:TTNLoRaGw scott$ ls
TTNGW_Beta TTNGW_Develop TTNGW_Master TTNGW_Stable gateway hex
iMac5k:TTNLoRaGw scott$ grep -iR errata *
iMac5k:TTNLoRaGw scott$

Yes, my gateway appears to be functioning fine…
Yes, I am just a sick individual…

ScottM · June 15, 2018, 4:32pm

WORKAROUND for functional MPLAB X IDE debug build of gateway firmware:

Turning on gcc option"place data into it’s own section" appears to prevent whatever was allowing ssidString overlap with other data:

Production

iMac5k:firmware scott$ /Applications/microchip/xc32/v2.10/bin/xc32-objdump -t TTN_Gateway.X/dist/TTN_Gateway_v1/production/TTN_Gateway.X.production.elf | grep ssidString
8007ef8c l O .bss 00000021 ssidString.81547
8007ef68 l O .bss 00000021 ssidString.81555

Debug (bad link with overlapping data.)

iMac5k:firmware scott$ /Applications/microchip/xc32/v2.10/bin/xc32-objdump -t TTN_Gateway.X/dist/TTN_Gateway_v1/debug/TTN_Gateway.X.debug.elf | grep ssidString
80000424 l O .bss 00000021 ssidString.81547
80000400 l O .bss 00000021 ssidString.81555

iMac5k:firmware scott$ /Applications/microchip/xc32/v2.10/bin/xc32-objdump -t TTN_Gateway.X/dist/TTN_Gateway_v1/debug/TTN_Gateway.X.debug.elf | grep 80000400
80000400 l d .bss 00000000 .bss
80000400 l O .bss 00000021 ssidString.81555
80000400 g O .sbss 00000002 uartrx_len

Debug w/ "data in own sections"
iMac5k:firmware scott$ /Applications/microchip/xc32/v2.10/bin/xc32-objdump -t TTN_Gateway.X/dist/TTN_Gateway_v1/debug/TTN_Gateway.X.debug.elf | grep ssidString
8007ee6c l d .bss.ssidString.81555.ssidString 00000000 .bss.ssidString.81555.ssidString
8007ee90 l d .bss.ssidString.81547.ssidString 00000000 .bss.ssidString.81547.ssidString
8007ee90 l O .bss.ssidString.81547.ssidString 00000021 ssidString.81547
8007ee6c l O .bss.ssidString.81555.ssidString 00000021 ssidString.81555
iMac5k:firmware scott$