So I just had a strange situation where the packet forwarder stopped sending/processing packets!
MT firmware 1.4.1 with the latest released packet forwarder.
The packet forwarder was still running according to the linux process list but nothing was happening and no recent activity in the packet log. TTN console showed an ever increasing “last seen” value for the gateway. Internet connectivity checked and is fine.
Following a stop/start of the packet forwarder everything resumed as normal.
Could you stop the packet forwarder using:
/etc/init.d/ttn-pkt-forwarder stop
and then from your terminal run:
/opt/lora/mp_pkt_fwd -c /var/config/lora
This will start the process without output redirection making it easier to view the last lines of output without a buffer. Drawback is you have to keep the terminal session open or the forwarder will stop working.
It would help if you could post the last few lines of output you observe when the issue occurs.
mmm not sure what you asking, the messages also stops when the gateway stops. looking at the terminal the packet forwarding log also stops. can send it restart command and everything works again for another few minutes. also not a set amount of time. last session lasted 10 min before stopping.
Since the issue is unpredictable (at least from my experience so far) maybe one option is to run the process in a screen session. I will see if I can get screen installed on the Conduit.
I have just had the problem again after over 2 days of no issues. I will add the logging as you suggest and see if I can get some more info. My gut feeling at the moment is it is somehow related to a node connecting. I say this because during my testing where I had a single node coming and going off the gateway I saw the problem more frequently. No I have been away for the weekend with no nodes connected and the gateway was “seen” moments ago. The minute I connected my node to try some new code I worked on over the weekend I noticed it was not able to join. Sure enough the “seen” time started increasing past a minute and I had to restart the packet forwarder to get things working again.
So I would see to try and reproduce the problem one should have only one node sending to the gateway and then maybe make it come and go with varying intervals of time… just a thought
one should have only one node connected to the gateway
Just to be sure: nodes do not “connect to a gateway”. They just transmit and hope one or more gateways receive their packet and will forward that to TTN.
Correct, my bad. I was thinking about OTAA. Having said this I cant say at the moment if it might be OTAA related or not but I do use OTAA for the node I am testing.
Well… Same goes for OTAA: a node transmits a Join Request (uplink), and hopefully one or more gateways forward that to TTN. If accepted, after 5 or 6 seconds TTN will command one of the gateways to transmit a Join Accept with the new secrets (downlink). Hopefully one or more nodes receive the gateway’s transmission, but only the one that transmitted the original Join Request will be able to decipher it. Again, no “connection”.
If ABP works, you could try if the gateway fails after being told to send a downlink. (Or an ACK, or some ADR command.) Maybe it already fails after receiving the downlink command. Maybe while processing it. Maybe it does not return to listening mode after transmitting.
In TTN Console you can look at the gateway’s “traffic” page to see what TTN has received and what it has commanded the gateway to transmit.
I have the exact same problem. Since I have installed the new packet forwarder, the gateways stop working within a couple of minutes. I do not think the problem is directly related to OTAA, because no data ist received from activated nodes either. However, it may have something to do with downlinks, because the last message showing up in traffic was indeed a downlink.
Here is the dmesg output:
vizzini 2-3:1.2: ttyXRUSB1: XR21v14x usb uart device
usbcore: registered new interface driver vizzini
vizzini: Exar USB UART Driver for XR21V141x
usb 1-2.2: FTDI USB Serial Device converter now attached to ttyUSB0
[INFO] mts-io:mts_io_init:829: init: v1.1.1
[INFO] mts-io:mts_id_eeprom_load:754: detected board MTCDT-0.0
[INFO] mts-io:mts_id_eeprom_load:757: sizeof: 512
[INFO] mts-io:mts_id_eeprom_load:758: vendor-id: Multi-Tech Systems
[INFO] mts-io:mts_id_eeprom_load:759: product-id: MTCDT-H5-210L
[INFO] mts-io:mts_id_eeprom_load:760: device-id: 18751642
[INFO] mts-io:mts_id_eeprom_load:761: hw-version: MTCDT-0.0
[INFO] mts-io:mts_id_eeprom_load:768: mac-addr: 00:08:00:4A:0A:FF
[INFO] mts-io:mts_id_eeprom_load:770: imei: 356136071886214
[INFO] mts-io:mts_id_eeprom_load:771: capa-gps: no
[INFO] mts-io:mts_id_eeprom_load:772: capa-din: no
[INFO] mts-io:mts_id_eeprom_load:773: capa-dout: no
[INFO] mts-io:mts_id_eeprom_load:774: capa-adc: no
[INFO] mts-io:mts_id_eeprom_load:775: capa-wifi: no
[INFO] mts-io:mts_id_eeprom_load:776: capa-bluetooth: no
[INFO] mts-io:mts_id_eeprom_load:800: uuid: 00000000000000000000000000000000
[INFO] mts-io:load_port:653: no accessory card inserted in port 1
[INFO] mts-io:load_port:688: accessory card 2 vendor-id: Multi-Tech Systems
[INFO] mts-io:load_port:689: accessory card 2 product-id: MTAC-LORA-868
[INFO] mts-io:load_port:690: accessory card 2 device-id: 18948415
[INFO] mts-io:load_port:691: accessory card 2 hw-version: MTAC-LORA-1.0
[INFO] mts-io:load_port:712: accessory card 2 eui: 00:80:00:00:A0:00:01:CC
[INFO] mts-io:lora_setup:42: loading LORA accessory card in port 2
IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
macb f802c000.ethernet eth0: link up (100/Full)
IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
g_serial gadget: Gadget Serial v2.4
g_serial gadget: g_serial ready
ftdi_sio ttyUSB0: FTDI USB Serial Device converter now disconnected from ttyUSB0
ftdi_sio 1-2.2:1.0: device disconnected
According to netstat -a, the connection to the router/bridge is still open:
tcp 0 0 xxxxxxxxxxxxxxxx:51353 52.169.76.203:1883 ESTABLISHED
A new package (r3) is available on github. Requires manual download and install with
opkg install < filename >
Once installed edit local_conf.json and add
"flush_line": true,
This will push the log to ‘disk’ after every line of output. Not recommended for busy gateway as it will have performance impact, but better for the curious that want to see the logging while things are happening in stead of minutes later.
