Hy Jac, thanks for your support! We started investigation becaus of high packet loss rates on the kerlink, significantly higher than on the TTN gateways.
Here is a complete cycle (one stop to the next), I could not see any good reason for the stop. We can see that the process is started again by the watchdog, but inbetween packets are lost.
I tried to install previous versions of the SPF/Keros and found the same issue in spf_3.1.0-klk18_4.1.3-klk11_klk_wifc with Keros 3.4.4.
Now the gateway is running spf_3.1.0-klk11_4.1.3-klk3_klk_wifc from May 2017 and liveburner_3.1.14_klk-wifc, here we did not recognize any stops til now.
Here is part of a protocol from the latest version for the iFemtocell (referring to kerlink) spf_3.1.0-klk18_4.1.3-klk12_klk_wifc:
Sep 6 07:33:43 klk-wifc-040187 local1.notice spf: INFO: Exiting packet forwarder program
Sep 6 07:33:52 klk-wifc-040187 local1.notice spf: I/ Programming FPGA with spectral scan firmware
Sep 6 07:33:52 klk-wifc-040187 local1.notice spf: *** Beacon Packet Forwarder for Lora Gateway ***
Sep 6 07:33:52 klk-wifc-040187 local1.notice spf: Version: 3.1.0-klk18
Sep 6 07:33:52 klk-wifc-040187 local1.notice spf: *** Lora concentrator HAL library version info ***
Sep 6 07:33:52 klk-wifc-040187 local1.notice spf: Version: 4.1.3-klk12;
Sep 6 07:33:52 klk-wifc-040187 local1.notice spf: ***
Sep 6 07:33:52 klk-wifc-040187 local1.notice spf: INFO: Little endian host
Sep 6 07:33:52 klk-wifc-040187 local1.notice spf: I/ Using default Gateway EUID: 7276FF0039040187
Sep 6 07:33:52 klk-wifc-040187 local1.notice spf: INFO: found global configuration file global_conf.json, parsing it
Sep 6 07:33:52 klk-wifc-040187 local1.notice spf: INFO: found local configuration file local_conf.json, parsing it
Sep 6 07:33:52 klk-wifc-040187 local1.notice spf: INFO: redefined parameters will overwrite global parameters
Sep 6 07:33:52 klk-wifc-040187 local1.notice spf: INFO: Using Gateway EUID: 7276FF0039040187
Sep 6 07:33:52 klk-wifc-040187 local1.notice spf: INFO: FPGA supported features: [TX filter] [Spectral Scan]
Sep 6 07:33:55 klk-wifc-040187 local1.notice spf: INFO: [main] concentrator started, packet can now be received
Sep 6 07:33:55 klk-wifc-040187 local1.notice spf: INFO: Disabling GPS mode for concentrator's counter...
Sep 6 07:33:55 klk-wifc-040187 local1.notice spf: INFO: host/sx1301 time offset=(1567755232s:745020µs) - drift=1975391804µs
Sep 6 07:33:55 klk-wifc-040187 local1.notice spf: INFO: Enabling GPS mode for concentrator's counter.
Sep 6 07:33:56 klk-wifc-040187 local1.notice spf: INFO: Received pkt from mote: 26012558 (fcnt=2385)
Sep 6 07:33:56 klk-wifc-040187 local1.notice spf: JSON up: {"rxpk":[{"tmst":3313227,"time":"2019-09-06T07:33:56Z","chan":2,"rfch":0,"freq":867.500000,"stat":1,"modu":"LORA","datr":"SF7BW125","codr":"4/5","lsnr":7.5,"rssi":-71,"size":17,"data":"QFglASYAUQkCZvemmQncMVI="}]}
Sep 6 07:33:56 klk-wifc-040187 local1.notice spf: INFO: [up] PUSH_ACK received in 41 ms
Sep 6 07:34:09 klk-wifc-040187 local1.notice spf: INFO: Received pkt from mote: 26012DE2 (fcnt=141)
Sep 6 07:34:09 klk-wifc-040187 local1.notice spf: JSON up: {"rxpk":[{"tmst":17196428,"time":"2019-09-06T07:34:09Z","chan":6,"rfch":1,"freq":868.300000,"stat":1,"modu":"LORA","datr":"SF7BW125","codr":"4/5","lsnr":9.5,"rssi":-63,"size":17,"data":"QOItASaAjQABwICmkL889sQ="}]}
Sep 6 07:34:09 klk-wifc-040187 local1.notice spf: INFO: [up] PUSH_ACK received in 44 ms
Sep 6 07:34:16 klk-wifc-040187 local1.notice spf: INFO: Received pkt from mote: 26012558 (fcnt=2386)
Sep 6 07:34:16 klk-wifc-040187 local1.notice spf: JSON up: {"rxpk":[{"tmst":23704443,"time":"2019-09-06T07:34:16Z","chan":3,"rfch":0,"freq":867.700000,"stat":1,"modu":"LORA","datr":"SF7BW125","codr":"4/5","lsnr":11.0,"rssi":-72,"size":17,"data":"QFglASYAUgkCwYeaEq1EtlE="}]}
Sep 6 07:34:16 klk-wifc-040187 local1.notice spf: INFO: [up] PUSH_ACK received in 42 ms
Sep 6 07:34:25 klk-wifc-040187 local1.notice spf: INFO: [down] the last 3 PULL_DATA were not ACKed, exiting application
Sep 6 07:34:25 klk-wifc-040187 local1.notice spf: INFO: End of downstream thread
Sep 6 07:34:25 klk-wifc-040187 local1.notice spf: INFO: End of upstream thread
Sep 6 07:34:25 klk-wifc-040187 local1.notice spf: ##### 2019-09-06 07:34:25 GMT #####
Sep 6 07:34:25 klk-wifc-040187 local1.notice spf: ### [UPSTREAM] ###
Sep 6 07:34:25 klk-wifc-040187 local1.notice spf: # RF packets received by concentrator: 3
Sep 6 07:34:25 klk-wifc-040187 local1.notice spf: # CRC_OK: 100.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%
Sep 6 07:34:25 klk-wifc-040187 local1.notice spf: # RF packets forwarded: 3 (51 bytes)
Sep 6 07:34:25 klk-wifc-040187 local1.notice spf: # PUSH_DATA datagrams sent: 3 (678 bytes)
Sep 6 07:34:25 klk-wifc-040187 local1.notice spf: # PUSH_DATA acknowledged: 100.00%
Sep 6 07:34:25 klk-wifc-040187 local1.notice spf: ### [DOWNSTREAM] ###
Sep 6 07:34:25 klk-wifc-040187 local1.notice spf: # PULL_DATA sent: 3 (0.00% acknowledged, ping 0.00 ms)
Sep 6 07:34:25 klk-wifc-040187 local1.notice spf: # PULL_RESP(onse) datagrams received: 0 (0 bytes)
Sep 6 07:34:25 klk-wifc-040187 local1.notice spf: # RF packets sent to concentrator: 0 (0 bytes)
Sep 6 07:34:25 klk-wifc-040187 local1.notice spf: # TX errors: 0
Sep 6 07:34:25 klk-wifc-040187 local1.notice spf: # BEACON queued: 0
Sep 6 07:34:25 klk-wifc-040187 local1.notice spf: # BEACON sent so far: 0
Sep 6 07:34:25 klk-wifc-040187 local1.notice spf: # BEACON rejected: 0
Sep 6 07:34:25 klk-wifc-040187 local1.notice spf: ### [JIT] ###
Sep 6 07:34:25 klk-wifc-040187 local1.notice spf: /home/drd/jenkins/workspace/spf_release/lora_pkt_fwd/src/jitqueue.c:448:jit_print_queue(): INFO: [jit] queue is empty
Sep 6 07:34:25 klk-wifc-040187 local1.notice spf: ### [GPS] ###
Sep 6 07:34:25 klk-wifc-040187 local1.notice spf: # GPS sync is disabled
Sep 6 07:34:25 klk-wifc-040187 local1.notice spf: ##### END #####
Sep 6 07:34:25 klk-wifc-040187 local1.notice spf: INFO: concentrator stopped successfully
Sep 6 07:34:25 klk-wifc-040187 local1.notice spf: INFO: Exiting packet forwarder program