All my gateways disconnected suddenly and none are re-connecting

digidhamu · August 29, 2024, 2:54pm

Issue Summary:

I’m facing a critical issue where all my LoRaWAN gateways have suddenly disconnected, and I haven’t been able to reconnect any of them. Despite trying multiple troubleshooting steps, including setting up a new instance of the LNS (LoRaWAN Network Server) on both DigitalOcean and AWS, none of the gateways are establishing a connection. This issue is persistent across all gateways, and I’m at a loss for what might be causing it.

Environment Details:

Deployment Method: Docker Compose
Operating System: Ubuntu 24.04
LNS (Open Source) Versions Tried: 3.29.x, 3.30.x, 3.31.1

Logs and Troubleshooting Attempts:

Unfortunately, I couldn’t find any relevant logs that might indicate the root cause of the problem. Neither the gateways nor the LNS instances are providing error messages or any logs that could point me in the right direction.

Hardware Involved:

Gateway Model: RAK7268 WisGate Edge Lite 2
Firmware Version: 1.3.9_RAK b186

Steps Taken So Far:

Rebooted Gateways: Rebooted all the gateways multiple times to see if they would reconnect.
LNS Re-deployment: Set up new instances of the LNS on DigitalOcean and AWS, using the same Docker Compose deployment method. I tried with different versions (3.29.x, 3.30.x, 3.31.1) but faced the same issue with each.
Network Checks: Verified that there are no network issues or firewall rules that could be blocking the connection between the gateways and the LNS.
Firmware Verification: Ensured that all gateways are running the correct firmware version (1.3.9_RAK b186) as recommended by RAKwireless.

Request for Help:

Has anyone encountered a similar issue or can provide insight into potential causes? Could it be a compatibility issue with the LNS versions or something related to the Docker Compose setup? Any suggestions on further troubleshooting steps or logs I should be looking into would be greatly appreciated.

descartes · August 29, 2024, 3:58pm

How many is all? 2, 10, 100??

What do you mean by critical error - testing has come to a halt or 10,000 puppies run the risk of starvation?

I’ve not seen any spontaneous bit-rot in TTS OS, something must have changed else where - like an auto-update of the gateways for instance - or a change in Basic Station config if that’s what you are running (are you running that, you don’t say).

As that’s a pretty popular gateway and no one else has reported issues with TTN, there must be something in your mixture that is causing this. So you might want to try putting a gateway on to TTN to see what happens.

digidhamu · August 30, 2024, 12:44am

Thanks, @descartes, for your reply.

The situation is critical because we have numerous sensors deployed across farms to monitor environmental parameters, and these readings directly influence feeding decisions based on the data collected.

I’ve ensured that automatic updates for the gateways are turned off, and there hasn’t been any physical or remote access that could have altered their configuration.

To troubleshoot, I connected the gateways to TTN (Enterprise with a Discovery Subscription), and they are working perfectly, with data flowing as expected.

Any further advice or insights would be greatly appreciated.

descartes · August 30, 2024, 8:46am

If you startup TTS OS from the command line you can see a blizzard of info - I’ve never looked for the gateway activity but that may be worth grepping.

What is the backhaul - the internet connection that the gateways use - could this have changed?

Can you run TTS OS in your office with one of these RAK gateways to make sure there are no firewall or other port issues. If there are, you can then debug that more easily.

Jeff-UK · August 30, 2024, 9:05am

Particularly look at what PF the GW is running, and check what your backhaul comms provider might have started filtering wrt Port numbers (a know historic issues with some service providers) though fact you can connect GW ok to TTN/TTS(Sandbox) suggests this may not be an issue…, check also any ingress Port filtering/set-up to your cloudy services for private LNS instance.

That plus on same network segment if internal network large or segmented! (Tip - I try at home network if I find a field or corporate deployment has a problem!)

digidhamu · August 30, 2024, 2:06pm

One thing I’ve noticed in the TTN OS Docker console’s backend logs is that there’s no inbound communication from any of the gateways to the server. This is quite unusual. The firewall configuration, which was working fine earlier, hasn’t been changed. For troubleshooting purposes, I’ve even allowed all connections to rule out any issues, but unfortunately, that didn’t resolve the problem.

Backhaul: WiFi (Office Network) and WiFi (Home Network) and both has been configured for this troubleshooting

Firewall Config

Jeff-UK · August 30, 2024, 5:19pm

Fine if local testing and LNS on same local network as GW but by backhaul what we are interested in is the internet connection service provision from the premise (Office/Home to “The Internet” and ultimately the cloud service where remote LNS is hosted), so equivalent to say BT, Virgin Media, Vodafone or any of >100 AltNets here in UK, Verizon or Comcast or… in US, France Telecom, Bouyuge Telecom, Free, Orange… in France, Swisscom, Deutch Telecom, Fastnet, SingTel, or whoever is your internet communication service provider in your geography

descartes · August 30, 2024, 9:37pm

This:

was this:

There are other items suggested that haven’t been explored yet:

Asked by Jeff as well.

Just asked by Jeff above.

This is the fundamental test that will eliminate TTS OS & the gateways. Then you can start extending the network in your office between the server & the LNS.

Please consider answering questions, it’s how we can figure out ways to help you. Otherwise it’s all guessing.

digidhamu · August 31, 2024, 2:16am

I have connected my home GW to the internet using ACT Fibrenet as the service provider, with the connection going through a Wi-Fi-Fiber router. The TTN OS LNS is hosted on AWS.

digidhamu · August 31, 2024, 2:28am

How many is all? 2, 10, 100??

Currently about 40 GWs

or a change in Basic Station config if that’s what you are running (are you running that, you don’t say)

No configuration changed

Can you run TTS OS in your office with one of these RAK gateways to make sure there are no firewall or other port issues. If there are, you can then debug that more easily.

I am very sure there is no firewall in between.

I was seeing these logs as you advised via command line. This logs are captured after applying the new token in GW and even I have restarted GW too.

INFO    Setting up core component
INFO    Setting up Identity Server
INFO    Identity Server telemetry is enabled. Check the documentation for more information on what is collected and how to disable it   {"documentation_url": "https://www.thethingsindustries.com/docs/reference/telemetry/identity_server", "namespace": "identityserver"}
INFO    Setting up Gateway Server
INFO    Setting up Network Server
INFO    Setting up Application Server
INFO    Setting up Join Server
INFO    Setting up Console
INFO    Setting up Gateway Configuration Server
INFO    Setting up Device Template Converter
INFO    Setting up QR Code Generator
INFO    Setting up Packet Broker Agent
INFO    Setting up Device Repository
INFO    Setting up Device Claiming Server
INFO    Starting...
WARN    No cluster key configured, generated a random one       {"key": "2f9f9a71a25a98f7043cc305a3e3d1afa5daae74e377d797e0c18e5a38ab9cf6", "namespace": "cluster"}
INFO    Listening for connections       {"address": ":1884", "namespace": "grpc", "protocol": "gRPC"}
INFO    Listening for connections       {"address": ":8884", "namespace": "grpc", "protocol": "gRPC/tls"}
INFO    Listening for connections       {"address": ":1885", "namespace": "web", "protocol": "Web"}
INFO    Listening for connections       {"address": ":8885", "namespace": "web", "protocol": "Web/tls"}
INFO    Listening for connections       {"address": ":1886", "namespace": "interop", "protocol": "Interop"}
INFO    Listening for connections       {"address": ":8886", "namespace": "interop", "protocol": "Interop/tls"}
INFO    Finished unary call     {"duration": 0.0016, "grpc.method": "AuthInfo", "grpc.service": "ttn.lorawan.v3.EntityAccess", "namespace": "grpc", "peer.address": "pipe", "peer.real_ip": "172.225.219.54", "request_id": "01J6K29ECAGN3035G8H9E63021"}
INFO    Request handled {"duration": 0.0079, "http.method": "GET", "http.path": "/api/v3/auth_info", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29587", "peer.real_ip": "172.225.219.54", "request_id": "01J6K29ECAGN3035G8H9E63021"}
INFO    Request handled {"duration": 0.0004, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:56072", "peer.real_ip": "::1", "request_id": "01J6K2B70E7VNE30NC1FN6KRQC"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:56662", "peer.real_ip": "::1", "request_id": "01J6K2D1P53GD9XTZK3TX4P1F3"}
INFO    Request handled {"duration": 0.0002, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:38726", "peer.real_ip": "::1", "request_id": "01J6K2EWBBGBDRB2799R75RY2C"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:56192", "peer.real_ip": "::1", "request_id": "01J6K2GQ0XZK7ASNPECT70N44E"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/", "http.status": 302, "namespace": "web", "peer.address": "185.16.39.118:52216", "peer.real_ip": "185.16.39.118", "request_id": "01J6K2GSB2SQM62JH0ES14C7RQ"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:47420", "peer.real_ip": "::1", "request_id": "01J6K2JHP2K49B41JNP6CY34DS"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:44644", "peer.real_ip": "::1", "request_id": "01J6K2MCC0XM50HKECBESSGG2P"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:37766", "peer.real_ip": "::1", "request_id": "01J6K2P7111FD0FZN57CSCKZA0"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:40918", "peer.real_ip": "::1", "request_id": "01J6K2R1PDDM1Z5Z9PHKQBFCA5"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:59794", "peer.real_ip": "::1", "request_id": "01J6K2SWBX36R3TDAX4KYTZMFX"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:51870", "peer.real_ip": "::1", "request_id": "01J6K2VQ1HDDBMVD4QHRAC4Z55"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:47654", "peer.real_ip": "::1", "request_id": "01J6K2XHPGXFWNQPPYYW722R0S"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:42228", "peer.real_ip": "::1", "request_id": "01J6K2ZCCERV45E3ARQGNP23F5"}
INFO    Request handled {"duration": 0.0014, "http.method": "GET", "http.path": "/console/gateways/digidhamu-lorawan-02", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29231", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZEK0W1MRATNJ4S74RQRV"}
INFO    Request handled {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "duration": 719.8783, "http.method": "GET", "http.path": "/api/v3/console/internal/events/", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29964", "peer.real_ip": "172.225.219.54", "request_id": "01J6K29FMCRQ8038M6QJGDWSBX"}
INFO    Finished unary call     {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "auth.user_id": "admin", "duration": 0.0034, "grpc.method": "AuthInfo", "grpc.service": "ttn.lorawan.v3.EntityAccess", "namespace": "grpc", "peer.address": "pipe", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZEPYVWCFXJJJZW0BT4KM"}
INFO    Request handled {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "duration": 0.0045, "http.method": "GET", "http.path": "/api/v3/auth_info", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29231", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZEPYVWCFXJJJZW0BT4KM"}
INFO    Finished unary call     {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "auth.user_id": "admin", "duration": 0.0062, "grpc.method": "Get", "grpc.request.user_id": "admin", "grpc.service": "ttn.lorawan.v3.UserRegistry", "namespace": "grpc", "peer.address": "pipe", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZERCYANR1CY5XS8X8F0R"}
INFO    Request handled {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "duration": 0.0081, "http.method": "GET", "http.path": "/api/v3/users/admin", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29231", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZERCYANR1CY5XS8X8F0R"}
INFO    Request handled {"duration": 0.0025, "http.method": "GET", "http.path": "/assets/logo.svg", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29231", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZESZYFZN7C7D428F3JHF"}
INFO    Finished unary call     {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "auth.user_id": "admin", "duration": 0.006, "grpc.method": "ListRights", "grpc.request.gateway_id": "digidhamu-lorawan-02", "grpc.service": "ttn.lorawan.v3.GatewayAccess", "namespace": "grpc", "peer.address": "pipe", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZETQEQGTXG32DR08N2BX"}
INFO    Request handled {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "duration": 0.0075, "http.method": "GET", "http.path": "/api/v3/gateways/digidhamu-lorawan-02/rights", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29231", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZETQEQGTXG32DR08N2BX"}
WARN    No encryption key defined, return stored LBS LNS Secret value   {"grpc.method": "Get", "grpc.service": "ttn.lorawan.v3.GatewayRegistry", "namespace": "identityserver", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZEWAGZ7172GW3GQ2X3VR"}
INFO    Finished unary call     {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "auth.user_id": "admin", "duration": 0.0138, "grpc.method": "Get", "grpc.request.gateway_id": "digidhamu-lorawan-02", "grpc.service": "ttn.lorawan.v3.GatewayRegistry", "namespace": "grpc", "peer.address": "pipe", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZEWAGZ7172GW3GQ2X3VR"}
INFO    Request handled {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "duration": 0.0157, "http.method": "GET", "http.path": "/api/v3/gateways/digidhamu-lorawan-02", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29231", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZEWAGZ7172GW3GQ2X3VR"}
INFO    Finished unary call     {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "auth.user_id": "admin", "duration": 0.0195, "error": "error:pkg/gatewayserver:not_connected (gateway `digidhamu-lorawan-02` not connected)", "error_correlation_id": "21ac0d0a164f4530bf076eca5622f4c9", "error_name": "not_connected", "error_namespace": "pkg/gatewayserver", "gateway_uid": "digidhamu-lorawan-02", "grpc.method": "GetGatewayConnectionStats", "grpc.request.gateway_id": "digidhamu-lorawan-02", "grpc.service": "ttn.lorawan.v3.Gs", "grpc_code": "NotFound", "namespace": "grpc", "peer.address": "pipe", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZEZ2JJ8171PSSRHS7X6Y"}
INFO    Client error    {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "duration": 0.0217, "http.method": "GET", "http.path": "/api/v3/gs/gateways/digidhamu-lorawan-02/connection/stats", "http.status": 404, "namespace": "web", "peer.address": "172.225.219.54:29231", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZEZ2JJ8171PSSRHS7X6Y"}
INFO    Finished unary call     {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "auth.user_id": "admin", "duration": 0.0404, "grpc.method": "ListCollaborators", "grpc.request.gateway_id": "digidhamu-lorawan-02", "grpc.service": "ttn.lorawan.v3.GatewayAccess", "namespace": "grpc", "peer.address": "pipe", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZEZ1MGX88VRH525CK0Q4"}
INFO    Request handled {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "duration": 0.0425, "http.method": "GET", "http.path": "/api/v3/gateways/digidhamu-lorawan-02/collaborators", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29231", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZEZ1MGX88VRH525CK0Q4"}
INFO    Finished unary call     {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "auth.user_id": "admin", "duration": 0.0428, "grpc.method": "ListAPIKeys", "grpc.request.gateway_id": "digidhamu-lorawan-02", "grpc.service": "ttn.lorawan.v3.GatewayAccess", "namespace": "grpc", "peer.address": "pipe", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZEZ1Y9541083C2BZJ9FQ"}
INFO    Request handled {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "duration": 0.0458, "http.method": "GET", "http.path": "/api/v3/gateways/digidhamu-lorawan-02/api-keys", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29231", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZEZ1Y9541083C2BZJ9FQ"}
INFO    Finished unary call     {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "auth.user_id": "admin", "duration": 0.0082, "grpc.method": "List", "grpc.service": "ttn.lorawan.v3.ApplicationRegistry", "namespace": "grpc", "peer.address": "pipe", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZMC48W75K7TRYD9SR3V9"}
INFO    Request handled {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "duration": 0.0094, "http.method": "GET", "http.path": "/api/v3/applications", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29231", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZMC48W75K7TRYD9SR3V9"}
INFO    Finished unary call     {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "auth.user_id": "admin", "duration": 0.0096, "grpc.method": "List", "grpc.service": "ttn.lorawan.v3.GatewayRegistry", "namespace": "grpc", "peer.address": "pipe", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZN72SA0J2YZ3157QTA1K"}
INFO    Request handled {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "duration": 0.0107, "http.method": "GET", "http.path": "/api/v3/gateways", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29231", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZN72SA0J2YZ3157QTA1K"}
INFO    Finished unary call     {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "auth.user_id": "admin", "duration": 0.0038, "grpc.method": "AssertRights", "grpc.service": "ttn.lorawan.v3.GatewayBatchAccess", "namespace": "grpc", "peer.address": "pipe", "request_id": "01J6K2ZNAJGTANPQ8J9HT2R7KS"}
INFO    Finished unary call     {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "duration": 0.0048, "grpc.method": "BatchGetGatewayConnectionStats", "grpc.service": "ttn.lorawan.v3.Gs", "namespace": "grpc", "peer.address": "pipe", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZN8NDNVKHWA9WAQFSYA1"}
INFO    Request handled {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "duration": 0.0664, "http.method": "POST", "http.path": "/api/v3/gs/gateways/connection/stats", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29231", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZN8NDNVKHWA9WAQFSYA1"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:48646", "peer.real_ip": "::1", "request_id": "01J6K3171RKSDJ9A6KBVXNM548"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:48094", "peer.real_ip": "::1", "request_id": "01J6K331Q57XV463Q4ZS0DVFEP"}
INFO    Request handled {"duration": 0.0002, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:42246", "peer.real_ip": "::1", "request_id": "01J6K34WCZF4AZDY244AYX7XKB"}
INFO    Request handled {"duration": 0.0002, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:47458", "peer.real_ip": "::1", "request_id": "01J6K36Q1ZJKRXHCA6AJYQHB4J"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:34154", "peer.real_ip": "::1", "request_id": "01J6K38HQ17QNPQJ06MBHKT4J2"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:37400", "peer.real_ip": "::1", "request_id": "01J6K3ACCEZGFF5W0GJ90YY5K3"}
INFO    Request handled {"duration": 0, "http.method": "GET", "http.path": "/", "http.status": 302, "namespace": "web", "peer.address": "3.92.25.217:35442", "peer.real_ip": "3.92.25.217", "request_id": "01J6K3AKB9RRJEENKBKQC9WYXC"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:47326", "peer.real_ip": "::1", "request_id": "01J6K3C71G3D114GZBT1MQGY55"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:59660", "peer.real_ip": "::1", "request_id": "01J6K3E1QA5A65M5K0S3PM8PDN"}
INFO    Request handled {"duration": 0.0002, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:52380", "peer.real_ip": "::1", "request_id": "01J6K3FWCBT2KV667CZ333DKK1"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:42898", "peer.real_ip": "::1", "request_id": "01J6K3HQ24KQHH37SE7P47TCW3"}

netstat -nlt output

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 127.0.0.54:53           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:1883            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:1882            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:1881            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:1887            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:1886            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:1885            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:1884            0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:8883            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:8882            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:8881            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:8887            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:8886            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:8885            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:8884            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN     
tcp6       0      0 :::443                  :::*                    LISTEN     
tcp6       0      0 :::1883                 :::*                    LISTEN     
tcp6       0      0 :::1882                 :::*                    LISTEN     
tcp6       0      0 :::1881                 :::*                    LISTEN     
tcp6       0      0 :::1887                 :::*                    LISTEN     
tcp6       0      0 :::1886                 :::*                    LISTEN     
tcp6       0      0 :::1885                 :::*                    LISTEN     
tcp6       0      0 :::1884                 :::*                    LISTEN     
tcp6       0      0 :::8883                 :::*                    LISTEN     
tcp6       0      0 :::8882                 :::*                    LISTEN     
tcp6       0      0 :::8881                 :::*                    LISTEN     
tcp6       0      0 :::8887                 :::*                    LISTEN     
tcp6       0      0 :::8886                 :::*                    LISTEN     
tcp6       0      0 :::8885                 :::*                    LISTEN     
tcp6       0      0 :::8884                 :::*                    LISTEN     
tcp6       0      0 :::80                   :::*                    LISTEN     
tcp6       0      0 :::22                   :::*                    LISTEN

Hope I have answered all the questions asked.

Curious that I could not see 8887 port listening for connection in the command line log but I could see the port 8887 appearing in netstat.

Is this an issue in the configuration in YAML?