Mikrotik Gateway - Offline - not reconnecting

[cslorabox]

@descartes

Is there a repro for this?

It would be a small project to create from scratch, but like the majority of the LoRaWAN-related code I’ve written, the actual implementation I currently have is owned by a client as it is a small and integral piece of one of their systems. Although it came to have a different purpose it started out as a drop in replacement out of desperation for an older, buggy version of the LoRaServer UDP to MQTT bridge, quite literally all I did was lookup “python UDP example” and start grafting on logic to handle the sorts of messages the packet forwarder actually generated and wanted. Later when I switched to a more modern packet forwarder I had to fix a few places where I’d been lazy with the details.

How do you like my idea for an Open Gateway project?

It’s a nice idea, but a lot of work that tends to be justified when there’s a volume of similar boxes people are wanting to use. My impression is that people are already doing this for some of the Multitech boxes (eg, taking the open source build and doing more with it, particular with regard to fleet management for TTN communities), Dragino seems to effectively do it and while I might be mistaken I though Kersing had some past comments about doing a variation of their image with his packet forwarder on it. And then essentially anything based on a pi already is that way, unless someone chooses a vendor install with extra cheese on top.

[descartes (Nick McCloud)]

It is a cold day in hell when a client has enough money to pay for the IP, especially if I then can’t use it else where.

As for a gateway, no participation in a project that addresses your concerns then? Be careful, I may get a concentrator design done, implement it on OpenWRT and not fulfil your expectations.

[cslorabox]

@descartes

It is a cold day in hell when a client has enough money to pay for the IP

This is both off topic, and utter nonsense. Most people in the world, be they independent contractors or employees, are paid for their efforts. A comparatively smaller number sell (title or licenses to) something over which they ever had ownership.

I frankly forget if it was a cold day in January or a scorching one in July, but it was a quite ordinary day on which I billed a few hours for that particular tiny part of creating their (not “my”) codebase.

especially if I then can’t use it else where

This is even more true of boutique software: why develop at your own expense something that would be difficult to sell elsewhere, when the people who need it will simply pay you to create theirs?

If I’d invested in speculatively developing it, and then someone came along and wanted to buy out my investment,then sure that would be expensive. But in this more typical case, I was probably paid more to identify that this particular component needed creation, than I for the time spent to actually create it. I made good money doing it, but in an entirely ordinary and everyday sort of way.

Today, if I had an actual need for a python implementation of the Semtech UDP protocol over which I had ownership (for example, to give away licenses to), it would take an hour or two to create one, starting from a UDP example and the Semtech docs as I described above and did before. The thing is, I don’t have such a need, and the idea that even anyone else had one was simply debug brainstorming.

[descartes (Nick McCloud)]

@cslorabox

Most people in the world

I’m not most people - I retain IP in most instances as to get exclusivity to the research, design, implementation & testing is far outside most budgets - I choose not to be prevented from fulfilling similar business problems else where because I wrote some code for someone once that does something similar. I get reasonably remunerated for the first implementation and then make hay on subsequent implementations, often folding back enhancements in to the original/previous clients versions, certainly any fixes that arise. With a clearly identified market place, many potential clients will have similar business problems to solve.

Still interested to hear if you are up for some Open Gateway action.

[cslorabox]

Still interested to hear if you are up for some Open Gateway action

In terms of the technical effort, this is another “been there, done that but someone else paid and owns it.”

Nothing really prevents me from helping with an open effort, and if something gets going I’m happy to contribute assistance to solve key problems. But the reality is that the people I know who have a serious need for gateways and are willing to buy the parts, set them up in the field, etc, are precisely the people who paid me to design one when we could not find anything that met their full range of needs.

A key part of what that provided was a sufficient scope of need, focused on a single hardware configuration, to make doing it moderately sensible.

So, for example, if a bunch of RAK7258 owners were really clambering for a tuned DIY build, sure, I’d help out. Or some other viable configuration for which I had access to reference hardware.

But in the course of considering possibilities, I already own three concentrators, one of which matches my client’s current business plan. Sure, I could go buy a pygate and wire it to a pi zero… but in the grand scheme of things, what does that really accomplish, and why should I spend my money on it? I don’t have an unmet gateway need.

Personally, an 8-channel LoRaWAN concentrator is an expensive component.

Professionally? It’s all about the u.fl connectors and mPCIe standoff stacking height.

Building code to run on integrated hardware, or 8cm jumper wire hardware is the easy part.

Bolting stuff together in a box suitable for the field is where it turns out to be hard.

[cultsdotelecomatgmai (Tim Everitt)]

Please can @Johan_Scheepers advise if his gateway is connected to TTN and working normally.

If yes, then that’s good.
If no, then we have to acknowledge that this topic’s “you have a sprained ankle? let’s do a heart transplant” and “if you’re trying to get over there then don’t start from here” approach has not worked.

As a fan of reliability engineering I think that a community of >100k people with >12k connected gateways should be doing better.

[Johan_Scheepers]

Hi,

No still no luck, have upgrade , reset, reconfigured, but it still does nor pass the TTN traffic to LoRaWAN of the LoRaWAN traffic to TTN.

Have posted on the Mikrotik forum 3 days ago but they have not approve my post or looked at it.

Have sent them a mail now, but it can take up to 3 working days fro them to respond.

[bwooce]

@kersing

Its called Packet Sniffer and is located in the Tools menu.

Can you try sniffing some packets please? It’d be great to know which TTN endpoint it’s trying to connect to, for a start. Which one is configured in? The default mikrotik ones, and which one did you choose?

It is weird that it’s RX is okay but it’s not forwarding it on. It’d be great to see a trace of it trying to.

[arjanvanb (Arjan)]

@bwooce

although the existence of the proprietary line there reminds me this list shows CRC-error packets by default e.g. noise that looks like LoRa

This.

The gateway does not know any of the LoRaWAN secrets, so cannot validate a LoRaWAN MIC, hence has no way to rightfully assume it’s only seeing LoRaWAN. And yet, still it does. Also, it may indeed be configured to also show traffic that doesn’t pass a basic CRC. (Or even shows packets that have no CRC at all?)

So yes: we’ve seen these gateway logs to be confusing before. When it says something is a join request, join accept, uplink or downlink, then that’s only based on boldly interpreting the message’s first byte, without even knowing if it’s LoRaWAN to start with. (And given the IQ inversion it should not hear traffic from other LoRaWAN gateways, so should not receive join accepts nor downlinks?) And what it’s showing as the DevAddr is only based on whatever is in the message’s 2nd thru 5th byte (1…4 with zero-based index), reversing their order, again boldly assuming it’s LoRaWAN. Also note that LoRaWAN should always use coding rate 4/5.

Asides: gateway EUI, DevAddr, and anything else that’s transmitted through the air are not secret. No need to redact those in screenshots.

And as a workaround for the very annoying handling of images in desktop browsers on this installation of the Discourse forum software, one can add a relative image size to the Markdown text, like the ,70% and ,80% in ![image|1000x75,70%](upload://2RGfN8nRVM6LsGHHqkQPdfe2cPC.png) and ![image|799x341,80%](upload://k402ihXXpAp8H3rQgx6mPTNPAyS.png) to get:

[cslorabox]

@arjanvanb

(And given the IQ inversion it should not hear traffic from other LoRaWAN gateways, so should not receive join accepts nor downlinks?)

I seem to recall intentionally mis-setting this in the past as an experiment and still getting a moderate amount of traffic through

And what it’s showing as the DevAddr is only based on whatever is in the message’s 2nd thru 5th byte (1…4 with zero-based index), reversing their order, again boldly assuming it’s LoRaWAN. Also note that LoRaWAN should always use coding rate 4/5.

While you’re not wrong, the console logs of the current Semtech packet forwarder repo assume they can extract a device address from messages, too.

One can’t read too much credibility into these things, but they can be handy for debugging when you see something you were expecting to.

Also note that LoRaWAN should always use coding rate 4/5.

That’s probably the most important point, and I’ll readily admit I’d previously overlooked it. When filtering to only CR 4/5 messages, the log gets a lot shorter. But I suspect it does still show some valid traffic.

Although I’ve heard of radios getting into a mode (likely a mis-set internal bit) where they simply invent CRC error packets and never receive any good ones, I’ve not given up on suspicion of a software protocol or IP networking issue. But we’re operating on limited information.

Getting some raw information from the packet capture would be great. Not only would this expose networking issues, it should be possible to take a message from a known node seen in the packet capture, and compare it to the raw message seen by another gateway, or that the node was known to transmit (ie, the post-encryption buffer actually loaded into the node’s radio).

[fcojmontilla]

Hi,

@Johan_Scheepers Will try to give a hand (Mikrotik Consultant & Trainer), need screenshots of:

• RouterOS version (can be seen in winbox window titlebar, or system > packages)
• Routerboard firmware ( system > routerboard )
• IP > DNS
• IP > DNS [Static] button
• LoRa > Devices, double click yours, switch to Stats tab

Here the ratio between Valid RX Count (radio received messages) vs RX Forwarded (messages sent to TTN) is what tells us if traffic is being forwarded or not to the gateway. (Device uptime: 14 days.)

I’ve been running a LoRaWAN gw for months, rock solid so far, but for the TTN servers, whose connectivity is flaky, not to mention monitoring.

When it has failed and wasn’t the gateway (I found DigitalCatapultUK to be the most reliable from here) problem was usually the dns resolution.

Can you run tools > traceroute to your TTN gateway, leave it running some minutes and post a screenshot?

You have plenty of killer tools inside ROS for troubleshooting, as was mentioned in the thread. Packet sniffer, packet capture including live streaming to wireshark, etc)

I suspect problem here is either ROS version (I wouldn’t use anything older than 6.46.5 for LoRa deployments, while I only use long-term 6.45.x on pure network routers) or UDP masquerading issue.

Networking wise, the forwarder and protocol (UDP) aren’t the best to have several LoRaWan devices behind the same internet connection, may require some traffic mangling programming to get things right if you expect to have several LoRaWan devices behind same internet uplink.

[Johan_Scheepers]

Hi,

Thanks,

First I can ping via the ethernet interface the RouterOS but can not log into the RouterOS with WinBox.

Now I connect the the RouterOS via the WIFi AP WinBox works and I have full internet access.

system > packages

system > routerboard

IP > DNS

IP > DNS [Static] button

LoRa > Devices, double click yours, switch to Stats tab

Thank you for your time

[fcojmontilla]

Hello,

First I can ping via the ethernet interface the RouterOS but can not log into the RouterOS with WinBox.

Looks like you kept the default firewall, which blocks connections coming from ether1… as wAP is a one ether port device, is blocking winbox from connecting.

As this is LAN device, I assume you have another router/firewall protecting your LAN; so you can safely disable ip > firewall > filter rules; or more easily, go to interfaces, interface list tab, there will be a LAN and a WAN list; remove ether1 from the WAN list and add it to the LAN list.

that’s why you can connect via wireless, because that port (wlan1) is not filtered by the firewall.

---

Your firmware (sort of router BIOS) is out of sync… press [Upgrade] button, you’ll receive a firmware succesfully flashed, reboot afterwards.

Firmware upgrades come embedded with ROS upgrades, but ROS upgrades won’t flash firmware by default. Best practice is keeping firmware always in sync with ROS version to avoid issues.

---

I’m checking eu.thethings.network traceroute and sadly (specially because TTN monitoring is so unreliable is useless) seems they’re setup not to answer ICMP packets, making troubleshooting rather miserable… guess they’re scanned/attacked so much that they had to resort to that… but something tells me the problem doesn’t lie in L3 connectivity towards the TTN gateway.

---

This is odd… makes me think some sort of internal issue with the radio. Can you please post LoRa > Your device window (settings, specially antenna gain) masking the IDs is fine. Untick forwarding of packets with errors.

---

In summary:

0.- disable the firewall/add ether1 to LAN interface list to be able to log straight via ethernet into the wAP.
1.- upgrade firmware and reboot. Check the status of the LoRa device.

Let me know how it goes to guide you further if firmware upgrading doesn’t fix it.

[fcojmontilla]

Here’s a way to confirm if there’s L3 connectivity between your device and TTN gws.

You need to know the IP of the TTN gw first, e.g. 52.169.76.203 for router.eu.thethings.network.

Open IP > Firewall, go to connections tab, shall be some on your wAP, as aparently you have default firewall programming. If this is not the case, you’ll need to check this on your border router connecting to internet.

Now click the funnel icon, and set it, (with 52.169.76.203 as IP) like

Here we can see a ping I’m running w/o any response (0 bytes reply), and two Sucesfully “stablished” connections with the TTN gateway IP (same port 1700 for msg upload and download) that sit established but idle (0/0 Rx/Tx bytes/s).

SACs stands for Seen Reply, Active, Confirmed, src-natted.

The three first flags (SAC) means there’s proper bidirectional communication.

[Johan_Scheepers]

@fcojmontilla

52.169.76.203

Thanks all up now, have re-sync

From the fire wall I can see the TTN

Thank you for your help, very nice to meet you,and good to have Mikrotik Consultant & Trainer assistance. smiley:

[fcojmontilla]

Glad it helped

[bsiege]

Another Rule of thumb: Always check for newest Firmware when starting a new project. Or at least read release notes:

What’s new in 6.45.7 (2019-Oct-24 08:44):
MAJOR CHANGES IN v6.45.7:

!) lora - added support for LoRaWAN low-power wide-area network technology for MIPSBE, MMIPS and ARM;

This was very first release with lora support. Since then where 9 releases, many of them with lora improvements. We all know the first questions when calling support !!

[bsiege]

@descartes

Still interested to hear if you are up for some Open Gateway action.

Many of us own a “Open Gateway” aka TTN Kickstarter Gateway. But for some reasons nobody burnt it’s fingers for improving some of the well known issues. Not even “simple tasks” like deactivating wifi when not used where implemented… And as far as i know for TTIG it is about the same?

[kersing (Jac Kersing)]

@bsiege

And as far as i know for TTIG it is about the same?

Sources for TTIG are not available so fixing bugs is impossible.

When it comes to fixing bugs in the Kickstarter software, have you tried compiling from source? It is nothing like an easy build of a packet forwarder for a Linux host, it is serious embedded programming using a complex framework not many will be familiar with.

[bsiege]

@kersing

Sources for TTIG are not available so fixing bugs is impossible.

Then it’s worse But does not make a difference at the end.

have you tried compiling from source

I know my limits.
I partly agree with the easyness under Linux, i use your forwarder in a Multitech Conduit.
There are way too many linux-based appliances out there with catastrophic misconfigurations and security flaws in superfluous packages. In many cases there is only a shift of interests…