Console shows not connected, traffic goes through, how can I use the TTN mapper?

s54mtb · May 22, 2020, 12:14pm

This is strange: data
gww

now the question: how can I check the GW coverage when it is not shown on TTN map (errorneusly reported as not connected) ???

kersing · May 22, 2020, 4:34pm

No it is not, the magic word is timezone.

Basically you are out of luck.

@jpmeijers I just checked for my gateways and it seems ttnmapper no longer shows gateways which the console reports off-line where ttnctl provides the correct (working) status. Any hint on what to do except trying to get TTN fix the problem? (@johan @htdvisser ttnmapper not showing active gateways and coverage is annoying to say the least. Any progress on fixing this? All my MQTT connected gateways show offline in the console for 9 days now)

cultsdotelecomatgmai · May 23, 2020, 8:04am

Hi @s54mtb, I follow the normal industrial approach of “don’t trust the infrastructure to monitor the infrastructure”. I use the “coal-mine canary” approach.

I co-locate a low-cost LoRaWAN GPS tracker with each of my gateways. The trackers are fixed to uplink once every 30mins using SF7 with power of 1mW. The uplinks go to a dedicated application and MQTT feed to a cloud-based virtual machine. I get the following delivered completely outside the LoRaWAN infrastructure:

The location of the gateway so I can map independently with no admin.
The operational status of the gateway; up and running, or not.
The stability of the gateway RF receiver via metadata RSSI and SNR.
The operational status of the end-to-end delivery - LoRa–stuff–MQTT; up and running or not.

I have software that detects if the “coal-mine canary” stops singing and notifies me. Standard industrial exception reporting.

pe1mew · May 23, 2020, 8:32am

My canary stopped singing the day before yesterday after singing for more than 3 years while th coal mine is still operating. Now i am blind again

cultsdotelecomatgmai · May 23, 2020, 8:42am

You are completely correct! Canaries need care and feeding.

kersing · May 23, 2020, 9:07am

For monitoring that works like a charm and I would recommend such an approach for ‘critical’ infrastructure.
However it doesn’t help to keep the gateway on ttnmapper as it still considers the gateway to be down. And for the community network keeping the map up to date and have running gateways displayed on the map is important imho. Sadly about 1600 of the gateways in Europe (according to the numbers presented at the virtual conference) won’t be shown as all MQTT connected gateways are reported to be down it seems (at least all mine are while the UDP gateways are fine). Ttnctl reports them to be up however that doesn’t help with ttnmapper.

cultsdotelecomatgmai · May 23, 2020, 9:44am

Hi @kersing, you are correct to point out the tension between the commercial/industrial use of LoRaWAN and community use. Most of the commercial/industrial LoRaWAN that I see (in the UK) is on networks like Actility, Loriot and Orbiwise and is not normally visible to TTN users.

Most commercial/industrial users are naturally very selfish; they want to see their stuff, they want SLAs and they have zero interest in helping open-source or community efforts. I have chosen to put all the commercial/industrial work that I control on TTI cloud-hosted LoRaWAN. The “coal-mine canary” monitoring that I described is ideal for them. For community work, I fully agree that TTN and tools like TTN Mapper are the most appropriate.

jpmeijers · May 24, 2020, 8:10am

It seems like it could be possible for TTN Mapper to use ttnctl to get the gateway statuses. I’ve tried logging in with my user and I can see the status of a gateway I do not own and I’m not a contributor to. Easiest way would be to iterate through all known gateways in TTN Mapper and one by one request the status via ttnctl. This would however take a very long time and cause a lot of unneccesary load on the TTN servers, as a couple of api calls need to be done per gateway. And there are around 60 000 known gateways.

I would prefer not to have to wast time on writing a piece of code like this, as I want to focus on getting TTN Mapper comaptible with V3.

If anyone of you feel like it, please contribute.

The code that currently gets the gateway statuses from the TTN website is here:

github.com

ttnmapper/ttnmapper-web/blob/master/processing/rest_gateway_updates_web.py

#!/usr/bin/python
from __future__ import print_function
import MySQLdb
import paho.mqtt.client as mqtt
import json
import time
import datetime
import dateutil.parser
import os, sys
from geopy.distance import great_circle
import urllib, urllib2
import configparser
import subprocess

ignored_euis = ["0102030405060708", "0000000000001DEE", "000000000000000E", "000000000000FFFE", "3135323512003300", "323433362D005800"]

retry_using_noc = [] # gateways that does not have a lastheard set on this api

lockfile = os.environ['TTNMAPPER_HOME']+"/lockfiles/gateway-updates-web-lock"
#This is to check if there is already a lock file existing#

This file has been truncated. show original

And an example of how to get a list of known gateways from the TTN Mapper database can be seen here:

github.com

ttnmapper/ttnmapper-web/blob/master/processing/circles_all_geojson.py#L35


outputfile = "circles"


features = []


# you must create a Cursor object. It will let
#  you execute all the queries you need
cur_gateways = db.cursor()
cur_location = db.cursor()
cur_blocks = db.cursor()


cur_gateways.execute("SELECT DISTINCT(gwaddr) AS gwaddr FROM gateways_aggregated WHERE `last_heard` > (NOW() - INTERVAL 5 DAY) ORDER BY gwaddr ASC")
total = cur_gateways.rowcount
count = 0
for gwrow in cur_gateways.fetchall():
  count += 1
  features = []


  gwaddr = str(gwrow[0])
  gwlat = 0
  gwlon = 0

One problem with using ttnctl I see is that we won’t be discovering new gateways. For that to happen we need to change this micro service to also add gateways to the TTN Mapper database if they don’t exist yet. This would also cause much more load on the TTN Mapper database - something that is already a problem.

jpmeijers · May 24, 2020, 8:18am

If someone could write a python script that takes in a single gateway ID and returns the last seen time, that would help tremendously. Then I can add the database and TTN Mapper-specific logic.

I think the most difficult part is either wrapping ttnctl, using valid login details which should be stored somewhere, or talking to the API endpoints directly.

Basler · May 24, 2020, 6:12pm

Ignoring any ttnctl login difficulties … you could use this first suggestion:

from subprocess import check_output
from datetime import datetime, timedelta

def getGatewayLastSeenDateTimeFromEui(argGatewayEui):
    """
    Returns the last seen datetime (UTC Time) of a gateway acquired by ttnctl as datetime object, or 'None' if gateway not available. 

    ttnctl must be callable from the shell. User must be logged in. 
    Expects one status output line (if gateway available) in the format: Last seen: 2020-05-24 16:54:00.04129753 +0200 CEST

    Parameters:
    argument1 (string): Takes Gateway EUI as string

    Returns:
    datetime: Last seen datetime in UTC time or None

    """
    statusoutput = ""
    datetimeObj = None
    try:
        statusoutput = check_output('ttnctl gateways status '+ argGatewayEui, shell=True)
    except Exception:
        pass
    #print(statusoutput) # for debugging or curiosity
    indexLastSeen = statusoutput.find("Last seen: ")
    if (-1 != indexLastSeen):
        dateStr = statusoutput[indexLastSeen+11:statusoutput.find("\n",indexLastSeen)]
        datetimeObj = datetime(int(dateStr[0:4]), int(dateStr[5:7]), int(dateStr[8:10]), int(dateStr[11:13]), int(dateStr[14:16]), int(dateStr[17:19]))
        indexOfTimeZoneDiffMinus = dateStr.find("-", 20)
        indexOfTimeZoneDiffPlus = dateStr.find("+", 20)
        if (-1 != indexOfTimeZoneDiffMinus) or (-1 != indexOfTimeZoneDiffPlus):
            indexDiff = indexOfTimeZoneDiffMinus + indexOfTimeZoneDiffPlus + 2
            offsetToZuluTime = timedelta(hours=int(dateStr[indexDiff:indexDiff+2]), minutes=int(dateStr[indexDiff+2:indexDiff+4]))
            if (-1 != indexOfTimeZoneDiffMinus):
                datetimeObj = datetimeObj + offsetToZuluTime
            else:
                datetimeObj = datetimeObj - offsetToZuluTime
    return datetimeObj

jpmeijers · May 24, 2020, 7:46pm

Ahh excellent. Thanks a lot @Basler. I’ll try and get this running as soon as possible (tomorrow).

jpmeijers · May 25, 2020, 4:26pm

It’s running, but it’s very slow. Need to iterate through 47000 gateway IDs, and it takes about one second each. We’ll have to be patient.

I don’t think this is a sustainable solution.

Basler · May 26, 2020, 3:03pm

Yes it is slow, seems to be response time from ttnctl…
Lot’s of gateway icons disappeared from the map now, even though their blue coverage lines still appear.

descartes · May 26, 2020, 3:39pm

A little faux-AI could be introduced - each time you do a run, increment a score for any gateway still online - once you know who’s mostly online, you can then choose to check those that were recently online but have apparently gone off, to see if they are back, check new gateways until they build up their confidence score, check a % of the intermittent gateways and do a % of the known-good gateways as a well.

nielsmaneschijn · May 26, 2020, 7:49pm

My gateway disappeared 13 days ago, at the same day as Jac’s.
Sad canary is sad.

Roberto69 · July 20, 2021, 5:36pm

Maybe slightly off-the-topic, but anyway: which HW is good choice for TTN mapping? My old RAK811 died in v2 …

Thank You!

oberour · July 21, 2021, 7:20am

Hello, Dragino LGT-92 or TTGO T-Beam are good choices.

jpmeijers · July 21, 2021, 1:28pm

There are too many devices out there that are good, that it’s difficult recommending a specific one. My favourite one is the pink one. Why? Well, because it’s pink! And it’s actually a powerbank with some lora gps board taped to it.

20210721_151835

Maybe one day I’ll get the time to write reviews for these trackers (and all the other ones not in the photo).