TTN Mapper WebHook template status - WORK IN PROGRESS

On the status page it indicates that on the 13th of april an update was performed:

Does this implicate that running TTN Mapper with a GPS tracker and webhook on V3 is getting a step closer? Would like to migrate to V3 with my GPS trackers but only if that means that I can keep running TTN Mapper :slight_smile:

If you have an application and device set up on V3 then in the integrations under webhooks you should now see an option for TTN Mapper :slight_smile:

Fix Base URL

In the list of webhooks, click and open the newly added integration for TTN Mapper. Under Endpoint Settings, there is a Base URL. This value should read https://integrations.ttnmapper.org/tts/v3. If it’s different, change it and then click save changes at the bottom of the page.
Source

1 Like

Thanks guys, missed the part that it’s under webhooks.
Unfortuneately, my data doesn’t show up in TTN Mapper. I changes the webhook URL as per the instructions.

Even if I check in the CSV file on TTN Mapper, it stays empty since the migration to V3. In the V3 console I can see the uplink messages arriving. I assume that the problem must be between the V3 backend and the integration with TTN Mapper.

Is it a problem that I’m using the same DevID in V2 and V3? I cannot find documentation about that.

Or should that be https://ttnmapper.org/integrations/tts/v3 ??? (Default… if this is wrong we need to flag and get it corrected…)

Please read the docs linked to by @engelking. Corrections have already been made, but TTI has a release cycle that is delaying the fix from being deployed.

@emacee it sounds like your are doing everything correctly. If you followed the instructions to change the base url as per the documention, that is correct. Using the same DevId for both v2 and v3 should also be fine.

If you can send me the email address you used in the webhook as a private message, I can look at the logs and try and figure out what is going wrong. I’ve had a couple of reports that data isn’t appearing on TTN Mapper, but when I test it everything works. Must be some edge case we are hitting.

1 Like

JP, If the default as pulled up by the current version of the webhook/integration is incorrect then with respect you need to do something about highlighting that in the Forum vs just a belated ‘go read the docs’ message. :wink:

What I see is

I assume this is incorrect, that you have fixed with the TTI core team, and that yes there is a release cycle and users will get corrected version ‘soon’…

But, when was problem identified? When was fix issued? What has been done to communicate this to users - casual or otherwise? And before anyone shouts ‘use forum search’ I just did that and a) wouldn’t know to look/search for a problem as not aware of one and nothing mentioned previously and b) a search for either of the url versions listed above only calls out one thread - this one - where we called them out above! :slight_smile:

Users of V2 clicked the TTNMapper integration and data automagically appeared on the map, there is no reason to assume that on the V3 console they had to do anything different and they have to go read a different set of docs instead …unless you tell them. :man_shrugging:

I believe it has been previously suggested that you amend the TTMapper site to call out a specific link to the docs. At the moment all users see is:

The “please read here” links to the “migrate GWs last” thread on the TTN Forum, and even in the FAQ, the only link to documentation is a link to the doc on TTN - not to any specific TTNMapper docs! Casual users have no obvious way of knowing they need to go searching somewhere else.

For anyone who saw the original posts or who now are looking at adding TTNMapper Integration via V3 Webhook please go in and modify the default url as shown in the image above and correct it to

As highlighted by @engelking above

Until TTN V3 CE receives appropriate update to correct the error.

Do you want to help? I don’t have time to add V3 support and micromanage a community.

Per DM I will post a specific warning thread wrt this issue later :wink: :+1:

Update: Done: Temporary TTNMapper to TTS(CE)/TTN V3 Integration Issue - Work around available and fix in process

I have an issue with missing accuracy data using the v3 TTNmapper webhook.

I have upgraded the firmware on my Dragino LGT-92 tracker for it to report both altitude and HDOP - which shows up in the uplink message in e.g. Storage Integration:

{“result”:{“end_device_ids”:{“device_id”:“dragino-lgt-92”,“application_ids”:{“application_id”:“where-i-am”},“dev_eui”:“A84041DB61826E48”,“dev_addr”:“260B3B37”},“received_at”:“2021-06-24T06:06:28.622801409Z”,“uplink_message”:{“f_port”:2,“frm_payload”:“A5KaaACiIKwOB2QACfgx/zH2”,“decoded_payload”:{“ALARM_status”:“FALSE”,“Altitude”:127.9,“BatV”:3.591,“FW”:164,“HDOP”:2.55,“LON”:“ON”,“Latitude”:59.939432,“Longitude”:10.625196,“MD”:“Disable”,“Pitch”:635.37,“Roll”:0.09},“rx_metadata”:[{“gateway_ids”:{“gateway_id”:“eiksmarka-gw”,“eui”:“A840411DA9084150”},“time”:“2021-06-24T06:06:28.388355Z”,“timestamp”:2397012860,“rssi”:-129,“channel_rssi”:-129,“snr”:-11.8,“location”:{“latitude”:59.94865416,“longitude”:10.61466098,“altitude”:155,“source”:“SOURCE_REGISTRY”},“channel_index”:5}],“settings”:{“data_rate”:{“lora”:{“bandwidth”:125000,“spreading_factor”:12}},“coding_rate”:“4/5”,“frequency”:“867500000”,“timestamp”:2397012860,“time”:“2021-06-24T06:06:28.388355Z”},“received_at”:“2021-06-24T06:06:28.413527536Z”,“consumed_airtime”:“1.810432s”}}}

However, I am only able to find the GPS data searching for device-id in Advanced maps, and they do not contain HDOP information:

id, time, device_id, application_id, gateway_id, modulation, datarate, snr, rssi, freq, f_cnt, latitude, longitude, altitude, accuracy meters, hdop, sats, location provider, user_agent, experiment name
209297831,2021-06-24 06:06:28.622801+00,dragino-lgt-92,where-i-am,eiksmarka-gw,LORA,SF12BW125,-11.80,-129.00,867.500,0,59.939432,10.625196,127.9,

What could be causing this?

Have you read the project status link at the top of the TTN Mapper website???

Yes, I have. According to the status page, the heatmap should use data from all networks but I am not seeing my data. I thought this was related to missing accuracy data, but is not entirely clear to me whether altitude and HDOP is a prerequisite or a preference.

Nevertheless, it is strange that HDOP is sent through the webhook. Could it be case sensitivity in the property “hdop”?

Yes I believe so. JSON keys are case sensitive. Most things in the computer world are. Only Windows is the exception.

But let’s have a deeper look and make sure.

Looking at FAQ | TTN Mapper Documentation we see:

The JSON object should contain the keys “latitude ”, “longitude ” and “altitude ” and one of “hdop ”, “accuracy” or “sats”.

So the recommendation is indeed to use the lower case variant.

Then we also have a look at what the actual code does to make sure which variants are also accepted:

And indeed, only the lower case variant is checked.

So to fix this you have two options:

  1. Change your json key to lower case.
  2. File a pull request on Github to add the upper case variant.

The first option is the easiest, but the second option will benefit the community more.

2 Likes

Thanks for your reply! I have tested it now, and the HDOP issue is solved with using lower case. Location accuracy still “null”?. Now I’ll just have to wait til tomorrow to see if any of the data shows up… I am new to Lora and especially Github, but I’ll see what I can do. :slight_smile:

@edwin Perhaps you should update the payload formatter in the user manual for Dragino LGT-92 to use lower case for hdop so that it will work with the TTNmapper webhook out of the box?

1 Like

@jpmeijers

I have Abeeway trackers, their payload decoders are larger than 4k, so the decoding are not done in the V3 application, but done in my application in AWS.

Can these bee integrated into TTN Mapper?

For TTN Mapper you need a decoder on TTN’s side. Best is to rewrite the payload decoder to be smaller, and maybe only parse the location data. 4k for a parser for GPS trackers sounds like overkill to me.

1 Like

Looking at where TTS is headed regarding the location-solved webhook callback, it’s best to stick to these keys above.

Yes the Abeeway decoder is rather large.

context.Decoder = function(bytes, port) {

 function step_size(lo, hi, nbits, nresv) {
    return 1.0 / ((((1<<nbits) - 1) - nresv) / (hi - lo));
  }

  function mt_value_decode(value, lo, hi, nbits, nresv) {
    return (value - nresv / 2) * step_size(lo, hi, nbits, nresv) + lo;
  }

  // Gets the zero-based unsigned numeric value of the given bit(s)
  function bits(value, lsb, msb) {
    var len = msb - lsb + 1;
    var mask = (1<<len) - 1;
    return value>>lsb & mask;
  }

  // Gets the boolean value of the given bit
  function bit(value, bit) {
    return (value & (1<<bit)) > 0;
  }

  // Gets a hexadecimal representation ensuring a leading zero for each byte
  function hex(bytes, separator) {
    return bytes.map(function (b) {
      return ("0" + b.toString(16)).substr(-2);
    }).join(separator || "");
  }

  // Decodes 4 bytes into a signed integer, MSB
  function int32(bytes) {
    // JavaScript bitwise operators always work with 32 bits signed integers
    return bytes[0]<<24 | bytes[1]<<16 | bytes[2]<<8 | bytes[3];
  }

  // Decodes 4 bytes into an unsigned integer, MSB
  function uint32(bytes) {
    return (bytes[0]<<24 | bytes[1]<<16 | bytes[2]<<8 | bytes[3])>>>0;    
  }

  // Decodes 1 to 4 MAC addresses and their RSSI
  function mac_rssi(bytes) {
    var items = [];
    for (var offset = 0; offset < bytes.length; offset += 7) {
      items.push({
        mac_address: hex(bytes.slice(offset, offset + 6), ":"),
        // Sign-extend to 32 bits to support negative values; dBm
        rssi: bytes[offset + 6]<<24>>24,
      });
    }
    return items;
  }

  function message(code, descriptions) {
    return {
      code: code,
      description: code < 0 || code >= descriptions.length ? "UNKNOWN" : descriptions[code]
    };
  }

  var decoded = {};
  var i;

  var type = bytes[0];

  // All message types, except for Frame pending messages, share the same header
  if (type !== 0x00) {
    // Note: the Data Storage Integration stores nested objects as text
    decoded.status = {
      mode: message(bits(bytes[1], 5, 7), ["Standby", "Motion tracking", "Permanent tracking",
        "Motion start/end tracking", "Activity tracking", "OFF"]),
      sos: bit(bytes[1], 4),
      tracking: bit(bytes[1], 3),
      moving: bit(bytes[1], 2),
      periodic: bit(bytes[1], 1),
      on_demand: bit(bytes[1], 0)
    };

    // Trackers with a rechargeable battery:the percentage reflects the actual value
    decoded.batteryPersentage = bytes[2];
    
    decoded.temperature = Math.round(100 * mt_value_decode(bytes[3], -44, 85, 8, 0)) / 100;
    
    decoded.ack = bits(bytes[4], 4, 7);
    
    decoded.data = bits(bytes[4], 0, 3);
    
    decoded.lastResetCause = "lastResetCause: " + bytes[5];


  }

  switch (type) {
    case 0x00:
      decoded.type = "FRAME PENDING";
      decoded.token = bytes[1];
      break;

    case 0x03:
      decoded.type = "POSITION";
      switch (decoded.data) {
        case 0:
          decoded.position_type = "GPS fix";
          decoded.age = mt_value_decode(bytes[5], 0, 2040, 8, 0);
          // Signed 32 bits integers; LSB is always zero
          decoded.latitude = (bytes[6]<<24 | bytes[7]<<16 | bytes[8]<<8) / 1e7;
          decoded.longitude = (bytes[9]<<24 | bytes[10]<<16 | bytes[11]<<8) / 1e7;
          // Estimated Horizontal Position Error
          decoded.ehpe = mt_value_decode(bytes[12], 0, 1000, 8, 0);
          break;

        case 1:
          decoded.position_type = "GPS timeout";
          decoded.timeout_cause = message(bytes[5], ["User timeout cause"]);
          for (i = 0; i < 4; i++) {
            // Carrier over noise (dBm) for the i-th satellite seen
            decoded["cn" + i] = mt_value_decode(bytes[6 + i], 0, 2040, 8, 0);
          }
          break;

        case 2:
          // Documented as obsolete
          decoded.error = message(0, ["UNSUPPORTED POSITION TYPE " + decoded.data]);
          break;

        case 3:
          decoded.position_type = "WIFI timeout";
          for (i = 0; i < 6; i++) {
            decoded["v_bat" + (i + 1)] = mt_value_decode(bytes[5 + i], 2.8, 4.2, 8, 2);
          }
          break;

        case 4:
          decoded.position_type = "WIFI failure";
          for (i = 0; i < 6; i++) {
            // Most of time a WIFI timeout occurs due to a low battery condition
            decoded["v_bat" + (i + 1)] = mt_value_decode(bytes[5 + i], 2.8, 4.2, 8, 2);
          }
          decoded.error = message(bytes[11], ["WIFI connection failure", "Scan failure",
            "Antenna unavailable", "WIFI not supported on this device"]);
          break;

        case 5:
        case 6:
          decoded.position_type = "LP-GPS data";
          // Encrypted; not described in the documentation
          decoded.error = message(0, ["UNSUPPORTED POSITION TYPE " + decoded.data]);
          break;

        case 7:
          decoded.position_type = "BLE beacon scan";
          decoded.age = mt_value_decode(bytes[5], 0, 2040, 8, 0);
          // Remaining data: up to 4 beacons
          decoded.beacons = mac_rssi(bytes.slice(6));
          break;

        case 8:
          decoded.position_type = "BLE beacon failure";
          decoded.error = message(bytes[5], ["BLE is not responding", "Internal error", "Shared antenna not available",
            "Scan already on going", "No beacon detected", "Hardware incompatibility"]);
          break;

        // Test with: 0358D895090EC46E1FF44B9EB76466B3B87454AD500959CA1ED4AD525E67DA14A1AC
        // or 032CD1890900C46E1FF44B9EC5C83A355A3898A6
        case 9:
          decoded.position_type = "WIFI BSSIDs";
          decoded.age = mt_value_decode(bytes[5], 0, 2040, 8, 0);
          // Remaining data: up to 4 WiFi BSSIDs
          decoded.stations = mac_rssi(bytes.slice(6));
          break;

        default:
          decoded.error = message(0, ["UNSUPPORTED POSITION TYPE " + decoded.data]);
      }
      break;

    case 0x04:
      decoded.type = "ENERGY STATUS";
      break;

    case 0x05:
      decoded.type = "HEARTBEAT";
      decoded.mcuFirmware = bytes[6] + "." + bytes[7] + "." + bytes[8];
      decoded.bleFirmware = bytes[9] + "." + bytes[10] + "." + bytes[11];

      break;

    case 0x07:
      // Activity status message and configuration message share the same identifier
      var tag = bytes[5];
      switch (tag) {
        case 1:
          decoded.type = "ACTIVITY STATUS";
          decoded.activity_counter = uint32(bytes.slice(6, 10));
          break;

        case 2:
          decoded.type = "CONFIGURATION";
          for (i = 0; i < 5; i++) {
            var offset = 6 + 5 * i;
            decoded["param" + i] = {
            type: bytes[offset],
            value: uint32(bytes.slice(offset + 1, offset + 5))
            };
          }
            break;

        case 3:
          decoded.type = " SHOCK DETECTION";
            decoded.axisX = "fwVersion: " + bytes[7] + "." + bytes[8];
            decoded.axisY = "fwVersion: " + bytes[9] + "." + bytes[10];
            decoded.axisX = "fwVersion: " + bytes[11] + "." + bytes[12];
            //Byte 7-8 - X Axis
            //Byte 9-10 - Y Axis
            //Byte 11-12 - Z Axis
            break;

        default:
          decoded.error = message(0, ["UNSUPPORTED POSITION TYPE " + decoded.data + "/" + tag]);
      }
      break;
   
  

    case 0x09:
      decoded.type = "SHUTDOWN";
      break;

    case 0xFF:
      decoded.type = "DEBUG";
      break;

    default:
      decoded.error = message(0, ["UNSUPPORTED MESSAGE TYPE " + type]);
  }

  // Just some redundant debug info
  decoded.debug = {
    payload: hex(bytes),
    length: bytes.length,
    port: port,
    server_time: new Date().toISOString()
  };

  return decoded;
}