Server getting data from TTN (lots of "connections) with same counter)

A question here,
Gateway(mikrotik) and sensor(rak7200) side by side.
When trying to get some data via node.js…(see code)

const mqtt = require("mqtt");
const Datastore = require("nedb");
const db = new Datastore({ filename: "./tracker.db", autoload: true });
const opts = { 
  auth: { 
          username: "AAAAAA", 
          password: "ttn-account-v2.BBBBBB",
          clean: true,
          port: 1883,
  },
  topic: "CCCCCC/devices/DDDDDD/up",
  host: "mqtt://eu.thethings.network",
};

const client = mqtt.connect(opts.host, opts.auth);
client.on("connect", () => {
    client.subscribe(opts.topic);
});

client.on("message", (topic, msg) => {
    const stringBuf = msg && msg.toString("utf-8");
    const jsonMsg = JSON.parse(stringBuf);
    db.insert(jsonMsg, (err, doc) => {
	    if (err) console.log("error: " + err);
	    else console.log(`inserted [${doc.dev_id}] with counter [${doc.counter}.${doc.confirmed?"confirmed":"unconfirmed"}] `)
    });
    //console.log(jsonMsg);
});

everything is working properly, however I got plenty of entrys with same counter number!
The application was created asking for confirmation!

inserted [tracker01] with counter [1662.confirmed] 
inserted [tracker01] with counter [1662.confirmed] 
inserted [tracker01] with counter [1662.confirmed] 
inserted [tracker01] with counter [1662.confirmed] 
inserted [tracker01] with counter [1662.confirmed] 
inserted [tracker01] with counter [1662.confirmed] 
inserted [tracker01] with counter [1662.confirmed] 
inserted [tracker01] with counter [1662.confirmed] 
inserted [tracker01] with counter [1663.confirmed] 
inserted [tracker01] with counter [1663.confirmed] 
inserted [tracker01] with counter [1663.confirmed] 
inserted [tracker01] with counter [1663.confirmed] 
inserted [tracker01] with counter [1663.confirmed] 
inserted [tracker01] with counter [1664.confirmed] 
inserted [tracker01] with counter [1664.confirmed] 
inserted [tracker01] with counter [1664.confirmed] 
inserted [tracker01] with counter [1664.confirmed] 
inserted [tracker01] with counter [1664.confirmed] 
inserted [tracker01] with counter [1664.confirmed] 
inserted [tracker01] with counter [1664.confirmed] 
inserted [tracker01] with counter [1664.confirmed] 
inserted [tracker01] with counter [1665.confirmed] 
inserted [tracker01] with counter [1665.confirmed] 
inserted [tracker01] with counter [1665.confirmed] 
inserted [tracker01] with counter [1665.confirmed] 
inserted [tracker01] with counter [1665.confirmed] 
inserted [tracker01] with counter [1666.confirmed] 
inserted [tracker01] with counter [1666.confirmed] 
inserted [tracker01] with counter [1666.confirmed] 
inserted [tracker01] with counter [1666.confirmed] 
inserted [tracker01] with counter [1666.confirmed] 
inserted [tracker01] with counter [1666.confirmed] 
inserted [tracker01] with counter [1666.confirmed] 
inserted [tracker01] with counter [1666.confirmed] 

its supposed to be like this?
I need to keep (db write) only one of those (with the same counter)?
Anyway to minimize those connections?
regards,

full package example (using default RAK7200 decoder)

{
  app_id: 'test_gps_tracXXXX',
  dev_id: 'trackXXX',
  hardware_serial: 'AC1F09FFFXXXXXX',
  port: 8,
  counter: 1650,
  confirmed: true,
  payload_raw: 'CAIBeANxAAgAGAO2BYYAPABl/7CgXXXXXXXXXXX=',
  payload_fields: {
    acceleration_x: '0.008g',
    acceleration_y: '0.024g',
    acceleration_z: '0.95g',
    battery: '3.76V',
    gyroscope_x: '0.6°/s',
    gyroscope_y: '1.01°/s',
    gyroscope_z: '-0.67°/s',
    magnetometer_x: '43.5μT',
    magnetometer_y: '66.75μT',
    magnetometer_z: '-9.3μT'
  },
  metadata: {
    time: '2020-10-20T17:32:29.276946973Z',
    frequency: 918.2,
    modulation: 'LORA',
    data_rate: 'SF7BW125',
    airtime: 97536000,
    coding_rate: '4/5',
    gateways: [ [Object] ]
  }
}

That’s known to be a bad idea, it will at minimum cause false duplicate uplinks on multiple frequencies due to front end overload, and may also cause corruption of even the packet on the correct frequency

inserted [tracker01] with counter [1662.confirmed]

None of this data is coming from your actual JSON message received, but only from some mysterious document variables. You should log out something based on the actual received message instead, or as well

The application was created asking for confirmation!

You should not do that.

First, if confirmed traffic worked, it would burn through your device’s downlink allowance in no time.

But the protocol of confirmed uplink doesn’t really work very well anyway, due to design defects - if it misses the ack, it will never get another until it gives up on retrying to send that data item and sends a new one instead.

@cslorabox thanks for your promptly response!

I forgot to say I am in testing here discovering the possibilities!!
learning things here…

  • keep devices far from each other (done)
  • create applications without confirmation (anyway to disable confirmation within the app?)

but on the second topic you say “None of this data is coming from your actual JSON message received”, that one i didint got it! its everything I got from the ttn!! Whats/Where is the “actual received message”?

Uplink confirmation is a behavior of your node’s firmware. Don’t set that.

Downlink confirmation is a behavior of your downlink request, don’t set that either.

“None of this data is coming from your actual JSON message received”, that one i didint got it! its everything I got from the ttn!!

No, your log messages only logs document variables, it doesn’t log anything it actually got from the method call.

console.log(inserted [${doc.dev_id}] with counter [${doc.counter}.${doc.confirmed?"confirmed":"unconfirmed"}] )

None of this comes from the

client.on(“message”, (topic, msg)

So as a result you just see the same thing printed over an over each time you get a message, not but reflecting any details of that message.

@cslorabox, I dont know if I got you!
changing the script a little!!

const moment = require("moment-timezone");

const gps = (json) => {
  return json.payload_fields.latitude ? "with GPS" : "without GPS";
};

const time = (json) => {
  let time = json.metadata.gateways[0].time
    ? json.metadata.gateways[0].time	// gateway time
    : json.metadata.time;				// server time
  return moment	
    .utc(time)
    .tz("America/Fortaleza")
    .format("YYYY-MM-DD HH:mm:ss.SSS");
};

// and changing the mentioned line

 console.log(
    `inserted [${doc.dev_id}] with counter [${doc.counter}.${
      doc.confirmed ? "confirmed" : "unconfirmed"
    }] ${gps(jsonMsg)}, with time: ${time(jsonMsg)}`
 );

I got different timestamps on each message! I think They are not the same!
those messages are the output of the method call! isn’t it? If not, how to log the right messages?
You are right confirmation is device specific! I am gonna change it right now!
I forgot to mention I setup the device to send messages each 180 secs (20 samples/hour)
the jsonMsg came from the client.on(“message”…) event! and is the same as msg (after I parse it as json).
I really dont know What I am missing here!
output:

inserted [tracker01] with counter [1693.confirmed] with GPS, with time: 2020-10-20 18:06:24.554
inserted [tracker01] with counter [1693.confirmed] with GPS, with time: 2020-10-20 18:06:29.565
inserted [tracker01] with counter [1693.confirmed] with GPS, with time: 2020-10-20 18:06:34.739
inserted [tracker01] with counter [1693.confirmed] with GPS, with time: 2020-10-20 18:06:40.751
inserted [tracker01] with counter [1693.confirmed] with GPS, with time: 2020-10-20 18:06:45.090
inserted [tracker01] with counter [1693.confirmed] with GPS, with time: 2020-10-20 18:06:51.103
inserted [tracker01] with counter [1694.confirmed] with GPS, with time: 2020-10-20 18:10:44.433
inserted [tracker01] with counter [1694.confirmed] with GPS, with time: 2020-10-20 18:10:48.440
inserted [tracker01] with counter [1694.confirmed] with GPS, with time: 2020-10-20 18:10:54.190
inserted [tracker01] with counter [1694.confirmed] with GPS, with time: 2020-10-20 18:11:01.203
inserted [tracker01] with counter [1694.confirmed] with GPS, with time: 2020-10-20 18:11:08.448

That’s because, for confirmed uplinks, your device is repeating the very same uplink (with the very same counter value) until it receives a downlink with an acknowledgement, or until it gives up. And like already noted, apparently missing the first acknowledgement will never make it receive one for subsequent retries:

If you would print the is_retry attribute you would see that TTN has recognized it was a retry of an earlier message that was exactly the same. And along with that very same message you’re getting different details for things like the timestamp, the list of gateway(s) that received the retry, and the signal quality for each gateway.

Apparently your device has problems receiving downlinks, hence does not get the acknowledgement for confirmed uplinks, and assumes its uplink was not even received by any gateway.

I dont have access to the is_retry attribute (is it device specific?), but makes sense!

looking in the ttn console>data I noticed this!

image

its possible to verify that the first msg is confirmed followed by retry confirmed msgs!
thats the default behavior? Could you please point me in the direction to understand those type of messages?

Confirmed here, on the first message I dont have the is_retry attribute! followed by many of those! If the first message was confirmed why to retry confirmation on the followed ones?
As requested the printed json!!

{
  app_id: 'test_gps_tracker_01',
  dev_id: 'tracker01',
  hardware_serial: 'AC1F09FFFE00375A',
  port: 8,
  counter: 1747,
  confirmed: true,  <--- no is_rety attribute
  payload_raw:  'AYj/bqL6H3kAAtoIAgF1A3EAGAAqA7wFhgBOAGT/wAkCGdcKAhHBCwL9bA==',
  payload_fields: {
    acceleration_x: '0.024g',    
    acceleration_y: '0.042g',
    acceleration_z: '0.956g',
    altitude: '7.3m',
    battery: '3.73V',
    gyroscope_x: '0.78°/s',
    gyroscope_y: '1°/s',
    gyroscope_z: '-0.64°/s',
    latitude: '-3.7214°',
    longitude: '-38.5159°',
    magnetometer_x: '66.15μT',
    magnetometer_y: '45.45μT',
    magnetometer_z: '-6.6μT'
  },
  metadata: {
    time: '2020-10-21T01:26:14.33970477Z',
    frequency: 917.8,
    modulation: 'LORA',
    data_rate: 'SF7BW125',
    airtime: 112896000,
    coding_rate: '4/5',
    gateways: [ [Object] ]
  }
}
{
  app_id: 'test_gps_tracker_01',
  dev_id: 'tracker01',
  hardware_serial: 'AC1F09FFFE00375A',
  port: 8,
  counter: 1747,
  confirmed: true,  
  is_retry: true,
  payload_raw: 'AYj/bqL6H3kAAtoIAgF1A3EAGAAqA7wFhgBOAGT/wAkCGdcKAhHBCwL9bA==',
  payload_fields: {
    acceleration_x: '0.024g',
    acceleration_y: '0.042g',
    acceleration_z: '0.956g',
    altitude: '7.3m',
    battery: '3.73V',
    gyroscope_x: '0.78°/s',
    gyroscope_y: '1°/s',
    gyroscope_z: '-0.64°/s',
    latitude: '-3.7214°',
    longitude: '-38.5159°',
    magnetometer_x: '66.15μT',
    magnetometer_y: '45.45μT',
    magnetometer_z: '-6.6μT'
  },
  metadata: {
    time: '2020-10-21T01:26:18.350087725Z',
    frequency: 917.4,
    modulation: 'LORA',
    data_rate: 'SF7BW125',
    airtime: 112896000,
    coding_rate: '4/5',
    gateways: [ [Object] ]
  }
}
{
  app_id: 'test_gps_tracker_01',
  dev_id: 'tracker01',
  hardware_serial: 'AC1F09FFFE00375A',
  port: 8,
  counter: 1747,
  confirmed: true,
  is_retry: true,
  payload_raw: 'AYj/bqL6H3kAAtoIAgF1A3EAGAAqA7wFhgBOAGT/wAkCGdcKAhHBCwL9bA==',
  payload_fields: {
    acceleration_x: '0.024g',
    acceleration_y: '0.042g',
    acceleration_z: '0.956g',
    altitude: '7.3m',
    battery: '3.73V',
    gyroscope_x: '0.78°/s',
    gyroscope_y: '1°/s',
    gyroscope_z: '-0.64°/s',
    latitude: '-3.7214°',
    longitude: '-38.5159°',
    magnetometer_x: '66.15μT',
    magnetometer_y: '45.45μT',
    magnetometer_z: '-6.6μT'
  },
  metadata: {
    time: '2020-10-21T01:26:24.459928828Z',
    frequency: 917,
    modulation: 'LORA',
    data_rate: 'SF8BW125',
    airtime: 205312000,
    coding_rate: '4/5',
    gateways: [ [Object] ]
  }
}

This is exactly what happens when the ack of the first attempt is missed. No more ack will ever be generated for that message number, so transmission retries will continue until the node gives up and moves on to the next message number.

This colossal waste of airtime and battery resulting from the bad design of confirmed uplink in LoRaWAN is why you should not use confirmed uplink