Request gateway traffic over time period

Gateways
1

Hi,
I was recently experimenting with requesting the gateway status over the new API.
My goal is to see how many packets are transmitted via my organizations installed gateways.

When requesting the URL:
https://{ttn_server}/api/v3/gs/gateways/{gateway_id}/connection/stats
I can see metrics such as the “rxin, rxok and rxfw” that to my understanding resemble the stats I’m looking for.
However, I would currently have to keep requesting the metrics every ~30 seconds to check whether there was a new packet send over my gateway.
Is there another way to request the “gateway traffic” for a time period, say the total packets send over 1 day or 1 month?
Thanks in advance!

2

Simply request it now, note the the values and request it tomorrow, subtract the two from each other.

May I ask why are you concern if there were a new message over your gateway, use case?

3

Thanks for your answer.
The API returns the network traffic only since the last uplink message was received, so in my case I would have to request atleast every 2 minutes to not “miss” a status message.
The case you described where I can request it every day and subtract the values would be my go to.
Regarding the use case:
My organization invested into installing gateways around the city for public use and in the future would like to monitor whether these gateways are actually used and the investment was worth it.

4

That works within the same “session” however care needed as where the old V2 NOC used to keep a lifetime count of the traffic up and down (sadly sorely missed as great for GW planning!) for a GW under V3 it is now session specific - so e.g. power cycle the GW or is session is lost of has to restart, e.g. a TTIG checking every 24hrs and finding a new set of firmware/settings or doing a remote firmware update on others, e.g. using Balena, will reset the values to zero again… number may not match and you will need to handle potential rollover/reset to zero and start again.

1 Like
5

That was where the old V2 NOC was good as above, now if monitoring your own nodes (obviously wont work for handling annon 3rd party nodes) when you extract the data to your own database you can analyse the message metadata to see which GW’s handled the message and keep a count of where you see your GW being used…sadly doesnt account for total traffic though.

6

Thanks for the replies.
I noticed there is a list of gateways with every message received from our sensors, however our goal was really to monitor total usage and not just our own packets.
Guess I will have to use the workaround and request the traffic in close intervals, check the timestamps and add up the values to get an estimate of the total traffic.

7

And potentially find yourself disconnected - every API call to the TTS CE infrastructure comes at a price, a price we don’t pay and adding more servers to accommodate is becoming increasingly less feasible. So anything that queries very frequently has to be super essential.

So please before diving in, let’s all investigate all the potential and ask the TTI engineers for their view - particularly as it’s always nice to ask the people that have to solve the issues we create.

In the meanwhile, rather like footfall in a shopping centre, you could do worse than leave the console open and download the JSON for analysis - it won’t be single byte accurate but it will give you plenty of indication of traffic. And it may well be that this can be captured using other tools. What is effectively Server Side Events pushing data to a browser or Javascript app has far less impact than a database query and, bonus, gives you far more detail.

8

If you use the below API it is update ~6min, gives you a very good idea of how busy a gateway is.

IRC it is the average over the time period, most use cases this is sufficient

https://mapper.packetbroker.net/api/v2/gateways/netID=000013,tenantID=ttn,id='gareway id'
1 Like
9

And here we go, edited for brevity: https://eu1.cloud.thethings.network/api/v3/gs/gateways/descartes-m2m-tele-2/connection/stats

{
  "connected_at": "2022-09-14T11:35:19.407560827Z",
  "protocol": "udp",
  "last_status_received_at": "2022-09-30T09:46:37.818776459Z",
  "last_status": {
    "time": "2022-09-30T09:46:37Z",
    "versions": {
      "ttn-lw-gateway-server": "3.21.2-rc0-SNAPSHOT-43bc66caf"
    },


    "metrics": {
      "rxok": 1,
      "rxfw": 1,
      "ackr": 100,
      "txin": 0,
      "txok": 0,
      "rxin": 1
    }
  },


verses ...


  "last_uplink_received_at": "2022-09-30T09:46:08.306405002Z",
  "uplink_count": "27971",

  "last_downlink_received_at": "2022-09-29T20:57:38.157190434Z",
  "downlink_count": "479",

  "round_trip_times": {
    "min": "0.024917124s",
    "max": "0.024917124s",
    "median": "0.024917124s",
    "count": 1
  }
}

Those counters look very extra useful to me!

Doing this call on a 15 minute basis for 24 hours once a quarter to get some time of day traffic profiling - but not all your gateways - just one at a time, along with doing this call once every 24 hours so any restarts get latest figures would feel about right - but let’s ask @adrianmares what he thinks, as he’s the one that has to put money in the swear box every time we think of toasting the servers.

PS, I don’t think Adrian actually swears but I do suspect he is the Chuck Norris of Servers - he doesn’t need to type the command, the server just does it.

10

I understand that sending too many API requests might cause problems and really like the approach of requesting the data in intervals and kind of estimate the traffic between them. (assuming that the traffic for most sensors is evenly distributed over the day).
Just like in your example, I mostly get values of 0 and 1 for now which makes an interpolation less accurate, but I will see what I can make of it. Thanks for all the input!

11

What we need/would help is the reinstatement of the gw lifetime persistent message count per V2, rather than the session only count, that we were promised would be reinstated once V3 transition completed and settled, like much of the other community friendly options that no longer exist :frowning: :slight_smile: @adrianmares any chance?

1 Like
12

Scroll down in the box above and you get:

  "last_uplink_received_at": "2022-09-30T09:46:08.306405002Z",
  "uplink_count": "27971",

  "last_downlink_received_at": "2022-09-29T20:57:38.157190434Z",
  "downlink_count": "479",

LoRaWAN has many many moving parts - looking at everything is the trick to find the solution!

13

That’s true, but if I e.g. send the first request and get a response with “uplink_count: 100” and the second request with “uplink_count: 200” that just means that the traffic status was sent 100 times between my requests.
It does not contain any direct information about the number of packets sent in the meantime. In this example I would go about this something like this:
In 30% of the requests there was a packet send in this status message (i.e. rxok = 1), I missed 100 uplink messages, so I assume that there were around 30 packets sent between my requests.
It’s not as detailed as the old solution seemed to be, but it’s still better than having no information at all

14

Says nothing about status messages - it tells you you have had 100 uplinks!

15

Er, no, it means that 100 uplinks were sent which is what @Johan_Scheepers & @Jeff-UK was referencing in their first replies. @Jeff-UK wants more because he’s insatiable, he wants a field for lifetime count as well as these counters reset when your gateway restarts or updates.

16

Additionally, as above, please don’t run away to create a solution until we’ve had some input from TTI engineering - there may be something we’ve missed or someone else may have an idea. You’ve not had it before today, a couple of more days won’t hurt.

17

Sorry, then I just missinterpreted the uplink count.
We will discuss the options internally aswell, so it will take a while before we implement any of this anyways.

Edit: I’m blind, also figured out that those values are shown in the console by the up and down arrows. All good, maybe I don’t need to recall the API at all depending on how often the counters are reset.

18

TRy the link Nick supplied amending for your GW-ID… check again tomorrow - just dont try and grab data every few mins and for all your GW’s - servers wont like you!

Note the data you get back will depend on the device and packet forarder your are using…

e.g. This is a remotely deployed Solar powered RAK ODU that is running UDP/PF where next is a TTIG locally on Basic Station - in each case you see uplink data

|||
|---|---|
|connected_at|2022-09-30T01:54:49.260481829Z|
|protocol|udp|
|last_status_received_at|2022-09-30T11:45:04.113455121Z|
|last_status||
|time|2022-09-30T11:45:04Z|
|versions||
|ttn-lw-gateway-server|3.21.2-rc0-SNAPSHOT-43bc66caf|
|ip||
|0|185.69.XXX.XXX|
|metrics||
|txin|0|
|txok|0|
|rxin|3|
|rxok|2|
|rxfw|2|
|ackr|100|
|last_uplink_received_at|2022-09-30T11:45:07.374012020Z|
**|uplink_count|2950|**
|last_downlink_received_at|2022-09-30T11:35:33.754778949Z|
**|downlink_count|59|**
|round_trip_times||
|min|0.041130681s|
|max|0.104855442s|
|median|0.085358101s|
|count|3|
|sub_bands||
|0||
|min_frequency|863000000|
|max_frequency|865000000|
|downlink_utilization_limit|0.001|
|1||
|min_frequency|865000000|
|max_frequency|868000000|
|downlink_utilization_limit|0.01|
|downlink_utilization|0.0014654578|
|2||
|min_frequency|868000000|
|max_frequency|868600000|
|downlink_utilization_limit|0.01|
|downlink_utilization|0.0007327289|
|3||
|min_frequency|868700000|
|max_frequency|869200000|
|downlink_utilization_limit|0.001|
|4||
|min_frequency|869400000|
|max_frequency|869650000|
|downlink_utilization_limit|0.1|
|5||
|min_frequency|869700000|
|max_frequency|870000000|
|downlink_utilization_limit|0.01|

TTIG:

connected_at 2022-09-30T11:10:09.606516987Z
protocol ws
last_status_received_at 2022-09-30T11:10:09.639585891Z
last_status
time 2022-09-30T11:10:09.639458434Z
versions
package 2.0.4
platform minihub - Firmware 2.0.4 - Protocol 2
station 2.0.4(minihub/debug)
firmware 2.0.4
advanced
features rmtsh
model minihub
last_uplink_received_at 2022-09-30T11:59:33.931601521Z
uplink_count 216
last_downlink_received_at 2022-09-30T11:56:28.417722911Z
downlink_count 3
round_trip_times
min 0.031407882s
max 0.262213007s
median 0.034172995s
count 20
sub_bands
0
min_frequency 863000000
max_frequency 865000000
downlink_utilization_limit 0.001
1
min_frequency 865000000
max_frequency 868000000
downlink_utilization_limit 0.01
downlink_utilization 0.00039210668
2
min_frequency 868000000
max_frequency 868600000
downlink_utilization_limit 0.01
downlink_utilization 0.000019982223
3
min_frequency 868700000
max_frequency 869200000
downlink_utilization_limit 0.001
4
min_frequency 869400000
max_frequency 869650000
downlink_utilization_limit 0.1
5
min_frequency 869700000
max_frequency 870000000
downlink_utilization_limit 0.01
19

To understand GW utilisation and plan for any new deployments I used to have a routine where I would use console to look through the counts for my/my community collaborators & clients GW’s typically once per month, for busy units once per week for lightly used once per quarter and look at increments (manually but not much effort) if I saw high traffic I might plan for a new unit to densify…especially if seeing lots of downlnks - which render unit ‘deaf’ and impacts coverage… running a query such a Nicks (servers permiting) under similar regime may serve until I can get pesistent counts back on GW console pages :wink:

20

They are from the same source, as above, reset when your gateway resets / goes offline / updates.