This is rather a hint than a cookbook recipe, but if there is any interest I could add some more details.
I successfully set up the above “system”. In an nutshell and for dummies like me:
TTN sends its data to AWS IoT
In AWS IoT, a rule is set up to save incoming data to a AWS Timestream Database
This Timestream Database can then be used to create Dashboards in AWS Managed Grafana
The beauty of this solution (for me) is that you do not need to maintain any servers. You can basically set up the whole system without a single line of code (well, you need 2 SQL queries).
The thing is: AWS is incredibly intimidating… at least it was for me when I first started with it.
But in fact setting this system up is straightforward and takes maybe 30 mins.
So if anyone is interested in more details, feel free to answer to this thread. I also invite AWS experts (I am by no means one!) to participate.
This is the other setup I actually use as well.
The AWS IoT way is going to cost you money. I think we pay around 8 to 10 EUR per month.
I will write down a short how-to, it helps me too for a new deployment
OK, this is rather “quick and dirty”. Also, you should at least have some SQL and MQTT knowledge.
Hope it is useful for some.
To give you an idea of the costs: I currently have 5 devices sending a message every 20 minutes. This results in costs of around 2-3 EUR per month. I can’t tell exactly because I am using other AWS services. But as I understand the number of devices almost doesn’t matter since most of the cost is constant and doesn’t scale with the number of devices.
Create an AWS account and log in. Note that there are some free tier for AWS IoT Core (this is the service we are going to use).
AWS recommends 2 factors authentication as well as never to use root user access keys
I suggest creating a billing alarm to monitor your estimated AWS charges. Although this use-case won’t produce high costs, you can easily do so with other AWS services.
You need an application on TTN with at least one working device.
Follow the deployment guide to set up AWS IoT Integration in TTN. In this guide, you need to select an AWS region and click on the “Community” tab before clicking on the blue “Deploy for The Things Network” button. You can think of AWS regions as separate data centers, so pick one you like .
This is where you select your AWS region in AWS itself. Make sure this is always the same region when working with AWS.
After completing the deployment guide, go to AWS IoT Core service. The easiest way is to enter “iot core” in the search bar on the top left, you will see the IoT Core service pop up.
In the left navigation field, open the MQTT test client and subscribe to the # theme in order to monitor incoming messages from TTN. Make sure your messages from your TTN application are showing up here before going on.
Next you set up a Timestream database. Open Timestream service (again, easiest using the search bar). Select “create database”. Give your database a name, then click “create database”. Next, create a table. In the left navigaton field, select “Tables”, then “create table”. Choose your database in the drop-down menu, give your table a name. Under “data retention” you can specify for how long your data stays in memory and on magnetic store. Select “create table”.
Now create a rule that sends incoming messages from TTN to your Timestream database. Go to AWS IoT Core, "message routing ", “Rules”. Click on “create rule”. Give your rule a name (e.g. “ttn_to_timestream”). Next page, enter your SQL statement. You basically select attributes from your mqtt message and a mqtt topic. Those attributes are then stored into the Timestream database. The following is an example statement using attributes from an actual TTN MQTT message. Note how to set dots in the statement to access nested objects in your MQTT message. Also, your MQTT topic needs to be adjusted. With ‘lorawan/+/uplink’ you select all upllink messages from every device in your TTN application.
SELECT end_device_ids.dev_eui as DevEUI, uplink_message.decoded_payload.battery as vbatt, uplink_message.decoded_payload.distance as distanz_mm, settings.data_rate.lora.spreading_factor as SF FROM 'lorawan/+/uplink'
Next page, select your Rule action to be “Timestream table”. Select your Database and Table from the dropdown. You need to enter at least one dimension. I choose to take the DevEUI as a dimension. Note that the Dimension value needs to use the actual MQTT object (not the alias from the SQL statement you created before. Also you need to use some fancy curly brackets and a dollar sign
Under “IAM Role”, click on “create new role” and give it a name. After this, click “Next”,“Create” to finish your rule.
Your TTN messages are now stored in your Timestream database. To check, open Timestream service. Select your table. Under “Management tools” you will find a Query editor where you can use your SQL statements to check your data.
Just want to give you some additional informations on expected costs when using AWS.
Basically we use the following components of AWS:
AWS Iot Core: Costs seem to be consistently low with just a few cents per month.
AWS Timestream: We have seen an increase of costs with our use case to around USD 20/month in the lastest months. It is quite hard to find the cause for this. It seems that Timestream is charged per byte scanned in queries. This value was extremely high for us last month (up to 3 TB scanned), we are currently trying to get AWS to look into this. Beside that, you can lower your costs when choosing to move data quickly to magnetic store (you can choose this when setting up your Timestream database).
AWS Grafana: You pay per user, for us it was 9$/month per editor and 5$/month per viewer. So do not create users too enthusiastically.
The pricing system of AWS is really complex. Be careful, watch your daily costs regularly and set up a budget alarm.
So after quite a long time we have found the reason for the increase in costs related to Timestream. The reason was that we set up an alert in grafana which should notify us if some sensor values were off. This alert was evaluated every minute or so. Since AWS handles this as a query towards Timestream (and queries are billed with a minimum of 10MB per query…) this resulted in those high values.
One way to mitigate this would be to configure the alert to be evaluated at a lower rate, like every hour or so.
