This topic can be used for comments and questions related to the guide for deploying The Things Network’s routing services in a local or private environment using docker-compose.
Any discussions not specifically related to Docker or Docker-Compose deployments should go here.
One question, what happens with the auth-tokens in the configuration files for router, broker and handler? Aren’t they needed anymore? Should they be removed from those configuration files?
Before we can proceed, we first have to create the ttn.handler exchange in RabbitMQ. To do so, go to localhost:15672 and on the Exchanges page, create the ttn.handler exchange of type topic.
What is the user and password to access this service?
Thanks!
They are still needed to authenticate to the Discovery server. In this guide we only add or change config, we don’t remove anything.
I have tried again but still having this error, something wrong with the tokens?
> networkserver_1 | WARN Could not connect to Redis. Retrying... error=dial tcp [::1]:6379: getsockopt: connection refused
> handler_1 | WARN Could not connect to Redis. Retrying... error=dial tcp [::1]:6379: getsockopt: connection refused
> router_1 | DEBUG Could not connect to gRPC server, reconnecting... Address=discovery.mynetwork.local:1900 error=dial tcp 172.20.0.4:1900: getsockopt: connection refused
> broker_1 | DEBUG Could not connect to gRPC server, reconnecting... Address=discovery:1900 error=dial tcp 172.20.0.4:1900: getsockopt: connection refused
> discovery_1 | INFO ttn: Got public keys for token validation
> discovery_1 | DEBUG Connected to gRPC server Address=localhost:1900
> discovery_1 | DEBUG transport: http2Server.HandleStreams received bogus greeting from client: "\x16\x03\x01\x00\x9e\x01\x00\x00\x9a\x03\x03-\x8c}\x16E\x9d\xd3\x03\x03\x98*P\x92"
> discovery_1 | WARN Could not connect to gRPC server with TLS, reconnecting without it... Address=localhost:1900 error=tls: first record does not look like a TLS handshake
> discovery_1 | DEBUG Connected to gRPC server Address=localhost:1900
> networkserver_1 | WARN Could not connect to Redis. Retrying... error=dial tcp [::1]:6379: getsockopt: connection refused
> handler_1 | WARN Could not connect to Redis. Retrying... error=dial tcp [::1]:6379: getsockopt: connection refused
> networkserver_1 | WARN Could not connect to Redis. Retrying... error=dial tcp [::1]:6379: getsockopt: connection refused
> discovery_1 | DEBUG transport: http2Server.HandleStreams received bogus greeting from client: "\x16\x03\x01\x00\xb7\x01\x00\x00\xb3\x03\x03\x12\x9d'\xc2\xee\xcbhPЍ\xba\xad\x1a"
> handler_1 | DEBUG Connected to gRPC server Address=discovery.mynetwork.local:1900
> handler_1 | WARN Could not connect to gRPC server with TLS, reconnecting without it... Address=discovery.mynetwork.local:1900 error=tls: first record does not look like a TLS handshake
> handler_1 | DEBUG Connected to gRPC server Address=discovery.mynetwork.local:1900
> handler_1 | WARN MQTT is not enabled in your configuration
> handler_1 | WARN AMQP is not enabled in your configuration
> discovery_1 | DEBUG Handled request with error CallerID=handler CallerIP=172.20.0.7:46300 Duration=624.738µs ErrCode=PermissionDenied Method=/discovery.Discovery/Announce error=rpc error: code = 7 desc = permission denied: unable to parse token: crypto/ecdsa: verification error
> handler_1 | INFO ttn: Got public keys for token validation
> handler_1 | FATAL Could not initialize handler error=Failed to announce this component to TTN discovery: rpc error: code = 7 desc = permission denied: unable to parse token: crypto/ecdsa: verification error: permission denied: unable to parse token: crypto/ecdsa: verification error
> ttnbackbone_handler_1 exited with code 1
Other question, I see that discovery creates two certificates, do the have to be both mounted by the docker ?
Why does my discovery and networserver look for public keys in local host instead of in the IP of the thing network when using docker?
discovery_1 | WARN ttn: Failed to refresh public keys for token validation: Get https://account.thethingsnetwork.org/key: dial tcp: lookup account.thethingsnetwork.org on 127.0.0.11:53: read udp 127.0.0.1:54390->127.0.0.11:53: i/o timeout
networkserver_1 | WARN ttn: Failed to refresh public keys for token validation: Get https://account.thethingsnetwork.org/key: dial tcp: lookup account.thethingsnetwork.org on 127.0.0.11:53: read udp 127.0.0.1:44876->127.0.0.11:53: i/o timeout
handler_1 | WARN ttn: Failed to refresh public keys for token validation: Get https://account.thethingsnetwork.org/key: dial tcp: lookup account.thethingsnetwork.org on 127.0.0.11:53: read udp 127.0.0.1:58025->127.0.0.11:53: i/o timeout
Port 53, that’s a DNS lookup.
What do you mean, that my docker dns is wrong?
VirtualBox:~/TTNBackBone/host$ cat /etc/default/docker
# Docker Upstart and SysVinit configuration file
#
# THIS FILE DOES NOT APPLY TO SYSTEMD
#
# Please see the documentation for "systemd drop-ins":
# https://docs.docker.com/engine/articles/systemd/
#
# Customize location of Docker binary (especially for development testing).
#DOCKERD="/usr/local/bin/dockerd"
# Use DOCKER_OPTS to modify the daemon startup options.
DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4"
#DOCKER_OPTS="--insecure-registry discover.thethingsnetwork.org:1900"
# If you need Docker to use an HTTP proxy, it can also be specified here.
#export http_proxy="http://127.0.0.1:3128/"
# This is also a handy place to tweak where Docker's temporary files go.
#export TMPDIR="/mnt/bigdrive/docker-tmp"
and my hosts
> VirtualBox:~/TTNBackBone/host$ cat /etc/hosts
> 127.0.0.1 localhost
> 127.0.1.1 lorabackbone-VirtualBox
> 127.0.0.1 router handler discovery broker networkserver rabbitmq redis
> # The following lines are desirable for IPv6 capable hosts
> ::1 ip6-localhost ip6-loopback
> fe00::0 ip6-localnet
> ff00::0 ip6-mcastprefix
> ff02::1 ip6-allnodes
> ff02::2 ip6-allrouters
I am wondering, in private backend with docker (or without it), If I am using node-red, how can I authorize my nodes in the handler?
I see that the guide explains it using the access key, but, how do I do generate such as Access key in a private network?
Regards!
Using ttnctl 
hi @johan
Thanks for replying, I have used it but it hangs while trying to connect to my private handler, apparently it does a dns request but the dns server doesn´t know the name of my private-handler, or where to look for it. What could I do?
Furthermore, how can I use the ttnctl to generate an Access key?
Thank you!
Keep looking into the issue with the handler, I see that there are some issues with localhost and the broker apparently, why does the broker uses local host instead of its docker ip??
> Discovery_1 | DEBUG Handled request CallerID=artGuard-handler CallerIP=172.18.0.6:56794 Duration=210.561µs Method=/discovery.Discovery/Get
> handler_1 | DEBUG Could not connect to gRPC server, reconnecting... Address=broker.artGuard.local.thethings.network:1902 error=dial tcp 172.18.0.8:1902: getsockopt: connection refused
> handler_1 | DEBUG Could not connect to gRPC server, reconnecting... Address=broker.artGuard.local.thethings.network:1902 error=dial tcp 172.18.0.8:1902: getsockopt: connection refused
> handler_1 | DEBUG Could not connect to gRPC server, reconnecting... Address=broker.artGuard.local.thethings.network:1902 error=dial tcp 172.18.0.8:1902: getsockopt: connection refused
> broker_1 | DEBUG Could not connect to gRPC server, reconnecting... Address=localhost:1903 error=dial tcp [::1]:1903: getsockopt: connection refusedHello all,
I reply to my post reagarding to the error that the pirvate docker handler seems to get stack while registering, adding devices or trying to retrieve device list.
After a lot of debugging and attemps ( also understanding the architecture did help) , the issue relies on the fact that of missing configuration for the communication between the broker and the networkserver. The error
> Discovery_1 | DEBUG Handled request CallerID=artGuard-handler CallerIP=172.18.0.6:56794 Duration=210.561µs Method=/discovery.Discovery/Get
> handler_1 | DEBUG Could not connect to gRPC server, reconnecting... Address=broker.local.thethings.network:1902 error=dial tcp 172.18.0.8:1902: getsockopt: connection refused
> handler_1 | DEBUG Could not connect to gRPC server, reconnecting... Address=broker.local.thethings.network:1902 error=dial tcp 172.18.0.8:1902: getsockopt: connection refused
> handler_1 | DEBUG Could not connect to gRPC server, reconnecting... Address=broker.local.thethings.network:1902 error=dial tcp 172.18.0.8:1902: getsockopt: connection refused
> broker_1 | DEBUG Could not connect to gRPC server, reconnecting... Address=localhost:1903 error=dial tcp [::1]:1903: getsockopt: connection refused
is only displayed if the debug option for each service (handler discovery broker) is enabled, here below only the option enabled for the handler
handler:
image: thethingsnetwork/ttn:latest
command: handler --config /etc/ttn/handler/artGuardConfig.yml
depends_on:
- discovery
- redis
- rabbitmq
environment:
- TTN_DEBUG=true
networks:
default:
aliases:
- handler.artGuard.local
ports:
- "1904:1904"
- "8084:8084"
volumes:
- "/host/handler:/etc/ttn/handler"
The missing entry is on the broker/config.yml and it is the network server address, otherwise the broker cannot connect to it and therefore it falls back to the localhost, which refuses the connection. Thus the boker config should contain such as entry and the port of the broker, I think @htdvisser forgot to include it, although it’s briefdly mentioned in his private route with docker section changing the hosts, it is does not appear on the private route broker section, so I think that should be changed/added.
~$cat config.yml id: broker tls: true key-dir: /etc/ttn/broker/ auth-servers: ttn-account-v2: "https://account.thethingsnetwork.org" discovery-address: "discovery.local:1900" auth-token: YOUR TOKEN broker: server-address-announce: broker.artGuard.local networkserver-address: "networkserver.local:1903" networkserver-cert: /etc/ttn/broker/networkserver.cert networkserver-token: YOUR TOKEN
Likewise, there is missing an entry in the networkserver yml config, since its own advertise address is not present and it should be. This is missing as well for the private route guide
~$cat config.yml
id: networkserver
tls: true
#key-dir: /home/lorabackbone/TTNBackBone/host/networkserver/
key-dir: /etc/ttn/networkserver/
auth-servers:
ttn-account-v2: "https://account.thethingsnetwork.org"
networkserver:
redis-address: redis:6379
server-address-announce: networkserver.local
Adding those too elements made my private handler back operative and responsive ( the error is normal since that app wasn’t registered yet)
VirtualBox:~/TTNBackBone$ ./ttnctl-linux-amd64 devices list --config ./host/ttnctl/artGuardConfig.yml --data ./host/ttnctl/
INFO Using Application AppEUI=AAAAAAAAAAAA AppID=ab
INFO Discovering Handler... Handler=handler
INFO Connecting with Handler... Handler=handler.local:1904
FATAL Could not get devices. error=Could not get devices for application from Handler: Application not registered to this Handler: handler:application:ag not found
Looking into the logs of the backend I saw
discovery_1 | DEBUG Handled request CallerID=ttnctl-l@l-VirtualBox CallerIP=172.18.0.1:58502 Duration=334.669µs Method=/discovery.Discovery/Get
handler_1 | DEBUG Handled request with error CallerID=client-l@l-VirtualBox CallerIP=172.18.0.1:42358 Duration=706.174µs ErrCode=NotFound Method=/handler.ApplicationManager/GetDevicesForApplication error=rpc error: code = 5 desc = Application not registered to this Handler: handler:application:ab not found
Regards and enjoy your private docker ttn backend 
Hi,
just want to add we have our TTN V2 back-end now up and running, thanks to this excellent article and its predecessor. We can OTAA some of our devices okay (still working on the LoPys but that’s another issue for another time), and get messages into our back-end, but we cannot figure out what we should be configuring with regard to the node.js client in order to subscribe successfully to messages in order to consume them.
What do we need to set for the region parameter for example, its can’t be “eu” as that maps on to TTN’s own infrastructure(?).
var region = ‘eu’;
var appId = ‘hello-world’;
var accessKey = ‘2Z+MU0T5xZCaqsD0bPqOhzA6iygGFoi4FAgMFgBfXSo=’;
var client = new ttn.Client(region, appId, accessKey);
Note we are not running our own account registration but using TTN’s account server, but that is the only component of TTN services that we are making use of…
[this gives us our registered application and the access key, but all our devices are registered within our back-end only, not in TTN.]
I’m guessing this is part of our issue, but can’t figure out what this implies regards subscribing to our private handler to get data back out of it.
Do we also need to register our private handler with TTN as well? - along the lines of this article:
https://www.thethingsnetwork.org/article/setting-up-a-private-handler-connected-to-the-public-community-network
Any pointers greatly appreciated.
[TTN V1 was much simpler to set-up in our experience].
In the Node.JS client you can either use the region (it will then add thethings.network) or use tho hostname of IP address of your MQTT broker. See the API Reference for more information.
You don’t have to register your private handler if it’s part of your private network. If your private handler is connected to the public community network, you need to register it.
Indeed, v1 was easier to set up, but I’m sure you can agree that the greatly improved security is worth the extra effort 
Thanks for the reply.
We are still stuck unfortunately, and clearly we have a lack of knowledge which is not helping - but we are trying to get up to speed on how all the internals are working here.
We can connect to our localhost MQTT broker, with our own userid and password (i.e. that which we used to create the ttn.handler exchange), through the TTN clientjs, but this does not deliver us any application messages at all - which is not surprising since we’d have thought you need a specific application level password to get an application’s data out of our private TTN instance. If we try to use localhost as region parameter together with the application id and application password generated through TTN’s own account handler (and which we can view on the TTN public console), then we simply get no connection, i.e. bad username and password message is generated.
So, the bit that is missing in our knowledge includes, how can nodejs client be connecting directly to the internal MQTT broker endpoint and for us to expect it knows anything about the TTN application layer, since its not a TTN router/broker/handler and only those things know how to deal with the discovery service, account ids etc…
Apologies for our lack of knowledge at this time - we clearly have a big gap in our understanding.
How have other people been able to get the messages out of their private TTN version 2 deployment?
Thanks again to all for any help on this matter.
Hello.
First of all, thanks for the guide. It was extremely clear and useful in setting up my environment.
Speaking of, I’m running an Ubuntu Server VM on a Windows 10 host. My TTN Docker environment is all set up and properly running inside the VM, but now I need to make it receive data from outside. How can I find out the what IP is being used by the backend I deployed on Docker?
Thanks for all the help.
Just updating - have tried all possible combos of region=localhost, AppID and AccessKey based on what is generated in the TTN console (base64-ing the AccessKey as well), but to no avail, we simply can’t get authenticated to connect to our mqtt broker…
What are we missing?
Using ttnctl subscribe we see the OTAA activations going through correctly, we see the data packets entering our back-end through the debug logging on all the separate components, so as far as we can tell everything else is just working fine…
Okay, we solved our problem in the end.
The TTN node.js client code doesn’t quite work in our private context.
We have created a separate MQTT user account, assigned it full permissions, and use this as the connection for user/password to our MQTT broker. We maintain the internal AppID as the TTN assigned name for the application.
We have tweaked the internals of the TTN node.js client code to reflect this, and with region set to the IP of our MQTT broker, can now connect and retrieve the activation and device messages etc.
We also now have the LoPys successfully OTAA-ing and sending data over our private LoRaWAN along with our Waspmotes.
Hello
I m trying to set up a private lora server that comunicates with my end devices through my gateway.
I followed the steps from this excellent guide from github to set up the ttn backend on my private server.
Then i downloaded and compiled the ttn version of lora gateway bridge and point my gateway to my server (port 1700) . My bridge runs properly and communicates with my gateway:
INFO[9147] gateway: sending udp packet to gateway addr=94.70.239.218:47633protocol_version=1 type=PushACK
INFO[9154] gateway: received udp packet from gateway addr=94.70.239.218:37592 protocol_version=1 type=PullData
INFO[9154] gateway: sending udp packet to gateway addr=94.70.239.218:37592 protocol_version=1 type=PullACK
INFO[9156] gateway: received udp packet from gateway addr=94.70.239.218:47633 protocol_version=1 type=PushData
INFO[9156] gateway: rxpk packet received addr=94.70.239.218:47633 data=“QAdjXd8A8QAB0KCwDBvVFxQ=” mac=b827ebfffe13bcbf
INFO[9156] gateway: sending udp packet to gateway addr=94.70.239.218:47633 protocol_version=1 type=PushACK
INFO[9160] gateway: received udp packet from gateway addr=94.70.239.218:47633 protocol_version=1 type=PushData
INFO[9160] gateway: stat packet received addr=94.70.239.218:47633 mac=b827ebfffe13bcbf
INFO[9160] gateway: sending udp packet to gateway addr=94.70.239.218:47633 protocol_version=1 type=PushACK
INFO[9164] gateway: received udp packet from gateway addr=94.70.239.218:37592 protocol_version=1 type=PullData
INFO[9164] gateway: sending udp packet to gateway addr=94.70.239.218:37592 protocol_version=1 type=PullACK
INFO[9171] gateway: received udp packet from gateway addr=94.70.239.218:47633 protocol_version=1 type=PushData
INFO[9171] gateway: rxpk packet received addr=94.70.239.218:47633 data=“QAdjXd8A8gAB9LSvI/RsY/c=” mac=b827ebfffe13bcbf
INFO[9171] gateway: sending udp packet to gateway addr=94.70.239.218:47633 protocol_version=1 type=PushACK
INFO[9174] gateway: received udp packet from gateway addr=94.70.239.218:47633 protocol_version=1 type=PushData
INFO[9174] gateway: rxpk packet received addr=94.70.239.218:47633 data=“QEZqUy+AggQBniU8K7QE2Yz0MuK6mlkKQIc=” mac=b827ebfffe13bcbf
then i used ttnctl tool to register my end device.
When i run sudo docker-compose up i get this output :
router_1 | INFO Handled gateway status Duration=4.215µs GatewayID=eui-b827ebfffe13bcbf
router_1 | DEBUG Sending Status to monitor GatewayID=eui-b827ebfffe13bcbf Monitors=0
router_1 | INFO Handled uplink AppPayloadSize=4 CodingRate=4/5 Counter=250 DataRate=SF7BW125 DevAddr=XXXXX07 DownlinkOptions=2 Duration=414.956µs FCnt=250 Frequency=868100000 GatewayID=eui-b827ebfffe13bcbf MAC= Modulation=LORA NumBrokers=1 PayloadSize=17 Port=1 RSSI=-94 SNR=10.8
router_1 | DEBUG Sending UplinkMessage to monitor GatewayID=eui-b827ebfffe13bcbf Monitors=0
router_1 | DEBUG Sending UplinkMessage to broker Brokers=1
networkserver_1 | DEBUG Request started Auth-Type=tls+token CallerID=dev CallerIP=172.18.0.7:36494 DevAddr=XXXXX07 Method=/networkserver.NetworkServer/GetDevices ServiceName=broker ServiceVersion=v2.6.0-dev-8bedc07724acb72dc8a901311656444735578a4b (2017-05-01T22:16:30Z)
networkserver_1 | DEBUG Request completed Auth-Type=tls+token CallerID=dev CallerIP=172.18.0.7:36494 Code=OK DevAddr=XXXXX07 Duration=1.743018ms Method=/networkserver.NetworkServer/GetDevices ServiceName=broker ServiceVersion=v2.6.0-dev-8bedc07724acb72dc8a901311656444735578a4b (2017-05-01T22:16:30Z)
broker_1 | WARN Could not handle uplink AppPayloadSize=4 CodingRate=4/5 Counter=250 DataRate=SF7BW125 DevAddr=XXXXX07 DevAddrResults=1 Duplicates=1 FCnt=250 Frequency=868100000 GatewayID=eui-b827ebfffe13bcbf MAC= Modulation=LORA PayloadSize=17 Port=1 RSSI=-94 SNR=10.8 error=device that validates MIC not found
broker_1 | DEBUG Sending DeduplicatedUplinkMessage to monitor Monitors=0
router_1 | INFO Handled uplink AppPayloadSize=13 CodingRate=4/5 Counter=1159 DataRate=SF7BW125 DevAddr=XXXXX46 DownlinkOptions=2 Duration=302.147µs FCnt=1159 Frequency=868100000 GatewayID=eui-b827ebfffe13bcbf MAC=Adr Modulation=LORA NumBrokers=1 PayloadSize=26 Port=1 RSSI=-105 SNR=9.8
router_1 | DEBUG Sending UplinkMessage to monitor GatewayID=eui-b827ebfffe13bcbf Monitors=0
router_1 | DEBUG Sending UplinkMessage to broker Brokers=1
networkserver_1 | DEBUG Request started Auth-Type=tls+token CallerID=dev CallerIP=172.18.0.7:36494 DevAddr=XXXXX46 Method=/networkserver.NetworkServer/GetDevices ServiceName=broker ServiceVersion=v2.6.0-dev-8bedc07724acb72dc8a901311656444735578a4b (2017-05-01T22:16:30Z)
networkserver_1 | DEBUG Request completed Auth-Type=tls+token CallerID=dev CallerIP=172.18.0.7:36494 Code=OK DevAddr=XXXX46 Duration=2.328736ms Method=/networkserver.NetworkServer/GetDevices ServiceName=broker ServiceVersion=v2.6.0-dev-8bedc07724acb72dc8a901311656444735578a4b (2017-05-01T22:16:30Z)
broker_1 | WARN Could not handle uplink AppPayloadSize=13 CodingRate=4/5 Counter=1159 DataRate=SF7BW125 DevAddr=XXXXX46 DevAddrResults=1 Duplicates=1 FCnt=1159 Frequency=868100000 GatewayID=eui-b827ebfffe13bcbf MAC=Adr Modulation=LORA PayloadSize=26 Port=1 RSSI=-105 SNR=9.8 error=device that validates MIC not found
broker_1 | DEBUG Sending DeduplicatedUplinkMessage to monitor Monitors=0
router_1 | INFO Handled uplink AppPayloadSize=4 CodingRate=4/5 Counter=251 DataRate=SF7BW125 DevAddr=XXXXX07 DownlinkOptions=2 Duration=320.284µs FCnt=251 Frequency=868500000 GatewayID=eui-b827ebfffe13bcbf MAC= Modulation=LORA NumBrokers=1 PayloadSize=17 Port=1 RSSI=-95 SNR=8.5
As you can see my broker receive messages from my devices , but can not make the MIC validation with the handler (right?) .
When i run ttnctl subscribe i get this (nothing) :
/go/src/github.com/TheThingsNetwork/ttn$ ttnctl subscribe
INFO Connecting to MQTT… MQTT Broker=tcp://localhost:1883 Username=
INFO Connected to MQTT
INFO Subscribed to activations
INFO Subscribed to uplink
My goal is to run the backend lockaly .
Is there any idea what kind of configuration should i do ?
Thanks a lot in advance !
Well, finally there was no problem at all .
I had forgotten that the sketch that i had uploaded on my arduinos had these lines :
static const u4_t DEVADDR = 0xAF859F01 ;
static const PROGMEM u1_t NWKSKEY[16] = {MY NWKSKEY };
static const u1_t PROGMEM APPSKEY[16] = { MY APPSKEY };
so i registered my device using ttncl
ttnctl devices register mydevic
ttnctl devices set mydevice --dev-addr AF859F01
ttnctl devices set mydevice --nwk-s-key MY NWKSKEY
ttnctl devices set mydevice --app-s-key MY APPSKEY
ttnctl devices set mydevice --dev-eui 00000000AF859F01
now there is no problem whit the validation and my handler receives all the messeges.
I assume that i will be able to make the registration with the opposite way (first register my device on my server, persinalize my device , and use the generated nwkskey and appskey in my sketch on arduino)