Deploying a Private Routing Environment With Docker-Compose

AndreiL · May 28, 2017, 7:06am

Hello,

First of all thank you for the really robust guide.

I’m trying to setup the ttn and the gateway bridge on Ubuntu 16.04. Got Redis and Rabbit running and then I’ve followed it to copy-paste level and I seem to have gotten into some issues with the Broker. I’ve double checked and regened the tokens for both network server and discovery but there seems to be no difference. I’ll try to redo the setup from scratch but maybe someone had this before. I see the same error in @ndarkness log

Using config file: broker/ttn.yml
INFO Initializing The Things Network Auth Servers=map[ttn-account-v2:https://account.thethingsnetwork.org] ComponentID=mynetwork-broker Description= Discovery Server Address=localhost:1900 Monitors=map[]
INFO Initializing Broker Announce=localhost:1902 DeduplicationDelay=200 NetworkServer=localhost:1903 Server=0.0.0.0:1902
INFO Starting
INFO ttn: Got public keys for token validation
FATAL Could not initialize broker error=Failed to announce this component to TTN discovery: rpc error: code = PermissionDenied desc = permission denied: unable to parse token: crypto/ecdsa: verification error: permission denied: unable to parse token: crypto/ecdsa: verification error source=Wrapf: /go/src/github.com/TheThingsNetwork/ttn/utils/errors/errors.go:211

AndreiL · May 28, 2017, 7:09am

Going through the instructions again I believe I did not leave the discovery service running while setting up the other services. I’m confident that’s the issue. I’ll get back with news.

Not much luck - with the discovery started now the Broker fails to register the prefix:

./ttn broker register-prefix 26000000/20 --config ./broker/ttn.yml
Using config file: ./broker/ttn.yml
INFO Initializing The Things Network Auth Servers=map[ttn-account-v2:https://account.thethingsnetwork.org] ComponentID=mynetwork-broker Description= Discovery Server Address=localhost:1900 Monitors=map[]
ERROR Could not register prefix Prefix=26000000/20 error=rpc error: code = PermissionDenied desc = permission denied: unable to parse token: crypto/ecdsa: verification error

I have created the discovery token with this command:
./ttn discovery authorize broker mynetwork-router --config ./broker/ttn.yml
and the network server oe with this:
./ttn networkserver authorize mynetwork-broker --config ./networkserver/ttn.yml

Also starting the broker yields the same error as before.

mixpot · May 30, 2017, 8:00pm

Hi, I remember that i had similar errors when i first set up the backend. I dont remember exactly how i fixed this, but i remember that i had noticed that the ca.cert server.pub networkserver.cert in the .env files were not correct. So , go to the broker router networkserver and handler files check the certs and see if they are the same with those on github. If not, then coppy paste them.

mixpot · May 30, 2017, 8:07pm

I enjoy the ttn backend privately and everything works really awsome.
When I started to use it , i used : ttnctl user register name email
and i loged in with the code from my profile on ttn.
Is it possible to avoid this? and someone can login or register without the code from public ttn?

Thanks in advance!

htdvisser · May 31, 2017, 6:38am

Your installation uses the community account server. This allows users to log in with their TTN account. If you don’t want this, you can implement your own account server.

mixpot · June 7, 2017, 11:06pm

Hi,
I can’t find any way to implement my own acount server for now,so probably i will use the ttn acount.
I m developing an web interface where i manage my devices (subscribe and visualize using grafana).
I ve written some shell scripts for registering applications and devices automatically with ttnctl.
I noticed that when I run those scripts through php (on my page) the devices and applications aren’t registered on my private handler but on public ttn.Probably that happens because php run those scripts as root.
When i run the scripts on shell everything works fine.
When I run them as super user i cant register applications to my handler and i get this error :
INFO Discovering Handler… Handler=dev
FATAL Could not connect to Discovery server error=x509: certificate signed by unknown authority

Can i run ttnctl as root in my private backend?

Thanks in advance !

mixpot · June 12, 2017, 12:10pm

I was wrong. The scripts via php are executed as www-data user, not as root.
I made some progress and now i can register my apps and devices to my handler.
My scripts and my php pages are in /var/www/html/mike/
in this directry there is a file .ttnctl where the tokens ca.cert and app are placed in.
I created a configuration file .ttnctl.yml (in /var/www/html/mike/) but nothing changed, ttnct stil registered everything in public ttn.
Then i add these commands in every shell script :

export TTNCTL_MQTT_ADRESS=localhost:1883
export TTNCTL_DISCOVERY_ADDRESS=localhost:1900
export TTNCTL_AUTH_SERVER=https://account.thethingsnetwork.org
export TTNCTL_ROUTER_ID=dev
export TTNCTL_HANDLER_ID=dev
export TTNCTL_MQTT_ADRESS=localhost:1883

In this way my applications and devices are register in my private handler.
BUT there is still a problem. The mqtt-address is set as eu.thethings.network:1883 and not as localhost:1883. So the messages are comming to my backend but i can’t see them with mosquitto sub or with ttnctl subscribe.
Is there anyone who can help me configure ttnctl under www-data user ?

Thanks in advance!

htdvisser · June 20, 2017, 6:34am

You might want to correct the typo in your last export. Address is with two D’s

mixpot · June 20, 2017, 12:04pm

Thank you @htdvisser

mixpot · July 7, 2017, 9:40am

Hi everyone.

I noticed that a few hours ago something went wrong eith the system. I can’t see my messages anymore and I can’t figure out why.
I get this error when I run ttnctl devices list :

INFO Discovering Handler… Handler=dev
FATAL Could not find Handler error=discovery:announcement:handler:dev not found**

In docker-compose I have these logs:

router_1         |  DEBUG No brokers to forward message to         AppPayloadSize=62 CodingRate=4/5 Counter=55392 DataRate=SF7BW125 DevAddr=26000E6A DownlinkOptions=2 FCnt=55392 Frequency=868100000 GatewayID=eui-b827ebfffe13bcbf MAC=Adr Modulation=LORA PayloadSize=75 Port=3 RSSI=-93 SNR=9.2
networkserver_1  |  DEBUG Stats                                    ComponentStats=&ComponentStats{Uptime:480,Cpu:&ComponentStats_CPUStats{User:0.46,System:0.04,Idle:0,Percentage:0,},Memory:&ComponentStats_MemoryStats{Memory:23359488,Swap:0,Heap:7970816,Stack:622592,},Goroutines:13,GcCpuFraction:0.00023151626,}
discovery_1      |  DEBUG Stats                                    ComponentStats=&ComponentStats{Uptime:480,Cpu:&ComponentStats_CPUStats{User:0.42,System:0.06,Idle:0,Percentage:0,},Memory:&ComponentStats_MemoryStats{Memory:24047616,Swap:0,Heap:8552448,Stack:622592,},Goroutines:18,GcCpuFraction:0.00036507938,}
broker_1         |  DEBUG Stats                                    ComponentStats=&ComponentStats{Uptime:480,Cpu:&ComponentStats_CPUStats{User:0.45,System:0.04,Idle:0,Percentage:0.09999751,},Memory:&ComponentStats_MemoryStats{Memory:24031232,Swap:0,Heap:8437760,Stack:720896,},Goroutines:22,GcCpuFraction:0.0002757559,}
router_1         |  DEBUG Stats                                    ComponentStats=&ComponentStats{Uptime:480,Cpu:&ComponentStats_CPUStats{User:0.49,System:0.1,Idle:0,Percentage:0.19999506,},Memory:&ComponentStats_MemoryStats{Memory:28135424,Swap:0,Heap:8232960,Stack:720896,},Goroutines:25,GcCpuFraction:0.0002257463,}
handler_1        |  DEBUG Stats                                    ComponentStats=&ComponentStats{Uptime:480,Cpu:&ComponentStats_CPUStats{User:0.46,System:0.06,Idle:0,Percentage:0,},Memory:&ComponentStats_MemoryStats{Memory:24309760,Swap:0,Heap:8511488,Stack:786432,},Goroutines:49,GcCpuFraction:0.00026360407,}

Any ideas?
Thanks in advance!

mixpot · October 16, 2017, 9:23pm

Hi,
the backend stopped working 2 months ago.
I m trying to fix the problem, but my broker is down.

I have these logs:

Starting ttn_broker_1
Attaching to ttn_broker_1
broker_1 | Using config file: ./.env/broker/dev.yml
broker_1 | INFO Initializing The Things Network Auth Servers=map[ttn-account-v2:https://account.thethingsnetwork.org] ComponentID=dev Description= Discovery Server Address=discovery:1900 Monitors=map
broker_1 | INFO Initializing Broker Announce=broker:1902 DeduplicationDelay=200 NetworkServer=networkserver:1903 Server=0.0.0.0:1902
broker_1 | INFO Starting
broker_1 | INFO ttn: Got public keys for token validation
broker_1 | DEBUG rpc-client: call done auth-type=token duration=3.535208ms id=dev method=/discovery.Discovery/Announce service-name=broker service-version=v2.8.0-dev-775f58d0bc86328750f614086b8b1153bfa3f51e (2017-07-05T10:39:10Z)
broker_1 | INFO ttn: Announced to TTN discovery
broker_1 | DEBUG rpc-client: call done auth-type=token duration=1.767494ms id=dev method=/discovery.Discovery/GetAll service-name=broker service-version=v2.8.0-dev-775f58d0bc86328750f614086b8b1153bfa3f51e (2017-07-05T10:39:10Z)
broker_1 | panic: failed to parse root certificate
broker_1 |
broker_1 | goroutine 1 [running]:
broker_1 | github.com/TheThingsNetwork/ttn/api.DialWithCert(0xc42000e061, 0x12, 0xc420403de0, 0x1d, 0x1, 0x1, 0x0)
broker_1 | /go/src/github.com/TheThingsNetwork/ttn/api/dial.go:39 +0x20b
broker_1 | github.com/TheThingsNetwork/ttn/core/broker.(*broker).Init(0xc4206adee0, 0xc4204300b0, 0x12, 0xc420403de0)
broker_1 | /go/src/github.com/TheThingsNetwork/ttn/core/broker/broker.go:127 +0x361
broker_1 | github.com/TheThingsNetwork/ttn/cmd.glob..func2(0x1591340, 0xc4206b0ac0, 0x0, 0x2)
broker_1 | /go/src/github.com/TheThingsNetwork/ttn/cmd/broker.go:60 +0x583
broker_1 | github.com/TheThingsNetwork/ttn/vendor/github.com/spf13/cobra.(*Command).execute(0x1591340, 0xc4206b0a80, 0x2, 0x2, 0x1591340, 0xc4206b0a80)
broker_1 | /go/src/github.com/TheThingsNetwork/ttn/vendor/github.com/spf13/cobra/command.go:651 +0x23a
broker_1 | github.com/TheThingsNetwork/ttn/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x1592220, 0x9, 0x0, 0xc4206a76a0)
broker_1 | /go/src/github.com/TheThingsNetwork/ttn/vendor/github.com/spf13/cobra/command.go:726 +0x339
broker_1 | github.com/TheThingsNetwork/ttn/vendor/github.com/spf13/cobra.(*Command).Execute(0x1592220, 0xc4206c5f48, 0xad0175)
broker_1 | /go/src/github.com/TheThingsNetwork/ttn/vendor/github.com/spf13/cobra/command.go:685 +0x2b
broker_1 | github.com/TheThingsNetwork/ttn/cmd.Execute()
broker_1 | /go/src/github.com/TheThingsNetwork/ttn/cmd/root.go:111 +0x31
broker_1 | main.main()
broker_1 | /go/src/github.com/TheThingsNetwork/ttn/main.go:23 +0x14a
ttn_broker_1 exited with code 2

any help?
Thanks in advance.

bouchezb · November 15, 2017, 9:39am

Hello,

I have deployed a private backend network following this article and i’m currently facing an issue i can’t understand.
All my components are up and seem to work well.

ttnctl applications list command works fine but when i try to run ttnctl devices list command i get this issue

INFO Using Application                        AppEUI=70B3D57ED00082CB AppID=devapp-001
  INFO Discovering Handler...                   Handler=my-handler
 DEBUG rpc-client: call done                    auth-type=token duration=891.481µs id=ttnctl-root@ubuntu-dev method=/discovery.Discovery/Get service-name=ttnctl service-version=v2.9.0-dev-0c455d496d53c849bed3cac3b794ae51fa514e19 (2017-10-11T11:10:16Z)
  INFO Connecting with Handler...               Handler=my-handler:1904
 DEBUG grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: authentication handshake failed: EOF"; Reconnecting to {my-handler:1904 <nil>}
 DEBUG grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: authentication handshake failed: EOF"; Reconnecting to {my-handler:1904 <nil>}
 DEBUG grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: authentication handshake failed: EOF"; Reconnecting to {my-handler:1904 <nil>}
 DEBUG grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: authentication handshake failed: EOF"; Reconnecting to {my-handler:1904 <nil>}

I also added a line in my /etc/hosts to bind my-handler on 127.0.0.1

To me it looks like a certificate issue.
The problem is that i can’t find which certificate i need to configure.
Is it a certificate issue between network components or between ttnctl and the network ?

Thanks for your help.
Regards

bouchezb · November 15, 2017, 3:22pm

Hi @mixpot,

Have you found a solution ?
I had a similar issue with my bridge when router wasn’t annouced yet to TTN discovery.

Do you have a docker logs handler_1 like this

  INFO Starting
  INFO ttn: Announced to TTN discovery
  INFO mqtt: connected
  INFO Connected to AMQP

Also you should notice that docker-compose will start all components without waiting for announcement to discovery, even if you add a depends_on directive.
To avoid this i used a restart_policy in my docker-compose file for the bridge and the handler
like this one

# Needs broker/router to have announced himself to the discovery server. 
# Will exit if not so we got to restart
deploy:
  restart_policy:   
    condition: any
    delay: 60s
    max_attempts: 3

This will give you 3 retries to start these components when they fail at startup because of annoucements delays.

Hope this helps.
Regards.

bouchezb · November 16, 2017, 10:49am

Today i have different error messages with --allow-insecure flag

DEBUG grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: authentication handshake failed: EOF"; Reconnecting to {my-handler:1904 <nil>}
 DEBUG grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: authentication handshake failed: read tcp 127.0.0.1:38870->127.0.0.1:1904: read: connection reset by peer"; Reconnecting to {my-handler:1904 <nil>}
 DEBUG grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: authentication handshake failed: read tcp 127.0.0.1:38874->127.0.0.1:1904: read: connection reset by peer"; Reconnecting to {my-handler:1904 <nil>}
 DEBUG grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: authentication handshake failed: read tcp 127.0.0.1:38878->127.0.0.1:1904: read: connection reset by peer"; Reconnecting to {my-handler:1904 <nil>}
 DEBUG grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: authentication handshake failed: read tcp 127.0.0.1:38884->127.0.0.1:1904: read: connection reset by peer"; Reconnecting to {my-handler:1904 <nil>}
 DEBUG grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: authentication handshake failed: read tcp 127.0.0.1:38890->127.0.0.1:1904: read: connection reset by peer"; Reconnecting to {my-handler:1904 <nil>}
 DEBUG grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: authentication handshake failed: read tcp 127.0.0.1:38896->127.0.0.1:1904: read: connection reset by peer"; Reconnecting to {my-handler:1904 <nil>}
 DEBUG grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: authentication handshake failed: read tcp 127.0.0.1:38900->127.0.0.1:1904: read: connection reset by peer"; Reconnecting to {my-handler:1904 <nil>}
 DEBUG grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: authentication handshake failed: EOF"; Reconnecting to {my-handler:1904 <nil>}

Reading source code, i found this in ttn/ttnctl/util/handler.go

dscConn, client := GetDiscovery(ctx)
defer dscConn.Close()
handlerAnnouncement, err := client.Get(GetContext(ctx), &discovery.GetRequest{
  ServiceName: "handler",
  ID:          viper.GetString("handler-id"),
})
...
ctx.WithField("Handler", handlerAnnouncement.NetAddress).Info("Connecting with Handler...")
hdlConn, err := handlerAnnouncement.Dial(nil)

I’m not really sure what handlerAnnoucement.Dial(nil) should do and why insecure fallback doesn’t seems to work.
I’m still stuck with this issue and cannot list devices.

htdvisser · November 20, 2017, 10:19am

I’m afraid I can’t think of a quick solution to your problem. Couple of pointers that may help you:

/etc/hosts often doesn’t work with Docker
Don’t use the “insecure” flag. It’s not secure
Check the server-address-announce in your configs and generate new certificates
Check ~/.ttnctl/ca.cert

bouchezb · November 20, 2017, 10:44am

Hello @htdvisser,

Thanks for your answer.

In the article it is said that

In order to use ttnctl with the services that are now running in Docker, you have to add the following to your /etc/hosts file:

127.0.0.1 discovery.local.thethings.network router.local.thethings.network handler.local.thethings.network

You can also update discovery-address and mqtt-address in your ttnctl.yml configuration (although this is not strictly necessary).

I’m a little bit confused about this point.

I’ll check all these points and of course i’ll not use insecure mode again
I just need a light on who’s doing currently failing handshake.
Which components are involved ?
In the first article it is said that

ttnctl also needs to have the discovery server certificate, for that we have to create a ca.cert file in $XDG_DATA_HOME/ttnctl, $XDG_CACHE_HOME/ttnctl or ~/.ttnctl/

This is OK to me but here we are talking about the handler.
So i’m confused again !

Thanks.

htdvisser · November 20, 2017, 12:40pm

Technically you don’t need to update your /etc/hosts because discovery.local.thethings.network actually resolves to 127.0.0.1 now

How the certificates are done:

The discovery server either has a self-signed certificate or a CA-signed certificate. In the article, we build a self-signed certificate.
The router/broker/networkserver/handler components have self-signed certificates, which are announced to the Discovery server
ttnctl connects to the discovery server, assuming the self-signed certificate is stored in ~/.ttnctl/ca.cert (or the other locations). Then it requests the self-signed certificate of the handler (in this case) from the discovery server and uses that to connect.

bouchezb · November 22, 2017, 8:31am

Hi,

I did some tests.
When i update my/etc/hosts/ i have some certificate issues with handshake errors.
When i delete the entry in my/etc/hosts/ i have some dial issues. Like if the containers couldn’t talk to each others.
I thought that all containers where started on the same default network with docker compose.
I didn’t found a solution yet - i’m a docker-compose noob.

So i decided to delete docker-compose from the equation and to do a restart from my configuration files/certs and call TTN commands directly on my server.
All components seems to start well but i can still see some warns about account server connection.

WARN ttn: Failed to refresh public keys for token validation: Get https://account.thethingsnetwork.org/key: proxyconnect tcp: dial tcp 10.5.2.10:0: i/o timeout

When i want to list my devices i get a fatal error

FATAL Could not get devices.                  
 error=Could not get devices for application from Handler: permission denied: unable to parse token: 
Get https://account.thethingsnetwork.org/key: proxyconnect tcp: dial tcp 10.5.2.10:0: i/o timeout source=Wrap: /go/src/github.com/TheThingsNetwork/ttn/utils/errors/errors.go:222

Where 10.5.2.10 is my corporate HTTP proxy - which needs authentication.
The proxy port should be 8080 but is 0, i don’t know if it is important.

So thanks for your help.
As usual HTTP proxies are painfull. I’ll test my configuration on another machine which is not behind a proxy.

Regards.

bouchezb · November 22, 2017, 4:08pm

Hi again.

I’m kind of stubborn, so i challenged my self to deal with that corporate HTTP proxy …
I’m making some progress.

For the record, I had this issue

FATAL Could not get devices.                  
 error=Could not get devices for application from Handler: permission denied: unable to parse token: 
Get https://account.thethingsnetwork.org/key: proxyconnect tcp: dial tcp 10.5.2.10:0: i/o timeout source=Wrap: /go/src/github.com/TheThingsNetwork/ttn/utils/errors/errors.go:222`

I found that my docker-compose environnment was not set properly.
So i added the proxy conf in all possible files i could.

/etc/default/docker

# If you need Docker to use an HTTP proxy, it can also be specified here.
export http_proxy="http://user:pwd@host:port/"

Also in my docker compose file for all my services

environment:
      - TTN_DEBUG=True # May be helpfull
      - HTTP_PROXY=http://user:pwd@host:port/
      - HTTPS_PROXY=http://user:pwd@host:port/
      - NO_PROXY=discovery,router,broker,networkserver,bridge,handler #since they are supposed to start all on the same network

Now i get an x509 error, which is an improvment

INFO Using Application                        AppEUI=70B3D57ED00082CB AppID=devapp-001
INFO Discovering Handler...                   Handler=handler
DEBUG rpc-client: call done                   auth-type=token duration=5.147437ms id=ttnctl-root@ubuntu-dev method=/discovery.Discovery/Get service-name=ttnctl service-version=v2.9.0-dev-0c455d496d53c849bed3cac3b794ae51fa514e19 (2017-10-11T11:10:16Z)
INFO Connecting with Handler...               Handler=handler:1904
DEBUG rpc-client: call failed                 auth-type=token duration=139.815476ms error=rpc error: code = PermissionDenied desc = permission denied: unable to parse token: Get https://account.thethingsnetwork.org/key: x509: certificate signed by unknown authority id=client-root@ubuntu-dev method=/handler.ApplicationManager/GetDevicesForApplication
FATAL Could not get devices.                  error=Could not get devices for application from Handler: permission denied: unable to parse token: Get https://account.thethingsnetwork.org/key: x509: certificate signed by unknown authority source=Wrap: /go/src/github.com/TheThingsNetwork/ttn/utils/errors/errors.go:222

So i think i should bind (or add ?) all the CA certs from my host to my containers (like in this post about docker container ssl certificates).
I tried to mount volumes to CA certs paths but it doesn’t seems to work at all.
At this point I don’t even know if it’s the right thing to do.

    volumes:
      - "./handler:/etc/ttn/agrobox-001/handler"
      - /usr/share/ca-certificates/:/usr/share/ca-certificates/
      - /usr/local/share/ca-certificates/:/usr/local/share/ca-certificates/

On the other side ttnctl user login works fine.
Does anybody have any clue about this x509: certificate signed by unknown authority issue with ttnctl devices list ?

bouchezb · November 23, 2017, 3:47pm

In the handler log i found this error message

{
    "fields":{
      "auth-type":"tls,token",
      "caller-ip":"172.18.0.1:48706",
      "duration":172211195,
      "error":"permission denied: unable to parse token: Get https://account.thethingsnetwork.org/key: x509: certificate signed by unknown authority",
      "id":"client-root@ubuntu-dev",
      "method":"/handler.ApplicationManager/GetDevicesForApplication"
    },
    "level":"debug",
    "timestamp":"2017-11-22T15:53:21.66096357Z",
    "message":"rpc-server: call failed"
  }

I faced in the past a similar issue and needed to install my HTTP proxy CA cert on my host.
You can check if you are in the same configuration with this command

$ sudo proxytunnel -p {{proxyhost}}:{{proxyport}} -P {{user}}:{{password}} -d account.thethingsnetwork.org:443 -a 7000 --verbose & echo -n | openssl s_client -connect localhost:7000 -showcerts

Unfortunately having my proxy CA cert installed is not enough with Go.
So i ‘simply’ added an env directive in my docker-compose file

environment:
  - SSL_CERT_DIR=/usr/local/share/ca-certificates/

Now ttnctl devices list seems to work fine but ttnctl devices register doesn’t.

$ ttnctl devices list
> INFO Listed 0 devices                         AppID=devapp-003

$ ttnctl devices register test
> FATAL Could not register Device                error=Could not set device on Handler: permission denied: 
> Broker did not set device: permission denied: 
> unable to parse token: Get https://account.thethingsnetwork.org/key: 
> x509: certificate signed by unknown authority 
> source=Wrap: /go/src/github.com/TheThingsNetwork/ttn/utils/errors/errors.go:222