Setting up a Private Routing Environment

htdvisser · December 14, 2016, 8:54pm

This topic can be used for comments and questions related to the guide for setting up The Things Network’s routing services in a local or private environment.

htdvisser · December 15, 2016, 12:45pm

The guide was updated with the following:

Configured ttn-account-v2 account server
Added information about setting discovery server’s CA certificate to ttnctl

ndarkness · December 16, 2016, 10:34am

Hi @htdvisserm I followed the tutorial that you provided in the link above, and almost everything is going well, however now I get this error. I am runing the backend from a docker station, making use of your docker-compose file. I think that the IP address is not right in my case, is there a way to specify this IP address in the configuration?

discovery_1 | WARN Could not connect to Redis. Retrying… error=dial tcp: i/o timeout
networkserver_1 | WARN Could not connect to Redis. Retrying… error=dial tcp: i/o timeout
handler_1 | WARN Could not connect to Redis. Retrying… error=dial tcp: i/o timeout
discovery_1 | WARN Could not connect to Redis. Retrying… error=dial tcp: i/o timeout
networkserver_1 | WARN Could not connect to Redis. Retrying… error=dial tcp: i/o timeout
handler_1 | WARN Could not connect to Redis. Retrying… error=dial tcp: i/o timeout
router_1 | DEBUG Could not connect to gRPC server, reconnecting… Address=discovery:1900 error=dial tcp 172.20.0.4:1900: getsockopt: connection refused
broker_1 | DEBUG Could not connect to gRPC server, reconnecting… Address=discovery:1900 error=dial tcp 172.20.0.4:1900: getsockopt: connection refused
discovery_1 | WARN Could not connect to Redis. Retrying… error=dial tcp: i/o timeout
networkserver_1 | WARN Could not connect to Redis. Retrying… error=dial tcp: i/o timeout
handler_1 | WARN Could not connect to Redis. Retrying… error=dial tcp: i/o timeout

Thank you very much!

htdvisser · December 16, 2016, 11:54am

@ndarkness: The guide only explained how to set everything up locally. In my next article I will will explain how to deploy using docker.

ndarkness · December 16, 2016, 12:07pm

Hi @htdvisser,

Thanks for the reply, would it be quite soon if I ask whether this is related to IP issues in docker?

vitalypru · December 16, 2016, 5:01pm

Hi

I’m trying to start local ttn routing services on Windows.
I set discovery ttn.yml file as in your guide and try to get access token for router using:
ttn discovery authorize router my-router --config ./discovery/ttn.yml

I got warning:
WARN ttn: Failed to refresh public keys for token validation: Get file://discovery/key: unsupported protocol scheme “file”
and discovery server started wothout any information about access token.

Whats wrong ?

htdvisser · December 19, 2016, 8:09am

In the guide we use localhost everywhere. Docker uses different hostnames, so you’d need to have different certificates and different config. I’ll explain everything soon

This sounds like you’re using an old ttn version. Please check that you have the latest one.

vitalypru · December 19, 2016, 9:25am

Thanks.
You are right. I’ve used old version.
Now I update source to v2.0.1 and I’m able to form access token for router, using ttn discovery command.
But when I try to start router I’ve got fatal memory error:

D:\Projects\lora\TheThingsNet>"D:\Projects\lora\TheThingsNet\src\github.com\TheThingsNetwork\ttn\ttn.exe" router  --config ".\router\ttn.yml"
Using config file: .\router\ttn.yml
  INFO Initializing The Things Network          Auth Servers=map[ttn-account-v2:https://account.thethingsnetwork.org] ComponentID=vitaly-router Description= Discovery Server Address=localhost:1900 Monitors=map[]
  INFO Initializing Router                      Announce=localhost:1901 Server=0.0.0.0:1901
  INFO Starting
 FATAL Stopping because of panic                panic=runtime error: invalid memory address or nil pointer dereference stack=goroutine 1 [running]:
github.com/TheThingsNetwork/ttn/cmd.Execute.func1()
        D:/Projects/lora/TheThingsNet/src/github.com/TheThingsNetwork/ttn/cmd/root.go:110 +0x80
panic(0xce6920, 0xc042004090)
        D:/Go/src/runtime/panic.go:458 +0x251
crypto/x509.(*CertPool).AddCert(0x0, 0xc04235b200)
        D:/Go/src/crypto/x509/cert_pool.go:74 +0x4f
crypto/x509.(*CertPool).AppendCertsFromPEM(0x0, 0xc04235afdf, 0x0, 0x200, 0x45f)
        D:/Go/src/crypto/x509/cert_pool.go:113 +0x168
github.com/TheThingsNetwork/ttn/core/component.(*Component).initRoots(0xc0421ef0e0, 0x0, 0x0)
        D:/Projects/lora/TheThingsNet/src/github.com/TheThingsNetwork/ttn/core/component/auth.go:153 +0xd7
github.com/TheThingsNetwork/ttn/core/component.(*Component).(github.com/TheThingsNetwork/ttn/core/component.initRoots)-fm(0x0, 0x0)
        D:/Projects/lora/TheThingsNet/src/github.com/TheThingsNetwork/ttn/core/component/auth.go:32 +0x31
github.com/TheThingsNetwork/ttn/core/component.(*Component).InitAuth(0xc0421ef0e0, 0xc042171ce0, 0xc0423000f0)
        D:/Projects/lora/TheThingsNet/src/github.com/TheThingsNetwork/ttn/core/component/auth.go:39 +0x174
github.com/TheThingsNetwork/ttn/core/component.New(0x12f5b80, 0xc042371660, 0xde0064, 0x6, 0xc04238ef90, 0xe, 0xe, 0xdec4e4, 0x13)
        D:/Projects/lora/TheThingsNet/src/github.com/TheThingsNetwork/ttn/core/component/component.go:84 +0x522
github.com/TheThingsNetwork/ttn/cmd.glob..func15(0x13228a0, 0xc042360020, 0x0, 0x2)
        D:/Projects/lora/TheThingsNet/src/github.com/TheThingsNetwork/ttn/cmd/router.go:36 +0x1e6
github.com/TheThingsNetwork/ttn/vendor/github.com/spf13/cobra.(*Command).execute(0x13228a0, 0xc042305ea0, 0x2, 0x2, 0x13228a0, 0xc042305ea0)
        D:/Projects/lora/TheThingsNet/src/github.com/TheThingsNetwork/ttn/vendor/github.com/spf13/cobra/command.go:636 +0x44a
github.com/TheThingsNetwork/ttn/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x1322680, 0x461208, 0xc042241e80, 0xc042241e90)
        D:/Projects/lora/TheThingsNet/src/github.com/TheThingsNetwork/ttn/vendor/github.com/spf13/cobra/command.go:722 +0x36e
github.com/TheThingsNetwork/ttn/vendor/github.com/spf13/cobra.(*Command).Execute(0x1322680, 0xeb78e8, 0x4755bc)
        D:/Projects/lora/TheThingsNet/src/github.com/TheThingsNetwork/ttn/vendor/github.com/spf13/cobra/command.go:681 +0x32
github.com/TheThingsNetwork/ttn/cmd.Execute()
        D:/Projects/lora/TheThingsNet/src/github.com/TheThingsNetwork/ttn/cmd/root.go:116 +0x60
main.main()
        D:/Projects/lora/TheThingsNet/src/github.com/TheThingsNetwork/ttn/main.go:23 +0x175

htdvisser · December 19, 2016, 9:52am

Didn’t see that one coming…

Will fix that for you.

htdvisser · December 19, 2016, 11:09am

This should now be fixed in the develop branch, @vitalypru.

vitalypru · December 19, 2016, 11:47am

Thank you.
Now I can start all ttn services.
But I faced with absolutely the same problem for lora-gateway-bridge

htdvisser · December 19, 2016, 11:52am

lora-gateway-bridge uses the same code to get the CA Roots from the system. We still have to update it to make it work with Windows.

ndarkness · December 19, 2016, 1:50pm

@htdvisser Could you tell me what does this error mean? Is the token that I am using completely wrong?

broker_1 | FATAL Could not initialize broker error=Failed to announce this component to TTN discovery: rpc error: code = 7 desc = permission denied: Discovery:Token issuer local is not allowed to make changes to the network settings: permission denied: Discovery:Token issuer local is not allowed to make changes to the network settings
ttnbackbone_broker_1 exited with code 1

htdvisser · December 19, 2016, 2:14pm

Looks like your discovery server does not accept the “local” “master-auth-server”. Make sure your discovery config contains the following:

discovery:
  master-auth-servers:
  - ttn-account-v2
  - local

Pretty sure this is in the guide as well. If not, let me know

ndarkness · December 19, 2016, 2:18pm

This is my current configuration for the discovery,

id: discovery
debug: true
key-dir: “./env/discovery/”
auth-servers:
ttn-account-v2: “https://account.thethingsnetwork.org”
#ttn-account-preview: “https://preview.account.thethingsnetwork.org”
local: “file:///root/env/discovery/server.pub”

tls : true
discovery:
redis-address: redis:6379
server-address: discovery
master-aut-servers:

ttn-account-v2

local

with that I get still have the error

discovery_1 | INFO ttn: Got public keys for token validation
discovery_1 | DEBUG Connected to gRPC server Address=localhost:1900
discovery_1 | FATAL Could not start client for gRPC proxy error=x509: certificate is valid for discovery, not localhost
ttnbackbone_discovery_1 exited with code 1

htdvisser · December 19, 2016, 2:43pm

I suspect it’s because the typo in master-aut-servers (there’s a h missing)

ndarkness · December 19, 2016, 2:46pm

HI @htdvisser thanks again!

I have corrected it , but still the same issue

discovery_1      |   INFO ttn: Got public keys for token validation
discovery_1      |  DEBUG Connected to gRPC server                 Address=localhost:1900
discovery_1      |  FATAL Could not start client for gRPC proxy    error=x509: certificate is valid for discovery, not localhost
handler_1        |   WARN Could not connect to Redis. Retrying...  error=dial tcp [::1]:6379: getsockopt: connection refused

Edit: Why do I get the same error even if I run the discovery specifying the IP address of itself?

VirtualBox:~/TTNBackBone$ ./ttn-linux-amd64 discovery  --config="/home/lorabackbone/TTNBackBone/envLocal/discovery/conf.yml"
Using config file: /home/lorabackbone/TTNBackBone/envLocal/discovery/conf.yml
  INFO Initializing The Things Network          Auth Servers=map[local:file:///home/lorabackbone/TTNBackBone/envLocal/discovery/server.pub ttn-account-v2:https://account.thethingsnetwork.org] ComponentID=conf-discovery Description= Discovery Server Address=discover.thethingsnetwork.org:1900 Monitors=map[]
  INFO Initializing Discovery                   Database=localhost:6379/0 HTTP Proxy=0.0.0.0:8080 Server=0.0.0.0:1900
  INFO Starting                                
  INFO ttn: Got public keys for token validation
 DEBUG Connected to gRPC server                 Address=localhost:1900
 FATAL Could not start client for gRPC proxy    error=x509: certificate is valid for conf-discovery, not localhost

 
VirtualBox:~/TTNBackBone$ ./ttn-linux-amd64 discovery --discovery-address="172.20.0.4" --config="/home/lorabackbone/TTNBackBone/envLocal/discovery/conf.yml"
Using config file: /home/lorabackbone/TTNBackBone/envLocal/discovery/conf.yml
  INFO Initializing The Things Network          Auth Servers=map[ttn-account-v2:https://account.thethingsnetwork.org local:file:///home/lorabackbone/TTNBackBone/envLocal/discovery/server.pub] ComponentID=conf-discovery Description= Discovery Server Address=172.20.0.4 Monitors=map[]
  INFO Initializing Discovery                   Database=localhost:6379/0 HTTP Proxy=0.0.0.0:8080 Server=0.0.0.0:1900
  INFO Starting                                
  INFO ttn: Got public keys for token validation
 DEBUG Connected to gRPC server                 Address=localhost:1900
 FATAL Could not start client for gRPC proxy    error=x509: certificate is valid for conf-discovery, not localhost

vitalypru · December 19, 2016, 4:43pm

When I faced with the same problem, I found, that I miss to add “localhost” word in command line during discovery certificate creation.

$ ttn discovery gen-cert localhost --config ./discovery/ttn.yml

vitalypru · December 19, 2016, 4:46pm

Now I can’t perform OTAA activation:

D:\Projects\lora\TheThingsNet>“D:\Projects\lora\TheThingsNet\src\github.com\TheThingsNetwork\ttn\ttn.exe” broker --config “.\broker\ttn.yml”
Using config file: .\broker\ttn.yml
INFO Initializing The Things Network Auth Servers=map[ttn-account-v2:https://account.thethingsnetwork.org] ComponentID=vitaly-broker Description= Discovery Server Address=localhost:1900 Monitors=map[]
INFO Initializing Broker Announce=localhost:1902 DeduplicationDelay=200 NetworkServer=localhost:1903 Server=0.0.0.0:1902
INFO Starting
DEBUG Connected to gRPC server Address=localhost:1900
INFO ttn: Got public keys for token validation
INFO ttn: Announced to TTN discovery
DEBUG Connected to gRPC server Address=localhost:1903
INFO Prefix found in Discovery DPrefix=26000000/20 NSPrefix=26000000/20
DEBUG Start stream CallerID=vitaly-handler CallerIP=127.0.0.1:55949 Method=/broker.Broker/Publish
DEBUG Start stream CallerID=vitaly-handler CallerIP=127.0.0.1:55949 Method=/broker.Broker/Subscribe
DEBUG grpc: Server.Serve failed to complete security handshake from “127.0.0.1:55961”: EOF
DEBUG Start stream CallerID=vitaly-router CallerIP=127.0.0.1:55962 Method=/broker.Broker/Associate
DEBUG Connected to gRPC server Address=localhost:1904
WARN Could not dial handler for Activation AppEUI=70B3D57EF0000023 AppID=counter_app DevEUI=0088444D571864EB DevID=local_dev GatewayID=eui-a75c3affffe606e0 NumHandlers=2 error=x509: certificate signed by unknown authority (possibly bec
ause of “x509: ECDSA verification failure” while trying to verify candidate authority certificate “serial:279521433737929252104238787983170638246”)
DEBUG Connected to gRPC server Address=localhost:1904
DEBUG Forward Activation AppEUI=70B3D57EF0000023 AppID=counter_app DevEUI=0088444D571864EB DevID=local_dev GatewayID=eui-a75c3affffe606e0 HandlerID=vitaly-handler NumHandlers=2

ndarkness · December 20, 2016, 8:14am

Thanks @vitalypru, the thing it’s that I am using another “user” to sign the certificate, so I used

$ ttn discovery gen-cert conf-discovery --config ./discovery/ttn.yml

Did you run your private network in a docker container as well?

Setting up a Private Routing Environment

Thanks. You are right. I’ve used old version. Now I update source to v2.0.1 and I’m able to form access token for router, using ttn discovery command. But when I try to start router I’ve got fatal memory error:

Thanks.
You are right. I’ve used old version.
Now I update source to v2.0.1 and I’m able to form access token for router, using ttn discovery command.
But when I try to start router I’ve got fatal memory error: