Fresh install of The Things Stack on Ubuntu fails with open /run/secrets/key.pem: permission denied

How to get started
1

Hi,

I’ve followed the Installation Guide and the video to “localhost” install The Things Stack on a new Ubuntu 20.04.

At the end, when starting with “docker-compose up”, I can see a great number of errors (see below) and the web server does not respond to the browser in https.

I’m using Ubuntu 20.04, docker-compose 1.29.2, docker engine 20.10.12

Two other people reported the same issue in the discussion attached to the YouTube video.

Any idea what is wrong ?

Thanks

fabien@fabien-System-Product-Name:~/the-things-stack$ docker-compose up
Starting the-things-stack_redis_1     ... done
Starting the-things-stack_cockroach_1 ... done
Starting the-things-stack_stack_1     ... done
Attaching to the-things-stack_redis_1, the-things-stack_cockroach_1, the-things-stack_stack_1
redis_1      | 1:C 16 Feb 2022 03:48:32.196 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1      | 1:C 16 Feb 2022 03:48:32.196 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1      | 1:C 16 Feb 2022 03:48:32.196 # Configuration loaded
cockroach_1  | *
cockroach_1  | * WARNING: ALL SECURITY CONTROLS HAVE BEEN DISABLED!
cockroach_1  | * 
cockroach_1  | * This mode is intended for non-production testing only.
cockroach_1  | * 
cockroach_1  | * In this mode:
cockroach_1  | * - Your cluster is open to any client that can access any of your IP addresses.
cockroach_1  | * - Intruders with access to your machine or network can observe client-server traffic.
cockroach_1  | * - Intruders can log in without password and read or write any data in the cluster.
cockroach_1  | * - Intruders can consume all your server's resources and cause unavailability.
cockroach_1  | *
cockroach_1  | *
cockroach_1  | * INFO: To start a secure server without mandating TLS for clients,
cockroach_1  | * consider --accept-sql-without-tls instead. For other options, see:
cockroach_1  | * 
cockroach_1  | * - https://go.crdb.dev/issue-v/53404/v21.2
cockroach_1  | * - https://www.cockroachlabs.com/docs/v21.2/secure-a-cluster.html
cockroach_1  | *
redis_1      | 1:M 16 Feb 2022 03:48:32.198 * monotonic clock: POSIX clock_gettime
cockroach_1  | *
cockroach_1  | * WARNING: neither --listen-addr nor --advertise-addr was specified.
cockroach_1  | * The server will advertise "ffac1881044d" to other nodes, is this routable?
cockroach_1  | * 
cockroach_1  | * Consider using:
cockroach_1  | * - for local-only servers:  --listen-addr=localhost
cockroach_1  | * - for multi-node clusters: --advertise-addr=<host/IP addr>
cockroach_1  | * 
cockroach_1  | *
redis_1      | 1:M 16 Feb 2022 03:48:32.199 * Running mode=standalone, port=6379.
redis_1      | 1:M 16 Feb 2022 03:48:32.199 # Server initialized
redis_1      | 1:M 16 Feb 2022 03:48:32.199 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis_1      | 1:M 16 Feb 2022 03:48:32.203 * DB loaded from append only file: 0.003 seconds
redis_1      | 1:M 16 Feb 2022 03:48:32.203 * Ready to accept connections
cockroach_1  | CockroachDB node starting at 2022-02-16 03:48:33.17312961 +0000 UTC (took 0.6s)
cockroach_1  | build:               CCL v21.2.5 @ 2022/02/07 21:01:07 (go1.16.6)
cockroach_1  | webui:               http://ffac1881044d:26256
cockroach_1  | sql:                 postgresql://root@ffac1881044d:26257/defaultdb?sslmode=disable
cockroach_1  | sql (JDBC):          jdbc:postgresql://ffac1881044d:26257/defaultdb?sslmode=disable&user=root
cockroach_1  | RPC client flags:    /cockroach/cockroach <client cmd> --host=ffac1881044d:26257 --insecure
cockroach_1  | logs:                /cockroach/cockroach-data/logs
cockroach_1  | temp dir:            /cockroach/cockroach-data/cockroach-temp255587863
cockroach_1  | external I/O path:   /cockroach/cockroach-data/extern
cockroach_1  | store[0]:            path=/cockroach/cockroach-data
cockroach_1  | storage engine:      pebble
cockroach_1  | status:              restarted pre-existing node
cockroach_1  | clusterID:           ccbd7701-3957-41d7-bff7-b7dbb9c128fe
cockroach_1  | nodeID:              1
stack_1      | INFO	Setting up core component
stack_1      | WARN	No cookie hash key configured, generated a random one
stack_1      | WARN	No cookie block key configured, generated a random one
stack_1      | INFO	Setting up Identity Server
stack_1      | INFO	Setting up Gateway Server
stack_1      | INFO	Setting up Network Server
stack_1      | INFO	Setting up Application Server
stack_1      | INFO	Setting up Join Server
stack_1      | INFO	Setting up Console
stack_1      | INFO	Setting up Gateway Configuration Server
stack_1      | INFO	Setting up Device Template Converter
stack_1      | INFO	Setting up QR Code Generator
stack_1      | INFO	Setting up Packet Broker Agent
stack_1      | INFO	Setting up Device Repository
stack_1      | INFO	Starting...
stack_1      | WARN	No cluster key configured, generated a random one	{"key": "bacd4a01965e9b0cd41cfc5fcd97d72637aa4a639603811ec3aa60b3d02ccb21", "namespace": "cluster"}
stack_1      | INFO	Listening for connections	{"address": ":1884", "namespace": "grpc", "protocol": "gRPC"}
stack_1      | ERROR	Could not start gRPC server	{"error": "error:pkg/component:listener (could not create `gRPC/tls` listener)", "error_cause": "open /run/secrets/key.pem: permission denied", "protocol": "gRPC/tls"}
stack_1      | go.thethings.network/lorawan-stack/v3/pkg/log.(*zapHandler).HandleLog
stack_1      | 	/home/runner/work/lorawan-stack/lorawan-stack/pkg/log/zap_handler.go:84
stack_1      | go.thethings.network/lorawan-stack/v3/pkg/log.(*Logger).Use.func1
stack_1      | 	/home/runner/work/lorawan-stack/lorawan-stack/pkg/log/logger.go:55
stack_1      | go.thethings.network/lorawan-stack/v3/pkg/log.HandlerFunc.HandleLog
stack_1      | 	/home/runner/work/lorawan-stack/lorawan-stack/pkg/log/handler.go:38
stack_1      | go.thethings.network/lorawan-stack/v3/pkg/log/middleware/observability.(*observability).Wrap.func1
stack_1      | 	/home/runner/work/lorawan-stack/lorawan-stack/pkg/log/middleware/observability/observability.go:86
stack_1      | go.thethings.network/lorawan-stack/v3/pkg/log.HandlerFunc.HandleLog
stack_1      | 	/home/runner/work/lorawan-stack/lorawan-stack/pkg/log/handler.go:38
stack_1      | go.thethings.network/lorawan-stack/v3/pkg/log.(*Logger).commit
stack_1      | 	/home/runner/work/lorawan-stack/lorawan-stack/pkg/log/logger.go:75
stack_1      | go.thethings.network/lorawan-stack/v3/pkg/log.(*entry).commit
stack_1      | 	/home/runner/work/lorawan-stack/lorawan-stack/pkg/log/entry.go:69
stack_1      | go.thethings.network/lorawan-stack/v3/pkg/log.(*entry).Error
stack_1      | 	/home/runner/work/lorawan-stack/lorawan-stack/pkg/log/entry.go:94
stack_1      | go.thethings.network/lorawan-stack/v3/pkg/component.(*Component).Start
stack_1      | 	/home/runner/work/lorawan-stack/lorawan-stack/pkg/component/component.go:309
stack_1      | go.thethings.network/lorawan-stack/v3/pkg/component.(*Component).Run
stack_1      | 	/home/runner/work/lorawan-stack/lorawan-stack/pkg/component/component.go:348
stack_1      | go.thethings.network/lorawan-stack/v3/cmd/ttn-lw-stack/commands.glob..func16
stack_1      | 	/home/runner/work/lorawan-stack/lorawan-stack/cmd/ttn-lw-stack/commands/start.go:477
stack_1      | github.com/spf13/cobra.(*Command).execute
stack_1      | 	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:856
stack_1      | github.com/spf13/cobra.(*Command).ExecuteC
stack_1      | 	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:974
stack_1      | main.main
stack_1      | 	/home/runner/work/lorawan-stack/lorawan-stack/cmd/ttn-lw-stack/main.go:26
stack_1      | runtime.main
stack_1      | 	/opt/hostedtoolcache/go/1.17.6/x64/src/runtime/proc.go:255
stack_1      | WARN	[transport]transport: http2Server.HandleStreams failed to read frame: io: read/write on closed pipe	{"namespace": "grpc"}
stack_1      | error:pkg/component:listener (could not create `gRPC/tls` listener)
stack_1      |     protocol=gRPC/tls
stack_1      |     correlation_id=bee754bd253d462aba83091e4f4ba24f
stack_1      | --- open /run/secrets/key.pem: permission denied
^CGracefully stopping... (press Ctrl+C again to force)
Stopping the-things-stack_stack_1     ... done
Stopping the-things-stack_redis_1     ... done
Stopping the-things-stack_cockroach_1 ... done
fabien@fabien-System-Product-Name:~/the-things-stack$ ^C
fabien@fabien-System-Product-Name:~/the-things-stack$
2

Installing as localhost is problematic, hence the warnings. The error message makes reference to TLS and key.pem, not a huge neon sign of a pointer, but easily Googleable for info …

You can’t use https if you don’t have a certificate and you can’t get a certificate if you use localhost.

So either remove the s from all the https’s in the main config file and just go unencrypted (which will be fun logging in but you can do that with a manual callback)

OR

Use a static IP on your network for this server, get a DNS entry setup for it and then you can get a free certificate.

OR

Spend $10/month at Linode on a virtual machine - they give all new users $100 credit so that is 10 months for free - and run it on the Interwebs thereby allowing you to have gateways outside your network.

1 Like
3

Linode is indeed an option, but one I would like to avoid for different reasons.

Regarding the certificates, I have followed the specific steps (here in the video at 2:10, and here in the doc) to create a custom certificate for the localhost using cloudflare PKI/TLS toolkit). In the video, the presenter installs TTS on a Mac and ends by showing how to tell his Mac to use the new certificate. That’s the only step I didn’t do on my Ubuntu. I don’t know what would be the equivalent on Linux. Perhaps that’s key for this to work.

Unfortunately it didn’t work and I don’t know enough about TTS and certificates to debug. A few pointers would be of great help to put me on the right path.

4

P.S. Further info, there is no “/run/secrets” directory on my Ubuntu. I tried to create one, also to copy my *.pem files, but I still get the same error.

5

I have a local stack running on my Ubuntu machine. I could never get certificates to work, so I could still use it, just not with https and also not with LBS.

Did you remember to type

sudo chown 886:886 ./cert.pem ./key.pem

after creating your certificates?

1 Like
6

It won’t be, its inside the docker container …

Which means you need to learn Docker …

7

Perfect @jssani, that was it.

sudo chown 886:886 ./cert.pem ./key.pem

I can now log in to the console using https from that local machine, and also from across the LAN from a Windows machine. However they both complain that this certificate is unknown. Once I click “ignore and proceed anyway” it connects.

8

Now how should I set up “docker-compose up -d” to automatically start upon boot up?
Is it with systemd ?

9

Once you’ve learn’t Linux, you’ll know.

10

Mr McCloud, we are in the “getting started” forum. If you can’t bear this kind of questions, perhaps you could just let other people take over.

11

I see now that the TTS starts just fine by itself on boot up, no need to further “docker-compose up” at every boot up.

12

I apologise, I didn’t see that this was in getting started,

It would never cross my mind that this would be a getting started activity.

You are deploying a complicated software stack for a rather complicated communications system that has many many moving parts whilst asking questions that are the very beginning of the SysOps journey. As we are all volunteers answering, it becomes very hard to assist when there is a fundamental lack of domain knowledge that’s needed to be able to answer the question. This becomes particularly tricky when people are given the answers without any context and come unstuck at a later date because the background information isn’t there.

So a solid grasp of the basics of Docker is a minimum requirement - because whilst there are now instructions on backing up, it requires you to get in to Docker beyond the instructions provided on the website.

The implication of knowing Docker is that you have a solid grasp of the core parts of Linux. The error message told you that it couldn’t open the file due to a permissions issue. Which is very much the core of any 'nix based system.

So, to answer your question, yes, setup a startup script in systemd, but I don’t advise it and I don’t do that. Because I want to control the startup of a server like this, there’s a lot going on at startup and even if you defer it in the script for a minute or so, there is the question of how the server got to be off in the first place, it it wasn’t a controlled shutdown then you could have database issues and if you start collecting data in to that instance but need to roll back to a backup, then it gets very messy pulling data out of the ‘accidentally live’ system and getting it in to the restored known good system. So if it’s not automatic, I can control that.

If it’s for casual use entirely on your internal network, as I suggested above, it is sooo much easier to edit out all the references to https and replace them with http in ttn-lw-stack-docker.yml - then you won’t have your browsers trying to stop you accessing a site with a suspect looking certificate.

However, as mentioned above, you can configure your server so that it can have a proper certificate for which the ACME installer that works with the free LetsEncrypt certificates will do all the work for you.

It would be very useful to understand how you ended up coming to install TTS OS on Linux - people come to LoRaWAN from all sorts of angles and after some struggles to get started, end up having to fill in some of core info before they can make real progress. Installing TTS OS on Ubuntu without being a regular Linux user is definitely brave!

13

This topic was automatically closed 24 hours after the last reply. New replies are no longer allowed.