Issues with login and self-signed certificates

RTEC · June 18, 2020, 9:22am

Dear all,

I am setting up a demo in a private environment and I definetely need some help in troubleshooting a (I think) certificate-related issue.
I am working on a virtual machine (Virtual Box) with Debian 10.
The stack is configured and after executing the “docker-compose up” command, it goes up and running.
I can access the console and display the login page, but after inserting the admin credentials, I get a “Forbidden - Token Exchange refused”.
I am using self-signed certificates that have been generated by:

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 700 -nodes
sudo chown 886:886 ./cert.pem ./key.pem

The configuration files are:

docker-compose.yml

version: ‘3.7’
services:

#If using CockroachDB:
cockroach:
image: cockroachdb/cockroach
command: start --http-port 26256 --insecure
restart: unless-stopped
volumes:
- ${DEV_DATA_DIR:-.env/data}/cockroach:/cockroach/cockroach-data
ports:
- “127.0.0.1:26257:26257” # Cockroach
- “127.0.0.1:26256:26256” # WebUI

redis:
image: redis
command: redis-server --appendonly yes
restart: unless-stopped
volumes:
- ${DEV_DATA_DIR:-.env/data}/redis:/data
ports:
- “127.0.0.1:6379:6379”

stack:
image: thethingsnetwork/lorawan-stack
entrypoint: ttn-lw-stack -c /config/ttn-lw-stack.yml
command: start
restart: unless-stopped
depends_on:
- redis
# If using CockroachDB:
- cockroach
volumes:
- ./blob:/srv/ttn-lorawan/public/blob
- ./config/stack:/config:ro
# If using Let’s Encrypt:
#- ./acme:/var/lib/acme
environment:
TTN_LW_BLOB_LOCAL_DIRECTORY: /srv/ttn-lorawan/public/blob
TTN_LW_REDIS_ADDRESS: redis:6379
# If using CockroachDB:
TTN_LW_IS_DATABASE_URI: postgres://root@cockroach:26257/ttn_lorawan?sslmode=disable
# test:
TTN_LW_TLS_SOURCE: file
TTN_LW_TLS_ROOT_CA: /run/secrets/cert.pem
TTN_LW_TLS_CERTIFICATE: /run/secrets/cert.pem
TTN_LW_TLS_KEY: /run/secrets/key.pem
#####
ports:
  # If deploying on a public server:
  - "80:1885" #
  - "443:8885" #
  - "1881:1881"
  - "8881:8881"
  - "1882:1882"
  - "8882:8882"
  - "1883:1883"
  - "8883:8883"
  - "1884:1884"
  - "8884:8884"
  - "1885:1885"
  - "8885:8885"
  - "1887:1887"
  - "8887:8887"
  - "1700:1700/udp"
  
secrets:
  - cert.pem
  - key.pem
#If using (self) signed certificates:
secrets:
cert.pem:
file: ./cert.pem
key.pem:
file: ./key.pem

ttn-lw-stack.yml

#Example ttn-lw-stack configuration file
#Keep in sync with doc/content/guides/getting-started/configuration.md

#Identity Server configuration
is:
#Web UI configuration for “thethings.example.com”:
oauth:
ui:
canonical-url: ‘https://i4s.example.com/oauth’
is:
base-url: ‘https://i4s.example.com/api/v3’

#HTTP server configuration
http:
cookie:
block-key: ‘0011223344556677001122334455667700112233445566770011223344556677’ # generate 32 bytes (openssl rand -hex 32)
hash-key: ‘00112233445566770011223344556677001122334455667700112233445566770011223344556677001122334455667700112233445566770011223344556677’ # generate 64 bytes (openssl rand -hex 64)
metrics:
password: ‘i4s’ # choose a password
pprof:
password: ‘i4s’ # choose a password

#If using (self) signed certificates:
tls:
source: file
root-ca: /run/secrets/cert.pem
certificate: /run/secrets/cert.pem
key: /run/secrets/key.pem

#If Gateway Server enabled, defaults for “thethings.example.com”:
gs:
mqtt:
public-address: ‘i4s.example.com:1882’
public-tls-address: ‘i4s.example.com:8882’
mqtt-v2:
public-address: ‘i4s.example.com:1881’
public-tls-address: ‘i4s.example.com:8881’

#If Gateway Configuration Server enabled, defaults for “thethings.example.com”:
gcs:
basic-station:
default:
lns-uri: ‘wss://i4s.example.com:8887’
the-things-gateway:
default:
mqtt-server: ‘mqtts://i4s.example.com:8881’

#Web UI configuration for “thethings.example.com”:
console:
ui:
canonical-url: ‘https://i4s.example.com/console’
is:
base-url: ‘https://i4s.example.com/api/v3’
gs:
base-url: ‘https://i4s.example.com/api/v3’
ns:
base-url: ‘https://i4s.example.com/api/v3’
as:
base-url: ‘https://i4s.example.com/api/v3’
js:
base-url: ‘https://i4s.example.com/api/v3’
qrg:
base-url: ‘https://i4s.example.com/api/v3’
edtc:
base-url: ‘https://i4s.example.com/api/v3’

oauth:
authorize-url: ‘https://i4s.example.com/oauth/authorize’
token-url: ‘https://i4s.example.com/oauth/token’
client-id: ‘console’
client-secret: ‘i4s’ # choose or generate a secret

When I open the console the log on the terminal is:

stack_1 | INFO Request handled duration=293.505µs method=GET namespace=web remote_addr=172.18.0.1:55450 request_id=01EB36EMX3A8MSN7YR21MCB72Y response_size=0 status=302 url=https://i4s.example.com/
stack_1 | INFO Request handled duration=857.744µs method=GET namespace=web remote_addr=172.18.0.1:55450 request_id=01EB36EMY07K58BC6X575DAMNP response_size=790 status=200 url=https://i4s.example.com/console/
stack_1 | INFO Request handled duration=333.103µs method=GET namespace=web remote_addr=172.18.0.1:55450 request_id=01EB36ENPY6ETVG18QZZ6AVGW0 response_size=35390 status=200 url=https://i4s.example.com/assets/console-touch-icon.png
stack_1 | INFO Request handled duration=159.287µs method=GET namespace=web remote_addr=172.18.0.1:55450 request_id=01EB36ENPZY8F5PZJWZ8QNSZZW response_size=6239 status=200 url=https://i4s.example.com/assets/console-favicon.svg
stack_1 | INFO Client error duration=443.447µs method=GET namespace=web remote_addr=172.18.0.1:55450 request_id=01EB36ENZ04G3KMYZZ4B1M6Z5T response_size=198 status=401 url=https://i4s.example.com/console/api/auth/token
stack_1 | INFO Request handled duration=292.645µs method=GET namespace=web remote_addr=172.18.0.1:55450 request_id=01EB36ENZG4RNAA2945M7K59N1 response_size=25656 status=200 url=https://i4s.example.com/assets/source-sans-pro-v13-latin_latin-ext-regular.80c998aa03640281d556814a14e1d9c8.woff2
stack_1 | INFO Request handled duration=196.797µs method=GET namespace=web remote_addr=172.18.0.1:55450 request_id=01EB36EP2S01PQZ4FF0HGC5CEJ response_size=5004 status=200 url=https://i4s.example.com/assets/logo.svg
stack_1 | INFO Request handled duration=1.00027ms method=GET namespace=web remote_addr=172.18.0.1:55450 request_id=01EB36EP3MF7BAER7QXYDQJ4H4 response_size=0 status=302 url=https://i4s.example.com/console/login/ttn-stack?next=/
stack_1 | INFO Request handled duration=172.138µs method=GET namespace=web remote_addr=172.18.0.1:55450 request_id=01EB36EP66PVBEMJ1HYHBS78SG response_size=0 status=302 url=https://i4s.example.com/oauth/authorize?client_id=console&redirect_uri=%2Fconsole%2Foauth%2Fcallback&response_type=code&state=Cy_gvOZ_NjHzMp6T
stack_1 | INFO Request handled duration=972.592µs method=GET namespace=web remote_addr=172.18.0.1:55450 request_id=01EB36EP7MK4DSPV8JT08FKBGK response_size=682 status=200 url=https://i4s.example.com/oauth/login?n=%2Foauth%2Fauthorize%3Fclient_id%3Dconsole%26redirect_uri%3D%252Fconsole%252Foauth%252Fcallback%26response_type%3Dcode%26state%3DCy_gvOZ_NjHzMp6T
stack_1 | INFO Request handled duration=426.452µs method=GET namespace=web remote_addr=172.18.0.1:55450 request_id=01EB36EPASHHVB11DSCZSMATNE response_size=82038 status=200 url=https://i4s.example.com/assets/oauth.e86e67ee80615254acbd.css
stack_1 | INFO Request handled duration=6.413105ms method=GET namespace=web remote_addr=172.18.0.1:55450 request_id=01EB36EPAWY3TCZ9XVV2YQQTJF response_size=1307132 status=200 url=https://i4s.example.com/assets/oauth.cb07e406a0e31a9259cd.js
stack_1 | INFO Client error duration=422.648µs method=GET namespace=web remote_addr=172.18.0.1:55450 request_id=01EB36EPWHRR218HEBYJDAMCA6 response_size=177 status=401 url=https://i4s.example.com/oauth/api/me
stack_1 | INFO Request handled duration=336.214µs method=GET namespace=web remote_addr=172.18.0.1:55450 request_id=01EB36EPXAGZMM0XDRH1NQZ22Q response_size=35390 status=200 url=https://i4s.example.com/assets/oauth-touch-icon.png
stack_1 | INFO Request handled duration=177.044µs method=GET namespace=web remote_addr=172.18.0.1:55450 request_id=01EB36EPXCEWCK9C83GV3YPQDZ response_size=6239 status=200 url=https://i4s.example.com/assets/oauth-favicon.svg
stack_1 | INFO Request handled duration=326.753µs method=GET namespace=web remote_addr=172.18.0.1:55450 request_id=01EB36EQ12QHCGCZT54Z0SJPPP response_size=25520 status=200 url=https://i4s.example.com/assets/source-sans-pro-v13-latin_latin-ext-600.117e12cdb861ed7356c805f6f515afbb.woff2
stack_1 | INFO Request handled duration=288.804µs method=GET namespace=web remote_addr=172.18.0.1:55450 request_id=01EB36EQ1914G7R8A0S9R4HTKZ response_size=25348 status=200 url=https://i4s.example.com/assets/source-sans-pro-v13-latin_latin-ext-700.639c2738552a0376c91e7d485e476fda.woff2

and after logging in I get:

stack_1 | INFO Request handled duration=11.012µs method=GET namespace=web remote_addr=127.0.0.1:33272 request_id=01EB36J16RCT427A59PVM3RTG8 response_size=3 status=200 url=http://localhost:1885/healthz/live
stack_1 | INFO Request handled duration=205.096888ms method=POST namespace=web remote_addr=172.18.0.1:55450 request_id=01EB36J3G657ESG1FWRAH4PVJW response_size=0 status=204 url=https://i4s.example.com/oauth/api/auth/login
stack_1 | INFO Request handled duration=73.729781ms method=GET namespace=web remote_addr=172.18.0.1:55450 request_id=01EB36J3Q56GAFD5PAH0H5KV77 response_size=0 status=302 url=https://i4s.example.com/oauth/authorize?client_id=console&redirect_uri=%2Fconsole%2Foauth%2Fcallback&response_type=code&state=Cy_gvOZ_NjHzMp6T
stack_1 | WARN error=unauthorized_client, internal_error= get_client=client check failed, client_id=console namespace=identityserver
stack_1 | WARN OAuth error error=error:pkg/oauth:unauthorized_client (client is not authorized to request a token using this method) method=POST namespace=web remote_addr=127.0.0.1:33274 request_id=01EB36J3SZ62ST0JQ5EEJV4SGY url=http://localhost:1885/oauth/token
stack_1 | INFO Client error duration=79.980169ms method=POST namespace=web remote_addr=127.0.0.1:33274 request_id=01EB36J3SZ62ST0JQ5EEJV4SGY response_size=209 status=403 url=http://localhost:1885/oauth/token
stack_1 | INFO Client error duration=97.311293ms method=GET namespace=web remote_addr=172.18.0.1:55450 request_id=01EB36J3SRFH96N7F35NTAS08E response_size=998 status=403 url=https://i4s.example.com/console/oauth/callback?code=MF2XI.N6H5EFOCFOLF2QVQ2AET4ZDZC3HOVDPAL37UTWY.ZRYQMXLVDWSUEHKWSNEWPU5Q6BG7ZPJLOTFJF3AIIU25CTB3YLPQ&state=Cy_gvOZ_NjHzMp6T
stack_1 | INFO Client error duration=1.289496ms method=GET namespace=web remote_addr=172.18.0.1:55450 request_id=01EB36J4Z0Z0CD5XZ7535NB99F response_size=198 status=401 url=https://i4s.example.com/console/api/auth/token

I am not so experienced in docker and such level of programming, so I am probably missing something easy. I tried to work it out myself, but I am stuck and I would need someone pointing me to the solution or troubleshooting path.

Thank you

RTEC · June 19, 2020, 1:05pm

Hi,

I am doing some further test.
I noticed that the address i4s.example.com was resolved only by the host, but not inside the container. So, I added a row in the docker-compose.yml:

extra_hosts:
- ‘i4s.example.com:127.0.0.1’
A ping confirmed it:

sudo docker exec -it --user root usr_stack_1 ping i4s.example.com

Now some curl from the host curl
but, if I execute the same from the ash console inside the docker, I get a

(7) Failed to connect to i4s.example.com port 443: connection refused

Now another curl:

From ash, I typed:

cat /var/run/secrets/key.pem
cat /var/run/secrets/cert.pem

and both were successfull.

benolayinka · July 2, 2020, 11:13am

Hello @RTEC, sorry for the delay getting back to you. Did you get this working?

Can you log in with the CLI?

We had a similar issue here: https://github.com/TheThingsNetwork/lorawan-stack/issues/2353#issuecomment-613511474

RTEC · July 2, 2020, 2:36pm

Hello @benolayinka, thank you for your reply.
No, I haven’t solved the issue, yet. I am working on it, trying several ways, but at the moment I am getting small progresses.

I tried accessing with the CLI but I get an error which basicaly told me that the certificates were for “i4s.example.com”, while I was trying to access localhost.
So, I generated the certificates for localhost and checked them with

“openssl verify -CAfile ca.pem cert.pem”

also, from inside the container:

cat /var/run/secrets/cert.pem
cat /var/run/secrets/ca.pem
cat /var/run/secrets/key.pem

In both case I got a positive response, but once I tried to access the CLI I got an error due to unknow authority. Still working to fix it… not sure if maybe I have to re-initialize the stack from the beginning.

benolayinka · July 2, 2020, 3:42pm

I think your issue is similar to the one we solved in github - certificates for the host and the docker container are different. Try following along here - https://github.com/TheThingsNetwork/lorawan-stack/issues/2353#issuecomment-617020487

If you’re accessing the docker container from outside the container, it won’t be at localhost.

system · July 18, 2020, 9:22am

This topic was automatically closed after 30 days. New replies are no longer allowed.