Token Exchange refused Error

1

Hello,

Would you please help as I am currently stuck with this issue for a week now.

I am running Things Stack version: ‘3.7’ on a ubuntu 18.04 virtual machine sitting on a LAN, it has domain name and public IP assigned and configured to automatically retrieve and update Let’s Encrypt certificates as per these instructions.

Following same config instructions as above link, I have edited every instance of the domain name thethings.example.com with my host local IP address. I am able to access host via SSL but , I get Forbidden Token Exchange Refused so the authorisation request generated from within the docker container is not getting redirected to ports 1885 & 8885.

So I get that this issue is around correctly handling requests between the docker container and my ubuntu VM.

Here’s my log:

    stack_1     |   INFO Request handled                          duration=360.55µs method=GET namespace=web remote_addr=10.2.80.39 request_id=01ER02XGK788MF8NYV9A1D4ZPS response_size=0 status=302 url=https://mydomain.com/
    stack_1     |   INFO Request handled                          duration=2.641752ms method=GET namespace=web remote_addr=10.2.80.39 request_id=01ER02XGKFZHG6N03RRVQKNF82 response_size=982 status=200 url=https://mydomain.com/console/
    stack_1     |   INFO Client error                             duration=1.400846ms method=GET namespace=web remote_addr=10.2.80.39 request_id=01ER02XHWFN35TFJJ6YEE2M4DE response_size=198 status=401 url=https://mydomain.com/console/api/auth/token
    stack_1     |   INFO Request handled                          duration=481.657µs method=GET namespace=web remote_addr=10.2.80.39 request_id=01ER02XJ38R5CBVGWT57PEGNAK response_size=0 status=302 url=https://mydomain.com/console/login/ttn-stack?next=/
    stack_1     |   INFO Request handled                          duration=534.572µs method=GET namespace=web remote_addr=10.2.80.39 request_id=01ER02XJ4PNCD8TJ28WTYRCJX2 response_size=0 status=302 url=https://mydomain.com/oauth/authorize?client_id=console&redirect_uri=%2Fconsole%2Foauth%2Fcallback&response_type=code&state=OrQvGCe9SRzNAMaG
    stack_1     |   INFO Request handled                          duration=1.480533ms method=GET namespace=web remote_addr=10.2.80.39 request_id=01ER02XJ56F4981SHFG7QB9Y9M response_size=846 status=200 url=https://mydomain.com/oauth/login?n=%2Foauth%2Fauthorize%3Fclient_id%3Dconsole%26redirect_uri%3D%252Fconsole%252Foauth%252Fcallback%26response_type%3Dcode%26state%3DOrQvGCe9SRzNAMaG
    stack_1     |   INFO Request handled                          duration=11.699325ms method=GET namespace=web remote_addr=10.2.80.39 request_id=01ER02XJ6DF2TR59N4F2ZM201Q response_size=95755 status=200 url=https://mydomain.com/assets/oauth.c475f2586d4d38b3f6a3.css
    stack_1     |   INFO Request handled                          duration=47.449028ms method=GET namespace=web remote_addr=10.2.80.39 request_id=01ER02XJ6D76NJR4JEQBJ8KYW8 response_size=1629970 status=200 url=https://mydomain.com/assets/oauth.3eaa6be75c3c43aa14f9.js
    stack_1     |   INFO Client error                             duration=1.638817ms method=GET namespace=web remote_addr=10.2.80.39 request_id=01ER02XJWDDZSSFSMX2778Y6VH response_size=177 status=401 url=https://mydomain.com/oauth/api/me
    stack_1     |   INFO Request handled                          duration=400.577µs method=GET namespace=web remote_addr=10.2.80.39 request_id=01ER02XJWQKWV92FAG4KGSRPXY response_size=6239 status=200 url=https://mydomain.com/assets/oauth-favicon.svg
    stack_1     |   INFO Request handled                          duration=668.806µs method=GET namespace=web remote_addr=10.2.80.39 request_id=01ER02XJZPSW9X4Q2V73GGNF1B response_size=25520 status=200 url=https://mydomain.com/assets/source-sans-pro-v13-latin_latin-ext-600.117e12cdb861ed7356c805f6f515afbb.woff2
    stack_1     |   INFO Client error                             duration=32.678µs method=GET namespace=web remote_addr=10.2.80.39 request_id=01ER02XJZRKVJ92SV7HCCT26Q6 response_size=19 status=404 url=https://mydomain.com/assets/oauth-favicon.png
    stack_1     |   INFO Request handled                          duration=86.249µs method=GET namespace=web remote_addr=10.2.80.39 request_id=01ER02XK0BE4AK41R98FYKFRZT response_size=1096 status=200 url=https://mydomain.com/assets/oauth-favicon.ico
    stack_1     |   INFO Request handled                          duration=66.213893ms method=POST namespace=web remote_addr=10.2.80.39 request_id=01ER02Y53WCJCK41PS695AV0PE response_size=0 status=204 url=https://mydomain.com/oauth/api/auth/login
    stack_1     |   INFO Request handled                          duration=32.44624ms method=GET namespace=web remote_addr=10.2.80.39 request_id=01ER02Y56HF8YBV62BQ6NP5Z4W response_size=0 status=302 url=https://mydomain.com/oauth/authorize?client_id=console&redirect_uri=%2Fconsole%2Foauth%2Fcallback&response_type=code&state=OrQvGCe9SRzNAMaG
    stack_1     |   INFO Client error                             duration=1.045586957s method=GET namespace=web remote_addr=10.2.80.39 request_id=01ER02Y57S3F669S7Y9YASD6T7 response_size=1181 status=403 url=https://mydomain.com/console/oauth/callback?code=MF2XI.HTVSLP5DIYB4PMXGS6NFSSX7SS25LTUDYIFULCQ.5NLGJ7HUCUCUXVQSRHOAAF4VQS4YFKHXVNX3OQ3WT4MRHL3OY3MQ&state=OrQvGCe9SRzNAMaG
    stack_1     |   INFO Client error                             duration=4.737586ms method=GET namespace=web remote_addr=10.2.80.39 request_id=01ER02Y6W0E5C7PT6W2TPK8690 response_size=198 status=401 url=https://mydomain.com/console/api/auth/token
    stack_1     |   INFO Request handled                          duration=34.382µs method=GET namespace=web remote_addr=127.0.0.1 request_id=01ER02YZRJPBYWNN3PRNW7KF2Z response_size=3 status=200 url=http://localhost:1885/healthz/live

Is there a known clear way of debugging this issue or a way to test that the config is working as it is designed to work?

2

So far it has been The Things STUCK for me. Where am I going wrong with this? Anybody willing to help out please?

3

Ok, I have opted to purge docker-engine and docker-engine docker-ce & docker-ce-cli from the ubuntu vm to start this process again from scratch and this time round post all my steps. Please bear with me as I am learning about TTN as I go and I am no expert.

The objective - to have a fully working Things Stack CE installation running on a ubuntu 18.04 virtual machine and set up a private LoRaWAN network server using The Things Stack for LoRaWAN.

I am starting off with a Ubuntu 18.04.5 LTS box running on a LAN. It has a local IP address and is also assigned a public IP address and a domain name. The domain name A record has been updated to point to the public IP address of the host.

I want to make sure I can make requests to the host publicly. I decide to install apache2 so that I can access the host via http. Now when I visit my-domain.com from a seperate 4G connection I get the Apache 2 default page. Also when I connect to the host via ssh and run command curl -4 icanhazip.com on the cli, the result is my public IP address.

So this looks good to me because I want to meet the requirements in order to use Automatic Certificate Management on The Things Stack which requires a domain name publicly accessible for TLS certs from Let’s Encrypt.

4

Next I move on to Installing The Things Stack and I am following these instructions, I am assuming that I am on the right track and if I am lost, I would be delighted if anyone lends a hand to help me along.

5

Prerequisites

  1. A server with a recommended 4 virtual CPUs and 16GB RAM running Docker and Docker Compose*
  2. DNS records pointing to your server’s IP address

My checks from previous post where I did a few things to make sure I can make requests to the host publicly actually satisfy prerequisite number 2. DNS records pointing to your server’s IP address.

To satisfy prerequisite number 1; I am following the Docker documentation suggested in the instructions.

On following the Docker ubuntu installation docs, I ran the following command:
sudo apt-get install docker-ce docker-ce-cli containerd.io

This installs the highest possible version as I did not see anywhere on The Things Stack installation that instructs to install a specific Docker version.

I complete this step by running sudo docker run hello-world which downloads a test image and runs it in a container. The result is:

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Next I also follow the Docker docs to install Docker Compose; and run docker-compose --version which results: docker-compose version 1.27.4, build 40524192

At this point I am assuming Prerequisites 1 & 2 are both satisfied.

6

Hi @Tigere, you state that your aim is…

This forum is for the TTN community network that uses shared (not private) gateways and servers. The TTN is currently running the V2 software. So, you probably won’t get any detailed support here unless one of the developers of the TT Stack responds. TTI offers commercial support for their products used to build private networks.

7

Hi @cultsdotelecomatgmai, am I confused? because the getting started installation guide has the following notice:

Enterprise and Open Source Only

The following instructions apply to Enterprise and Open Source distributions. See the downloads page for a feature breakdown of distributions of The Things Stack.

This is a guide for setting up a private LoRaWAN network server using The Things Stack for LoRaWAN.

Am I missing something here?

8

Hi @Tigere,

The Things Network (TTN) is a PUBLIC LoRaWAN network run for the common good of all users.
Your network will be a PRIVATE LoRaWAN network.
This forum is for the users of the public network.

The origin of the software used on both public and private networks is The Things Industries (TTI). The TTI software is open source so you can download it and use it yourself. TTI also offers commercial support, hosted services and Amazon images because - as you are finding out - engineering and operating a LoRaWAN core is a complex business.

9

Actually the forum is for support of the open source software as well. However there are far fewer forum users working with the open source stack. As a result knowledge about what might be the OP issue is scarce.

1 Like
10

Hi @cultsdotelecomatgmai, Thank you very much for clarifying this for me.

11

Thank you very much @kersing.

12

Hi @kersing and @Tigere,

Mr. Kersing is, of course, completely correct, my mistake. As the public TTN currently operates mostly on V2 of the software and as very few forum users operate private networks it is almost entirely about the public network whereas you are trying to build a V3 private network.

13

Update:

I now have can access the console but with a hack for now, i.e I have edited all occurrences of https:// to http:// in my ttn-lw-stack-docker.yml . This is enough to move on in a test environment while I dive deeper into this issue with Token exchange, SSL, ubuntu 18.04 and Things Stack running in docker container.

I will get to the bottom of this and I shall keep you posted.

image

14

Hey @Tigere, I’ve just managed to get https with a custom certificate working on my open source TTS server, and it looks like you want to use HTTPS, so I’ll post instructions here.

First, download docker-compose.yml and ttn-lw-stack-docker.yml and configure them as normal. In my case, this meant replacing thethings.example.com with the static ip address of the machine hosting my docker containers. Then, uncomment the sections about using custom certificates.

At the bottom of docker-compose.yml:
image

Near the top of ttn-lw-stack-docker.yml (don’t forget to comment out the let’s encrypt section):
image

Next, install cfssl and cfssljson:
go get -u github.com/cloudflare/cfssl/cmd/cfssl
go get -u github.com/cloudflare/cfssl/cmd/cfssljson

You can run the installed binaries from ~/go/bin or add them to your PATH.

You are now ready to follow the instructions for using a custom certificate. I created ca.json exactly as shown. For cert.json, I replaced thethings.example.com with the static ip address of the host (as I had done in docker-compose.yml and ttn-lw-stack-docker.yml). After creating the two json files, I ran the commands to generate the certificates and put the relevant files into my project:

cfssl genkey -initca ca.json | cfssljson -bare ca
cfssl gencert -ca ca.pem -ca-key ca-key.pem cert.json | cfssljson -bare cert
mv cert-key.pem tts/key.pem
mv cert.pem tts/cert.pem
mv ca.pem tts/ca.pem

Like the instructions say, your project directory (tts in my case) should look like this:

.
├── blob
├── ca.pem
├── cert.pem
├── config
│   └── stack
│       └── ttn-lw-stack-docker.yml
├── docker-compose.yml
└── key.pem

Finally, add ca.pem to the certificate store(s) on any machines that will be interacting with your TTS installation.

sudo cp ca.pem /usr/local/share/ca-certificates/ca.crt
sudo update-ca-certificates

You can discard the other files created during the certificate signing process - you no longer need them (ca-key.pem, ca.csr, ca.json, cert.csr, cert.json).

Now, run the things stack as normal and you should be good to go.

15

By the way, can you explain how you’ve set up the redirects to allow logging in to the things stack? When I log in using the console I get HTTP 403 Forbidden due to an invalid redirect URL. If I log in using ttn-lw-cli I can see that it’s basing all the URL paths off localhost, which is not gonna work. I’ve followed solution 1 in the config instructions for a local server (and tried solution 2 with the same result) so I’m not sure why the redirect isn’t working.

EDIT: For more detail, here’s my attempt to log in using ttn-lw-cli --redirect=false. You can see it’s trying to use localhost, which is incorrect:

/ $ ttn-lw-cli login --callback=false
INFO Opening your browser on https://localhost/oauth/authorize?client_id=cli&redirect_uri=code&response_type=code
WARN Could not open your browser, you'll have to go there yourself error=exec: "xdg-open": executable file not found in $PATH
INFO After logging in and authorizing the CLI, we'll get an access token for future commands.
INFO Please paste the authorization code and press enter
> MF2XI.BRLWSW36IE6UTIBFLH2X5FHM7ULP4SH3NERMY3A.4UOKLQWNJP7QKBZ7XYN7MN4EWB2FGTSRUTKI673BS236XAKZ3CZQ
ERROR Could not exchange OAuth access token    error=Post "https://localhost/oauth/token": dial tcp 127.0.0.1:443: connect: connection refused
Post "https://localhost/oauth/token": dial tcp 127.0.0.1:443: connect: connection refused
16

This thread appears to have been neglected. However, further information on the Token Exchange Refused Error can be found here: TheThingsStack V3.8.7 with self-signed certificate -> Forbidden Token exchange refused

17

Note: one command is missing from my post above about setting up TLS. After creating the certificates and placing them in the project directory, run sudo chown 886:886 ./cert.pem ./key.pem

2 Likes
18

Can you post the results of ttn-lw-stack config | grep localhost ?

19

Thanks for pointing this out - I’ll add a note in the documentation

20

Thanks mate, if you could add that to the documentation would be great. I ended up getting it working after I wrote that post using instructions from IronNinja in the other thread I mentioned. There are a few loose ends to tie up but it will be easiest to just do it all in that thread, so I’ll post there with a full explanation.