We were facing many issues with a docker TTN v3.16.3 setup, TTN was restarting many times, so I went ahead with upgrading it a few times first to 3.19.0 then 3.24.0 when it complained about outdated DB same time I upgraded redis to 7.2.4 and cockroach to latest-v21.2, when I also did the following each time:
docker compose run --rm stack is-db migrate
docker compose run --rm stack ns-db migrate
after I also pull and run the TTN stack to recreate container.
The problem is even after upgrading to 3.29.0 we now have a stable TTN that doesn’t restart but we can’t login or work with it since we get driver error (that’s even after doing is-db migrate)!
I also attempted AOV fix utility and still not working out.
I also have a backup of redis DB from 3.16 that is 33GB in size so far.
Would appreciate any help you can give me to fix this problem.
We have less than 700 devices at this point and all were sending uplinks every hour at start last year we reduced it to every 4 hours we also later on reduced to 12 hours - besides that we also have downlinks but very few around 1 or 2 per month for controlling device, and we are using 2 gateways. In total, they have been connected for +2 years.
A webhook is also setup that sends decoded packets to the Application Dashboard.
TTN is TTS Sandbox but was TTS Community Edition. You have TTS Open Source - the self-hosted version.
If you only have 700 devices then you may consider that the pain you are currently experiencing and the general maintenance of the setup plus the hosting could all go away for around one & a day by using a TTI instance - @rish1 can advise further. Any reasonably well paid engineer taking more than 2 hours a week on SysOps would be time saved - and I can’t imagine that you can run an instance without at least an hour of checks & log reviews & hosting status as a matter of routine.
In the meanwhile, a 33Gb Redis seems rather large and all sorts could have happened and updating could be timing out - you could try exporting the devices and importing them in to a new setup.
What is the spec of the server in terms of CPU & memory? I’m assuming it has lots of disk space for the database!
Former, only contributes to TTN and therefore part of it if fully peered with the TTN LNS infrastructure such that open traffic is possible over your infrastructure (via Packet Broker). There are (too?) many private instances where peering doesnt exist or if in place connections are blocked/neutered such that the private instance gets benefit of traffic using the TTN deployments of GW’s but where the TTN Community doesnt get access to private deployments and therefore - even if showing in the stats and on TTN global map - are not actually TTN resources.
Latter, is not TTN as TTN is not just s/w!
TTN is the holistic collection - the hardware, the software, network connectivity, backend infrastructure, associated server software (TTS, Join Server, Application Server, Integration servers & connections), and most importantly the TTN Community instances and contributors - the meatware and meetware if you will We are TTN. The Things Network runs, sustains and hopefully grows by the efforts of its contributors, it utilises an underlying infrastructure and the software stack that underpins that infrastructure is The Things Stack. The version of TTS that is utilised is TTS-Sandbox, formerly TTS(CE) - Community Edition. The software is not the network
Sorry about the confusion with the proper name, I sometimes use both names interchangeably, which is wrong!
With regard to server spec it’s got a 12 core Xeon CPU with huge space available 2TB and 64GB RAM the system should be more than enough to handle it I’m sure.
In regard to the problem I mentioned, what advice can you give me? What possible procedures can I try regards sorting the database without resorting to a new setup.
My emphasis! Do you know what the actual system requirements are for TTS OS - or more specifically, what the recommended ones are?
Sorry, no specific ideas, I run TTS OS in house for a handful of devices and I’m relatively on top of updates after the Cockroach migration fun.
If you chose to pay for support with TTI they may be able to help you.
But I suspect that anything you try is likely to be harder than just exporting the devices with session and importing them in to a new instance - that’s all done for you with the CLI and it’s all documented as well!
My primary recommendation is to go to a TTI instance - I’d be surprised if that doesn’t save you money and free up your enormous server to do something else.
Thanks for all the advice, I did try to use command line but to export I do need to login right? and when I do attempt that, I still get the driver error and remain stuck as in screen shot I’m sharing.
If I recall from previous issues raised, this is best done by moving to the first version that uses Postgres as jumping too many versions didn’t always co-operate.
If I recall from previous issues raised, this is best done by moving to the first version that uses Postgres as jumping too many versions didn’t always co-operate.
That’s indeed correct.
You should run is-db migrate. You don’t need to recreate the admin user since that’s part of the existing database.
inside the cockroach db I recovered it and can export the devices table however the appkeys are not included in the table fields where does it end up saved?
& 
a day by using a TTI instance - 
We are TTN. The Things Network runs, sustains and hopefully grows by the efforts of its contributors, it utilises an underlying infrastructure and the software stack that underpins that infrastructure is The Things Stack. The version of TTS that is utilised is TTS-Sandbox, formerly TTS(CE) - Community Edition. The software is not the network 
