One of the Raspberry Pi’s weaknesses is its microSD flash card that serves as its main storage.
The lifetime of Flash cards is limited by the maximum number of writes per memory cell.
A flash card has an on-card memory controller that performs wear leveling management. This tries to spread write actions over all memory cells - through time - evenly, to prevent premature partial wear out.
The wear levelling unfortunately does not completely prevent microSD cards in Raspberry Pi’s (and similar devices) from wearing out.
One way to minimize flash card wear out is to use a ram drive for the storage of log files that are frequently written to. I have seen several solutions for this but most of them lacked a proper explanation.
I am looking for articles that properly explain:
Which ramdrives are available that can be used on a Raspberry Pi?
What are the differences between these ram drives? I would assume that a standard ram drive would be included with the OS but that may just be wishful thinking.
Which ramdrive(s) is/are preferred and why?
In examples I have seen that only the systemd journal log was written to ram drive.
However, there are more logs written on Linux systems. What about other logs? Is it not necessary to write other logs to ram drive as well (and which and how)?
How to properly dimension the ram drive, how to determine the preferred/best size?
How to properly configure log sizes and rotation?
How and when to make a copy of the log from ram drive to flash card to prevent logs from vaporizing at power down?
On regular intervals, on power down only (probably not because loses logs in case of power failure), when the log reaches a certain size?
Who can help?
It will be useful if we can collect easy to apply recipies, tips and (links to) articles here to help prevent flash card wear out for Raspberry Pi (and similar microSD flash card based) gateways.
There’s a few bits of discussion on the DietPi forums at https://dietpi.com/phpbb/viewtopic.php?t=2794 , Diet Pi is a lightweight OS Image which has some tweaks such as logging to RAM as easy to enable features.
The other alternative I know of is https://github.com/ali1234/rpi-ramdisk , this with the right compiling and configuration loads the entire OS into RAM. I’ve been wondering about creating a gateway image using this as this should be the most reliable as the SD Card can even be taken out once booted.
Hi @bluejedi - I agree with you; the issue of flash failure due to excessive write cycles is a critical issue where any operating system is run on typical NAND flash devices, including SSD drives. Here is an authoritative description of the problem and how it may be alleviated by wear levelling - https://www.micron.com/~/media/documents/products/technical-note/nand-flash/tn2942_nand_wear_leveling.pdf
Looking at running linux on a Raspberry Pi where the storage is a micro SD card, and the action of updating files in /var/log (for example), it is entirely likely that the card will fail with 12-18 months of continuous use.
One quick and easy solution is to run /var/log in a RAM drive and then perform an update to flash on a regular basis - I tend to use log2ram - https://github.com/azlux/log2ram
A USB flash drive has the same wear issues as a flash card. Won’t make a difference.
An SSD is nice but much larger and (much) more expensive (still a NAND flash device but with probably much better memory controller).
I’m still looking for a compact and relatively cheap SSD solution for Raspberry Pi (not a hat implementation). Good tips are welcome.
Unfortunately SSD (and also network) throughput will be limited by USB 2.0 speeds, even with the latest RPi 3 model B+.
Using a remotely managed solution that uses it’s own OS is indeed one way to minimize flash wear but a quite radical solution if only for reducing wear of a flash card. And not free (for >10 devices).
There are ready to use recipes to install and build the software, the default setup is more secure than most of the RPi based instructions for a gateway I’ve seen, it allows for remote management without requiring complex setups for NAT traversal and it is free for the first 10 devices (from the pricing page: Your first 10 devices are always free and full-featured)
A lot of positives. I do understand it is not everyone’s cup tea, but when talking about easy setup it is hard to beat.
I don’t myself and will be using Raspbian lite for the basis of my Raspberry Pi Gateway SD Card Image I’m building. However have heard good things about it.
I use dietpi with both original Raspberry Pi and the nanopi devices from FriendlyArm. I have two LoRaWAN gateways, one based on the original Multitech LoRa PCIe device and the other on the RAKwireless 831 module both running dietpi on nanopi with log2ram installed.
I use dietpi because I know that someone better than me has built the distribution with kernel tailored for the specific hardware. I use log2ram based on my experience with NAND flash wear-levelling going back with thin clients over 20 years - and I like muffins!
I run my own fork of JPMeijers’ setup on my gateways, either with resin.io or plain docker – both have pros and cons, but eventually it is the same code base.
For the resin.io setup, it is easy, their image minimize writes – nothing to do.
For the plain docker setup, I set the docker log driver to journald, that way all the logging ends up in the same place (Including the packet forwarder logging).
I am using the following configuration in /etc/docker/daemon.json:
Now that most volatile data is in /var/log, I use log2ram and go for the muffins.
I don’t do any backup, resin.io re-installs itself in minutes on a new SD-Card and for plain docker I am using Ansible scripts to pre-configure raspbian so it does not take longer.
No incident so far, but I only started 6 months ago.
Aside TTN, I have quite some RPi’s running since years, but none of them actually writes on SD-Cards. I am using one of these solutions:
Completely read-only filesystem (doable but not very practical for TTN)
Network boot from NFS server (works well, but you need an NFS server and your node must stay local)
USB SSD or HDD (E.g. WD PiDrive) (Also very good, but it only makes sense if you need more storage)
My experience with sd cards and USB drives on the Pi does not agree with your comment about not making a difference. I found that pi’s writing to their SD cards would typically fail after 6 months of continuous service as a data logging system. I modified the software such that the Pi performed only the initial boot from the SD card with the file system being installed on the USB drive and all logging to the USB drive (No RAM disk). I cannot claim any credit for this as it is well documented elsewhere. After doing this change I have had many years of service without an sd card or USB failure from these systems.
It may be that the wear leveling on the USB media is superior or it may be because the USB media was larger than the sd cards and therefore the wear leveling has more media to operate over increasing the apparent life of the media.
There many factors that play a role here so it may be difficult to generalize. I think probably only an expert in different flash technologies and -media could explain the exact differences and their effects on practical use.
If I remember correctly there’s different methods of interfacing with SD Cards as well. Of which I believe the Raspberry Pi uses the SPI Method rather than a SD Bus Mode due to the extra costs they incur. I could be incorrect.
This could also affect.
I wouldn’t be surprised if physical size is also a factor. Most USB Keys the flash memory on them is bigger than a Micro SD Card.
We can now with Raspberry Pi 3s boot off a USB drive with no SD card required or for even better results you could run the pi off a Network Boot now which would be the best for repair ability.
