Soap boxing here but I feel these kinds of use cases is what NixOS is built for.
Declarative config to setup the system, users, and apps.
Declarative and customizable impermanence exactly how you want it.
I use Tails as well but NixOS is my daily driver. Anything not marked explicitly to persist is dropped each reboot. I’m the only user so I keep the last 30 days of non persisted data for like a power outage but that’s something I had to go out of my to setup for my use case.
For our lower environments we use rsync like the author but skip the pipeline altogether. The servers have a watch script to restart when files are rsynced. We then have a local watch script that rsyncs on file changes.
Relatively instant deploy (2-5s) whenever a file is saved.
Good call on a simulated failure. When I first set it up, it was LVM/BTRFS or ZFS as my top choices. It was a coin toss at the time because I hadn’t built this sort of setup before.
I use immutable nixos installs. Everything to redeploy my OS is tracked in git including most app configurations. The one exception are some GUI apps I’d have to do manually on reinstall.
I have a persistence volume for things like:
I have 30 days (or last 5 minimum) of system rollbacks using BTRFS volumes.
The personal files are backed up hourly to a local server which then backs up nightly to B2 Backblaze using rclone in an encrypted volume using my private keys. The local server has a mishmash of drives in a mirrored LVM setup. While it works well for having mixed drives, I’ll warn I haven’t had a drive failure yet so I’m not sure the difficulty of replacing a drive.
My phone uses the same flow with RoundSync (rclone + GUI).
Git repos are backed up in git.
Logs aren’t backed up. I just persist them for debugging and don’t want them lost after every reboot.
Caches/Games are persisted but not backed up. Nixos uses symlinks and BTRFS to be immutable. That paradigm doesn’t work well for this case. The one exception is a couple game folders are part of my personal files. WoW plugin folder, EvE online layouts, etc.
I used to use Dropbox (with rclone to encrypt). It was $20/mo for 2Tb. It is cheaper on paper. I don’t backup nearly that much. Backblaze started at $1/mo for what I use. I’m now up to $2/mo. It will be a few years before I need to clean up my backups for cost reasons.
The local server is a PC in a case with 8 drive bays plus some NVME drives for fast storage. It has a couple older drives and for the last couple years I typically buy a pair of drives on sale (black Friday, prime day, etc). I have a little over 30TB mirrored, so slightly over 60TB in total. NVME is not counted in that. One NVME is for the system, the others are a caching layer (monero node) or temp storage (transcoding as it also my media server).
I like the case, but if I were to do it again, I’d probably get a rack mountable case.
For the networking I found some repos with Nix and Gluetun (OCI containers). I don’t see them in my bookmarks, so it was probably a day project when I set up and didn’t keep the references.
That part is still in docker / podman. So any docker network guide just needs to be translated to nix.
Best resource I’ve found is searching GitHub.
My setup closely follows https://github.com/Misterio77/nix-config.
For servarr I just translated someone else’s docker compose setup to nix. There are some ready made nix ones you can look at like https://github.com/rasmus-kirk/nixarr/tree/main/nixarr.
The complex networking I just picked up over time once I knew my way around a little bit.
GitHub is your best resource. lang:nix search terms.
I wouldn’t run NixOS in a container. With native nix containers I’m pretty sure they share the store. For docker I’d use images built with nix (doesn’t run nix itself) or pull from docker hub.
OS: NixOS (high learning curve but its been worth it). Nix (the config language) is a functional programming language, so it can be difficult to grok. Documentation is shit as its evolved while maintaining backwards compatibility. If you use the new stuff (Nix Flakes) you have to figure what’s old and likely not applicable (channels or w/e).
BYOD: Just using LVM. All volumes are mirrored across several drives of different sizes. Some HDD volumes have an SSD cache layer on top (e.g., monero node). Some are just on an SSD (e.g., main system). No drive failures yet so can’t speak to how complex restoring is. All managed through NixOS with https://github.com/nix-community/disko.
I run stuff on a mix of OCI containers (podman or docker, default is podman which is what I use) and native NixOS containers which use systemd-nspawn.
The OS itself I don’t back up outside of mirroring. I run an immutable OS (every reboot is like a fresh install). I can redeploy from git so no need to backup. I have some persistent BTRFS volumes mounted where logs, caches, and state go. Don’t backup, but I swap the volume every boot and keep the last 30 days of volumes or a min of at least 10 for debugging.
I just use rclone for backups with some bash scripts. Devices back up to home lab which backs up to cloud (encrypted with my keys) all using rclone (RoundSync for phone).
Runs Arrs, Jellyfin, Monero node, Tor entry node, wireguard VPN (to get into network from remote), I2C, Mullvad VPN (default), Proton VPN (torrents with port forwarding use this), DNS (forced over VPN using DoT), PiHole in front of that, three of my WiFi vlans route through either Mulvad, I2C, or Tor. I’ll use TailsOS for anything sensitive. WiFi is just to get to I2C or Onion sites where I’m not worried about my device possibly leaking identity.
Its pretty low level. Everything is configured in NixOS. No GUIs. If its not configured in nix its wiped next reboot since the OS is immutable. All tracked in git including secrets using SOPS. Every device has its own master key setup on first install. I have a personal master key should I need to reinstall which is tracked outside of git in a password manager.
Took a solid month to get the initial setup done while learning NixOS. I had a very specific setup of LVM > LUKS encryption /w Secure Boot and Hardware Key > BTRFS. Overkill on security but I geek out on that stuff. Been stable but still tinkering with it a year later.
Download 5 seasons of some show from multiple sources or some artist’s entire discography, and want to normalize all the file names? It is way easier in the terminal.
I’ll check this out, but I use https://github.com/stevearc/oil.nvim for such tasks as I have nvim’s full suite of editor commands to rename all the files way faster than I could in a GUI. I’m sure there are GUI apps to perform a similar task, but I already know how to use nvim.
I haven’t made a keyboard in awhile but anything that supports QMK (or whatever is new and shiny today) should be able to support this.
QMK and the like are custom firmware so you can pretty much code up whatever feature you need.
If you are looking for a pre-built, I know my Tofu65 supports QMK from https://kbdfans.com/.
QMK is written in C but they do have a no code tool I used for my Tofu65: https://config.qmk.fm/#/.
If the tool doesn’t cover your use case and you are able to do a little C, these sections are good starting points for layers (what you call modes) and cursor keys.
Won’t auto update but you could add the upgrade command to a login script or something.
Won’t lie, nix has a high learning curve to get the most out of it, but installing a single app is pretty simple.
Most startups I’ve applied to are Linux friendly.
I currently work for a fortune 100 and managed to get a Linux machine purchased as a “lab” machine.
I’m fully in control. IT doesn’t even know it exists. I’m not allowed on the corporate network, but I managed to get some internal corporate access through another department’s lab network (IT sanctioned) that has a VPN with a few routes to things like ticketing, time cards, and our internal wiki. Most of the stuff I need to do my job is in AWS and we are allowed to add home IPs to the security groups.
IT still gives me a MacBook. I use it like once every 6 months.
nixos-unstable is the only thing I will use currently.
I’m running bleeding edge stuff like the latest kernel, Hyprland nightly, my own “shell” built from Gnome components and lots of custom stuff using GJS (Gnome JavaScript).
If you get one, and you are free to do whatever on it, encrypt your drives like your job depends on it. I have a memorized passphrase, pin protected hardware key, and a key in TPM. No biometrics.
As far as other nice things to have:
I work in software dev as FYI. For the few issues I have, my team has more issues getting stuff working consistently on macOS for our project. I used that as a justification when requesting the laptop: my dev environment should closely match our runtime environment. Most of that is moot now since we use Nix flakes in our repos for local dev envs.
Yeah I don’t want locally deleted media (to free up space) to sync those deletions to my remote.
My crypted remotes wrap a B2 Backblaze one which doesn’t delete, just hides. Periodically I go clean it up.
You are correct, fixed!
https://github.com/newhinton/Round-Sync. Not in any app store and have to download and install from GitHub.
It is an Android wrapper around rsync rclone.
Setup a remote, setup tasks, and setup triggers. Mine syncs every night. It supports encrypting with your own keys. Large number of remotes supported from self-hosted to cloud.
Immutable Nixos. My entire server deployment from partitioning to config is stored in git on all my machines.
Every time I boot all runtime changes are “wiped”, which is really just BTRFS subvolume swapping.
Persistence is possible, but I’m forced to deal with it otherwise it will get wiped on boot.
I use LVM for mirrored volumes for local redundancy.
My persisted volumes are backed up automatically to B2 Backblaze using rclone. I don’t backup everything. Stuff I can download again are skipped for example. I don’t have anything currently that requires putting a process in “maint mode” like a database getting corrupt if I backup while its being written to. When I did, I’d either script gracefully shutting down the process or use any export functionality if the process supported it.
I use rclone and the Round Sync Android client.
Supports a ton of back ends, self hosted, and commercial options. You can transparently encrypt with private keys you control.
I personally use B2 Backblaze for storage.
My phone backs up every night and Round Sync pushes them to B2. On my desktop I can mount as a volume. I can also access my storage from my phone going the other direction.
I’ve done the same using SFTP if I don’t want the overhead of persistent file storage.
It does not support indexing or previews for searching or finding say a photo. You can put whatever you want for data. So I have caches, indexes, and thumbnails that work in Linux. I can’t really make use of those on my phone though.
Rclones bisync feature is also a bit dangerous when I tried to use it a year ago. I more than once “deleted” everything. B2 doesn’t delete by default, just hides, so I was able to recover. I now do unidirectional syncs from my machines to different buckets until I’m motivated to investigate a proper 3-way merge solution.
I haven’t tested in Windows, but this is my setup Linux to Linux using rclone which the docs say works with Windows.
Server
Client
I use this setup for my local files and a similar setup to my Backblaze B2 off site backups.
The VFS implementation has been pretty good. You can also manually sync. Their bisync I don’t fully trust though.
I can access everything through android using https://github.com/newhinton/Round-Sync. Not great for photos though as thumbnails weren’t loading without pulling the whole file last I tested a year ago.
And track this stuff in git so you don’t need to remember how you did it when you inevitably forget, lol.
No. Its all text file config. You wouldn’t use live CD mode. You define your own how you want it to work.
Its a steep learning curve so if looking for off the shelf solutions, don’t use nix. If you need something custom but through a single config paradigm, nix is awesome.