I’ve had a little VPS chugging away for a bit more than a year now. It runs this site, it runs calibre-web, miniflux, wireguard and a bunch of other stuff.

This is a very much a “pets not cattle” machine, to reverse the common trope about treating servers as cattle rather than pets. I log into it directly. I administer to it by typing commands out and manually editing a bunch of files in /etc. I do have a couple docker containers on there but generally I try to avoid running things through docker if I can, I find them initially convenient but long term more annoying to maintain and debug.

Old skool.

I know that with the likes of ansible and/or nix one could make the whole more declarative and automated but honestly, I only have the one VPS, I’m not sure it makes sense.

My backup strategy is… ok. I don’t fully follow the 3-2-1 rule¹ because, again, I’m lazy and this is a hobby. I have restic setup to backup /etc and my home directory every night. It keeps a daily copy, a weekly copy and a monthly copy. I occasionally stick a static copy of some data I really would hate to lose in my google drive or on an external drive. But I don’t keep these that up to date. It’s good enough, no one will die if this data gets lost.

Recently something happened and the VPS started refusing to boot, I spent a few hours trying to fix GRUB and other such things but I had been meaning to reinstall and clean up this machine a bit anyway so decided now was a good time to give up on GRUB, as I have done oh so many times before.

So what follows is a rambly account of what I installed, probably only of interest if you also self host stuff.

Rambles

First I went a did some spot checks on the backup, restored a few files to my local machine and made sure they had contents. Then I took the plunge and wiped the VPS. I mounted a live CD and went through the painful process of installing Debian², only once that was done did I realise that Netcup provides ready made images I could have just used instead, oh well!

I logged in, copied my ssh keys over, turned off password logins with SSH. Installed sudo… because apparently that doesn’t come installed by default?!!?!? I made a user and added it to the sudo group. Amusingly, git also does not come as standard.

Controversially, I didn’t disable root login via ssh because… please don’t tell anyone, I quite like to login as root with vscodium and just open and edit whatever config files I like. Bit naughty I know.

I enabled unattended upgrades and, new to me, automatic reboots. This could be what caused the original boot issues but recovering from the backups is going fine so far so YOLO.

I didn’t want to restore all of /etc because I didn’t want to bring over unnecessary cruft, I only restored individual files as and when I needed them. For example to get back my config for [caddy], I did: restic -r $RESTIC_REPOSITORY restore latest --target / --include /etc/caddy/Caddyfile and boom it appeared!

I have a scrappy markdown file where I keep notes on each service running on the box, this usually includes the path to important config files and the systemd service definition. So I went through my list of services, first restoring their config, installing them through apt, checking or updating their service definition, and they getting them running.

Set RuntimeDirectory=… for unix sockets

The only real modification I’ve made to most of the service files, except the ones I wrote from scratch, is to add RuntimeDirectory=<NAME_OF_THING>, this gives you a dir at /run/<NAME_OF_THING> which is convenient as a location to put a unix socket.

Anything that I’ve got reverse proxied behind Caddy I’ll try to serve from a unix socket like this.

To use miniflux as an example:

1) miniflux.service has the line RuntimeDirectory=miniflux under [Service] which makes /run/miniflux with owner miniflux:miniflux. 2) miniflux.conf has a setting /run/miniflux/miniflux.sock which tells it to listen on that socket 3) I add the caddy user to the miniflux group so caddy can read the socket. 4) Caddy has a block that associates a subdomain with that socket:

miniflux.thod.dev {
    <extra authentication goes here>
	reverse_proxy unix//run/miniflux/miniflux.sock
}

Run your own postgres

Postgres is pretty easy to install and run. A lot of self hosty-type software has the option to either use a local sqlite or bring your own postgres. So I bring my own, I have one postgres db on the box supplying miniflux, linkding, [forgejo] and some dynamic vector maps underpinned by [pg_tileserv].

I’m not sure whether I would recommend this or not but I like that I can backup and restore the whole database easily, and occasionally I have a poke around inside the date stored by each thing. For example, I used this to steal miniflux’s full text search (which is really just postgres FTS) and integrate it elsewhere.

I use pg_dumpall -U postgres | gzip > /root/full_postgres_database_dump.sql.gz to dump the database to a file and then let restic back that file up.

Bare repos and SHA256

I serve a few static websites from this box, the one you’re reading as well as some private ones. They work like this, there’s a bare git repo for each website sitting on the box that I update with git via ssh.

Each has a post-receive script that checks the repo out and builds the site each time I push a commit.

I find this setup much more responsive than a github action. The only downside is that if the build fails I have to make a fake commit to trigger another build or login to the server and manually trigger a build.

Oh and these bare repos use the newer SHA256 hash. This may have been a mistake. I recently tried to use git filter-repo and git filter-branch on one of the repos and could get neither tool to work! I suspect SHA256 to be the culprit.

Things currently running on this VPS

• Caddy - webserver and swiss proxy knife
• Authelia - authentication
• Postgres - general db duties
• Miniflux - rss reader

Still to be restored

• Reinstall fail2ban to temporarily ban IPs doing obvious scraping and ssh have-a-goers
• Install docker and docker compose
• Install wireguard
• Modify backup script to ignore all .venv dirs
• Install NTFY