I woke up this morning and did the morning routine, getting the kids to school, and getting myself to work. After getting settled, I got a notification from a friend that my blog was down. That couldn’t be, right? tried to load it and was greeted by an HTTP 502 from Cloudflare… crap. The worse part was that after a cursory look, I could tell that my database was corrupted and that this was going to have to wait until after work for me to fix and I’d have to go through my whole workday knowing my personal blog was down but that there wasn’t anything I could do about it.
Given the title of this post, you already know the outcome of this story, but let’s talk about how we got here and hopefully how you won’t be in the spot I was in this morning.
First, here’s a crudely drawn diagram of how you reach the ghost container:
In short, you hit cloudflare, go through my firewall, then hit my reverse proxy that is in front of all my internet facing containers (only Ghost shown). The firewall only allows cloudflare IPs on :443, nothing else, and secondarily nginx also only trusts cloudflare IP addresses and will deny anything else (defense in depth anyone?). The containers run on Unraid which among other things, runs Docker.
Every Monday at 2AM the Docker service is stopped and backups of all the applications are taken. The backups are of the appdata folder, and then they are saved as a tarball (tar.gz). Afterwards, a process checks for updates and installs them, before starting the containers again. These are all personal projects and 10 minutes of downtime at 2AM is perfectly acceptable for me as I’m sleeping, along with most of my users.
What happened this morning was that Docker shut down the containers, but the MySQL container that stores this blog’s database was not shut down cleanly, and it was corrupted (problem 1). The corrupted database was then backed up, and the previous backup was overwritten (problem 2). Updates were then installed, and most other services came back up. I would have noticed the outage right away when I woke up, but apparently there is a bug in iOS 18.2 beta 1 where gmail doesn’t sync properly, so I didn’t see the multiple notifications that the blog was still down.
Problem 1 is something I’m solving at the application layer. I will be making database backups more regularly. I will need to explore of there is a way to more cleanly stop containers in an automated way, but for now, if this happens, I can restore from a database backup as opposed to restoring the container appdata. Problem 2 is fixed by storing more than one backup. This is something I had always planned on configuring, but life with kids got in the way.
Unfortunately the way back machine only captured one blog post, and I lost my blog post about an identity centric cybersecurity program. I do still have the corrupted database and ghost has the posts table separate, so I hope that when I have time I can restore that one. Luckily, my interview blog post I still had open in a browser, so I took a screenshot of the content and was able to copy/paste from the screenshot into another blog post.
If you are reading this and don’t have multiple backups configured, now may be a good time to do so!
