The deploy that ate the database

Five releases went out in one day. Counting the player records in each release's database afterwards gave this:

Six people played, then nobody existed, then people existed again. Nothing in the logs, no exception, no failed deploy.

What atomic deploys actually share

The deploy model here is the standard one: each release is cloned into its own timestamped directory, a set of hooks runs, and a current symlink is flipped to point at it. The flip is the atomic part, and it is why rollback is instant.

For that to work, anything durable has to live outside the release directory, in a shared/ folder that every release links back to. The deploy tool shares two things by default: .env and storage. That covers most Laravel applications, because most of them keep their data in MySQL or Postgres, somewhere else entirely.

This one runs SQLite. DB_CONNECTION=sqlite, database at database/database.sqlite, which is inside the release directory and was not in the shared list.

So every release got its own database file. Visitors wrote to it perfectly happily until the next deploy flipped the symlink, at which point their rows were still on disk, in a directory nothing pointed at any more, and the site was talking to a brand new empty file.

Why it was silent

Every layer behaved correctly.

SQLite has no server to refuse a connection. The database is a file path, and a new path is a new database rather than an error. Laravel does guard the obvious case — a missing file raises SQLiteDatabaseDoesNotExistException before you get anywhere — but that check fires on absence, not on emptiness. An empty database is a perfectly valid database. Zero rows is a perfectly valid answer to "how many players are there".

So the site worked. It just worked against nothing.

The numbers going up and down are the whole signature. A fresh release starts at zero and accumulates while it is live, so the count climbs back on its own and the loss reads as quiet traffic rather than as data going missing. It was found by counting rows per release directory, not by anything alerting.

Eleven stranded player rows were recovered from the old release directories and merged into the shared database.

The fix

A hook that runs before activation, so a release is pointed at the durable file before it can ever serve a request against its own copy:

DEPLOY_HOOKS_BEFORE_ACTIVATE: |
  bash deploy/link-shared.sh
  php artisan migrate --force
  npm i && npm run build
  ...

The script resolves shared/, creates the database file on first run, removes the release's own copy and symlinks it:

rm -f "${release}/database/database.sqlite"
ln -s "${shared}/database/database.sqlite" "${release}/database/database.sqlite"

chmod 775 "${shared}/database"
chmod 664 "${shared}/database/database.sqlite"

That chmod on the directory is not decoration. SQLite writes -wal and -journal siblings next to the database file, so the containing directory has to be writable, not just the file. A read-only directory produces attempt to write a readonly database at the first insert, which at least has the courtesy of being loud.

Three details worth stealing:

Fail closed. If shared/ cannot be found, the script prints why and exits non-zero, taking the deploy down with it. A release that starts with an empty database should never reach activation. The failure mode being fixed here is silence, so the guard against it has to be noisy.

Put it in a file, not in the YAML. The deploy tool may run hook lines individually, which drops shell variables between them. A script gets one process and one set -euo pipefail.

Check what else the application writes. The audit that follows this bug is "what does the running app create that the git clone does not own" — uploads, generated files, file-driven cache and sessions, and in this case the database. Everything in that list belongs in shared/.

The same commit added php artisan migrate --force, which had never been in the deploy at all — schema changes had only ever been applied by hand. It runs immediately after the link step, which is the right order: migrate the durable database, not a per-release copy of it.