笔记

周五晚上,我用一个看起来毫无危险的参数清空了客户的生产库

On a Friday evening, in the middle of dinner service, I emptied the production database of the restaurant whose entire ordering operation runs on software I wrote. Orders, menu, customers, the lot.

It came back. This is what happened, why the thing I thought was protecting me wasn't, and the one question I now ask before typing any database URL anywhere.

The command

I was adding a migration. Prisma needs a second, throwaway database to diff schemas against — the shadow database. It creates it, uses it, and drops it. Normally it provisions one for you; in this setup it needed to be passed explicitly, so I passed one:

npx prisma migrate dev \
  --shadow-database-url "postgres://…"   # ← I put the production URL here
shadow database 不是副本,是工具有权销毁的临时空间。

The URL I pasted was production. There was no confirmation prompt, and there shouldn't have been — from the tool's point of view I had just told it where its scratch space lives, and it did exactly what it is documented to do with scratch space.

That is what makes this class of mistake dangerous. Nothing malfunctioned. Every piece of software behaved correctly. The damage was entirely in which string went in which slot.

Why the habit that usually saves me didn't

I have a rule about never pointing anything at production. What I actually had was a mental blacklist of the commands I consider dangerous — drop, reset, truncate, --force. I check those.

--shadow-database-url was not on that list, because it doesn't look like a destructive flag. It looks like configuration. It reads like the sort of parameter that tells a tool where to find something, not what to flatten.

A blacklist can only stop the entries that are on it. The one that gets you is by definition the one you never thought to add — and the more innocuous a flag looks, the more likely it is to be missing.

So I replaced the blacklist with a question, asked of every parameter that takes a database URL, no exceptions:

Will this parameter write to, or clear, the database I'm about to name? If I can't answer that from memory, I go and read the docs before pressing enter.

It is a slower rule and a much better one. It applies to flags I have never seen before, which is precisely where the blacklist failed.

There was a second trap underneath

This wasn't purely carelessness. The production URL was the easiest string in the project to reach for, because it's the default value of DATABASE_URL in the API's own env file:

# apps/api/.env
DATABASE_URL="postgres://…prod…"   # this is production. it is the default.
本地开发默认连的就是生产库。这才是真正的根因。

When production is what your shell already has loaded, one careless paste is all it takes. The mistake was mine; the loaded gun on the table was a design decision I had made months earlier and stopped noticing.

The twenty minutes after

The database is on Neon, which keeps a continuous write-ahead log and can restore to any point in time within the retention window. I did not have to find a backup file, or hope a nightly dump had run, or know when the last one was.

Total downtime was about twenty minutes, during service, on the busiest evening of the week. Nothing was permanently lost.

What I changed

The honest ranking, most useful first:

# Refuse to run destructive tooling against anything that smells like prod.
case "$SHADOW_DATABASE_URL" in
  *prod*|*neon.tech*) echo "refusing: shadow db points at production"; exit 1 ;;
esac
加起来很便宜,能挡住这一次。但下一次换个形状的它照样挡不住。

The part worth keeping

I ran the command that did it. But the setup made it a one-paste mistake instead of a several-step one, and it stayed that way for months because nothing had gone wrong yet. Absence of an incident is not evidence that a setup is safe — it's the state every unsafe setup is in right up until it isn't.