Rotating a Live Credential Without Losing Any Data
Pasted a live database password into a screenshot today. Not great.
No dramatic reveal here - no breach, no attacker, nothing clever. Just a screenshot taken mid-session that had a real credential sitting in it, out in a place I didn't intend for it to be. The interesting part isn't the mistake. It's what "fixing it correctly" actually requires when the credential in question is attached to a database that's already holding real, ingested data.
Why you can't just change it
The naive fix is: edit the password, done. But a Postgres role's password isn't a standalone secret - it's the thing every running connection is authenticated against, right now, including the containers actively serving traffic. Change it in one place and forget the others, and you don't get a clean rotation. You get an outage that looks like a mystery until you remember what you just did.
So rotation here means three things happening in the right order, not one:
1. Change it where the database actually lives
ALTER ROLE radar_admin WITH PASSWORD 'new_password_here';
Not editing a config file first. The database itself is the source of truth for what it'll accept - everything else is downstream of this.
2. Update the config to match
.env gets the new value. Until this happens, the services are holding stale credentials in memory or on next restart, about to fail against a database that no longer recognizes them.
3. Force the containers to actually pick it up
docker compose up -d --force-recreate
A restart alone isn't guaranteed to reload environment variables depending on how a service reads them. --force-recreate rebuilds the containers from the current .env, so there's no ambiguity about which password each service is holding.
The check that mattered more than the fix
Rotating the password is the easy part, mechanically. The part that actually tells you whether it worked is verifying nothing broke on the way through:
SELECT COUNT(*) FROM cves;
103
Same as before. Every row still there, every service still able to read and write against the new credential, nothing silently orphaned by the change. Rotation isn't done when the password changes - it's done when you've confirmed the thing the password protects is still intact on the other side of it.
Why this one's worth writing about
It's not a technically hard fix. ALTER ROLE, edit a file, recreate a container — none of that is advanced. What it needed was not skipping a step under the mild adrenaline of "I just exposed a secret, fix it now." The instinct in that moment is to move fast. The actual requirement is to move in the right order and check your work at the end, which is exactly the discipline a real credential rotation demands whether the trigger is a careless screenshot or something worse.