The Database Emptied Itself Twice: Forensics on Shared Credentials and Truncate-Capable Scripts

In August 2026 the deploy restored 2,544 products at 10:32 UTC. By 11:05 the same endpoint returned zero. The homepage rendered “No products” and /api/products returned:

{"data": [], "pagination": {"totalItems": 0}}

Twice in 2026 the production database was emptied out from under the site. What follows is the forensics, the guards, and the architectural lesson, extracted from the TweakDeals case study.

The forensics

The investigation ran for weeks. Its findings, in order of embarrassment:

1. A DATABASE_URL baked into a Worker bundle by a build script. Production credentials lived inside a shipped artifact. Every credential that had ever touched the box was rotated once this surfaced.
2. Truncate-capable scripts with a path to production. forensics_truncate_* triggers were re-applied after every restore, which tells you what class of statement the estate had lying around.
3. No restore gate. Nothing stood between “database is empty” and “ship it”: the watchdog, the scoring pipeline, and CI all assumed a non-empty database rather than verifying one.

The guards

Four layers, each doing one job:

• forensics_truncate_* triggers, re-applied after every restore (the database itself refuses the dangerous statement class).
• A two-signal restore gate in the watchdog: restoration requires two independent confirmations, not one hopeful check.
• A mid-run circuit breaker in the scoring pipeline: a run that starts emptying things stops itself instead of completing the damage.
• An empty-DB protection gate in CI: any deploy landing with fewer products than expected is blocked before it ships. (The pipeline-side sibling of this thinking is the slim CI rebuild, where gates also learned to actually fail.)

The lesson that mattered

Public postmortems with transcripts are this site’s most distinctive asset. The 41-minute CSP outage and the prebuilt redirect saga already sit on the about shelf. This one extends the lineage to the most impressive live system: 2,084 products, 21 scheduled jobs, and a database that vanished twice without taking the site with it. The full system context is the TweakDeals case study.

Honest limitations

• Root cause, not root actor. The case documents the credential exposure and the truncate-capable scripts, not a named human-or-process culprit for either emptying. This postmortem claims the forensics, not an attribution.
• August timestamps are the only precise ones. The “twice in 2026” framing rests on the case body; only the August incident carries the 10:32/11:05 UTC timestamps quoted here.
• Credential rotation is described, not shown. Every credential was rotated; the rotation transcript is not part of the published record.