18,000 Observations a Day: Inside the TweakDeals Price Pipeline

The TweakDeals case study proves the what: 2,084 live products, 21 scheduled jobs, a 2-minute poll interval, ~18,000 PriceLog observations a day. What follows is the mechanism behind those numbers: the poll loop, the scoring curve, and the stale-by-design reads, written for the reader with a catalog who wants to know whether the person behind it could instrument theirs.

The poll loop: quota-aware by construction

poll.ts runs on a two-minute cadence against the AliExpress affiliate API. The transcript in the case is the whole contract:

$ pnpm --filter @tweakdeals/scraper exec tsx src/poll.ts
[poll] aliexpress: 200 (quota 61%) — 412 tracked SKUs checked
[poll] 118 price changes · 9 new · 3 delisted
[poll] PriceLog rows written: 118   Transaction: ok
[poll] next run in 120s

The expensive part isn’t the write: it’s the rate limit. The poller counts down remaining quota and logs a percentage on every run, and when a scrape fails it lands in the DeadLetterQueue with exponential backoff instead of throwing the batch away. At that rate the table takes on roughly 18,000 price observations a day:

$ psql $DATABASE_URL -c "select count(*) from \"PriceLog\" where observed_at > now() - interval '24 hours';"
 count
--------
 18422

That continuity is what makes the discount math defensible: every displayed discount is computed against the highest observed price, never the seller’s claim. A product seen at one price carries no badge (“no history” instead of an invented 50% off). Discovery grows the catalog the same unattended way: discover-from-reviews.ts mines competitor review pages, run-hotproduct-discovery.ts chases trending windows, find-affiliate-alternatives.ts links delisted products to live replacements. Quality gates sharpened across cleanup waves (DAR-506 through DAR-509, DAR-1057, DAR-1058) taught the pipeline to reject generic listings and miscategorized items.

The scoring curve: unclumping 75

Two LLM passes enrich the pipeline: one writes a two-sentence description, the other a clean displayTitle, because marketplace titles are keyword payloads, not headings. (That normalization ran across ~2,000 products and had to be re-run twice after a brand-truncation regex chopped names mid-word. Twice.)

The scorer produces a 0–100 qualityScore and dealScore from price history, review volume, and expert review coverage. Early calibration had a serious problem, quoted verbatim from the case: 55.8% of products landed at exactly 75, which meant the leaderboard was useless. Recalibrating the curve fixed the homogenization.

The lesson generalizes beyond deal sites: any scoring pipeline needs a distribution check, not just a correctness check. Every product scored “correctly” per the formula and the ranking was still meaningless — the output to monitor is the histogram, not the row. Watch the histogram.

Stale by design: the delivery split

The catalog is large, the pages are SEO-critical, and the source of truth has taken unplanned downtime more than once. That combination argues for the split the case documents:

• At build time, prebuild-data.ts pulls the catalog from Neon into a ~14 MB cache JSON (~2,200 products). If the database is unreachable, the build falls back to a committed cache snapshot instead of failing.
• At request time, the Cloudflare Worker serves static assets from the edge. No SSR database call on the hot path.
• In the browser, /api/pricing/[sku] hydrates live prices after first paint, and the freshness contract is explicit: the API reports freshnessMinutes, the UI shows it, and the nightly price-staleness-monitor fails the build past the window. A stale price that looks live is worse than an obviously old one.

The Neon Postgres note documents the storage split behind it: Prisma migrations in Node, raw SQL over HTTP from the edge. And the payoff is the case’s architectural lesson, quoted whole: the committed prebuild cache is not a fallback, it’s a durability guarantee — every 2026 database incident degraded the site to “prices a few hours stale” instead of a blank homepage. The empty-database postmortem is the forensics companion.

Honest limitations

• The scoring formula is described, not published. Inputs (price history, review volume, expert coverage) and the recalibration story are on the record; the weights aren’t. Treat the curve discussion as calibration discipline, not a reproducible recipe.
• Affiliate-API-shaped. The quota-aware degradation assumes a throttling upstream with a countable quota: poll loops against unmetered sources have a different failure mode (freshness, not budget).
• Free-tier economics throughout. The whole stack runs on free and launch tiers per the case, which constrains the design (static-first because incidents can’t be bought away) and bounds the transferability to catalogs that can afford otherwise.

If you run a catalog

If your catalog runs on seller-claimed discounts, observed history is the only correction I trust, and stale-by-design delivery is what turned every 2026 database incident into a slow afternoon instead of downtime. The Python ETL pattern behind this pipeline generalizes beyond deals, and so does the Prophet + SQLite forecasting work; the full numbers live in the case.