TweakDeals: Real Price History for AliExpress Tech

Overview

Most deal sites show a price crossed out next to a lower one. That number is whatever the seller chose to print, which is why the category is full of 90%-off everything.

TweakDeals (tweakdeals.com) takes the opposite approach: it never trusts the listed discount. Every price the site has ever seen a product at is stored in a PriceLog table, and the displayed discount is computed against the highest observed price. Not the seller’s original. If a product has only ever been seen at one price, the badge says so instead of inventing a 50% off.

Underneath that is a scraper fleet that polls 400+ tracked SKUs every two minutes and a scoring pipeline that grades each product on a value curve. The front end is Next.js 16 on Cloudflare Workers: the catalog goes out as static HTML, then live prices rehydrate from a micro-API in the browser.

The site is live today with 2,084 live products and a sitemap of 2,161 URLs.

---

1. The Problem: Discounts That Don’t Mean Anything

AliExpress does not publish a price history API. The originalPrice field is a marketing claim, not a record. You can walk into any category and find a “was $99.80, now $49.90” badge where the $99.80 was set five minutes ago.

Three requirements shaped everything else:

• Trustworthy discounts. Only show a percentage we can compute from observed data. If we have no history, show no badge.
• A catalog that grows on its own. A static list of 169 hand-picked products is a weekend’s work, not a product. Discovery has to be automatic.
• Cheap operation. The whole thing had to survive on free and near-free tiers, because a side project that costs $400/month in Neon compute is a hobby that ends.

---

2. Architecture

Cron fleet → AliExpress API + LLM enrichment → Neon Postgres (Product, PriceLog, CronRun, DeadLetterQueue) → prebuild cache → static export → Cloudflare Worker → client-side price hydration.

The monorepo

A pnpm + Turborepo workspace with three applications and two shared packages:

One decision worth calling out: the web app talks to Neon over raw SQL through @neondatabase/serverless, while the scraper uses Prisma. Prisma’s client generation step was fighting the Workers bundle; the HTTP driver doesn’t. The scraper runs on Node and benefits from the type-safe client, so both sides got what they needed.

---

3. The Ingestion Pipeline

Price polling

poll.ts runs on a two-minute cadence. It checks every tracked SKU against the AliExpress affiliate API, diffs the result against the last known price, and writes a PriceLog row for anything that moved.

The expensive part isn’t the write. It’s the rate limit. The API has a hard quota, so the poller is built to degrade: it counts down remaining quota and logs a percentage on every run, and when a scrape fails it lands in the dead-letter queue with exponential backoff. The batch survives.

$ pnpm --filter @tweakdeals/scraper exec tsx src/poll.ts
[poll] aliexpress: 200 (quota 61%) — 412 tracked SKUs checked
[poll] 118 price changes · 9 new · 3 delisted
[poll] PriceLog rows written: 118   Transaction: ok

At that rate the table takes on roughly 18,000 price observations a day, which is what makes the discount math defensible.

Observed price (PriceLog) versus the seller’s listed claim, plus the catalog growth that feeds the discount math: 169 products at first audit → 2,084 live as of Oct 2026.

Catalog growth, first audit → Oct 2026: 169 hand-picked → ~800 with discovery on → ~2,000 after cleanup waves (DAR-506…509, DAR-1057/1058) → 2,084 live, 2,161 sitemap URLs.

Discovery

Four separate discovery pipelines feed the catalog, because each finds different things:

• discover-products.ts seeds candidates from category sweeps.
• discover-from-reviews.ts mines competitor review pages for products people are asking about.
• run-hotproduct-discovery.ts goes after whatever is trending in a given window.
• find-affiliate-alternatives.ts, when a product goes delisted, finds a live replacement and links them.

Discovered candidates land in a queue and pass quality gates before they become products. Those gates got sharpened the hard way: early versions accepted generic, unbranded listings (“USB-C Cable 1m”), accessories sold as if they were the product, and miscategorized items. I once found a charger filed under keyboards and a CD player under IEMs. Several cleanup waves later (DAR-506 through DAR-509, DAR-1057, DAR-1058), the catalog is 2,000+ products of actual tech gear. I should have written the category classifier before the discovery pipelines, not six cleanup waves after them.

Enrichment and scoring

Two LLM passes sit in the pipeline. One writes a two-sentence description; the other generates a clean displayTitle, because marketplace titles are things like realme Buds Air 8 55dB Ultra Depth Noise Cancellation Global Version True Wireless Earphone Bluetooth 5.4 Earbuds LHDC 5.0 IP55, which is not a heading. That normalization pass ran across roughly 2,000 products, and it had to be re-run twice more after I found the brand-truncation regex was chopping names mid-word.

The scorer produces a 0–100 qualityScore and dealScore from price history, review volume, and expert review coverage. Early calibration had a serious problem: 55.8% of products landed at exactly 75, which meant the leaderboard was useless. Recalibrating the curve fixed the homogenization.

---

4. Delivery: Static Shell, Live Prices

The catalog is large, the pages are SEO-critical, and the source of truth is a database that has taken unplanned downtime more than once. That combination argues for a split.

At build time, prebuild-data.ts pulls the catalog from Neon and writes a cache JSON (roughly 14 MB, ~2,200 products). If the database is unreachable the build falls back to a committed cache snapshot instead of failing. The Next.js build then emits static HTML for every product, category, and guide, with OG images generated per page.

At request time, the Cloudflare Worker serves those static assets from the edge. No SSR database call on the hot path.

In the browser, a single micro-API at /api/pricing/[sku] hydrates live prices on the client after first paint. Users see real prices without the build ever needing to be current to the second.

GET /api/pricing/1005011976952613
{
  "sku": "1005011976952613",
  "currentPrice": "49.90",
  "observedHigh": "99.80",
  "discount": 50,
  "observedSince": "2026-02-11",
  "freshnessMinutes": 143
}

The price-freshness contract is explicit: the API reports how old the reading is, the UI shows it, and the nightly price-staleness-monitor job fails the build if anything drifts past its window. A stale price that looks live is worse than an obviously old one.

---

5. Operations: When the Database Disappears

This project had an unplanned lesson that most case studies skip. Twice in 2026, the production database was emptied out from under the site. Most recently in August 2026, the deploy restored 2,544 products at 10:32 UTC and the same endpoint returned zero again by 11:05 UTC. The homepage rendered “No products” and /api/products returned {"data": [], "pagination": {"totalItems": 0}}.

The forensics ran for weeks. The work included rotation of every credential that had ever touched the box (a DATABASE_URL had been baked into a Worker bundle by a build script), re-application of forensics_truncate_* triggers after every restore, a two-signal restore gate in the watchdog, a mid-run circuit breaker in the scoring pipeline, and an “empty-DB protection” gate in CI that blocks any deploy landing with fewer products than expected.

The architectural lesson is the one that mattered: the committed prebuild cache is not a fallback, it’s a durability guarantee. Because the front end can be built entirely from a file on disk, a database incident degrades the site to “prices are a few hours stale” instead of “the site is blank.”

---

6. Results

• Catalog: 169 products at first audit → 2,084 live today, with ~2,161 indexable URLs including category and guide pages.
• Freshness: prices polled every 2 minutes, ~18k price observations per day feeding the discount math.
• Operations: 21 scheduled jobs covering ingestion, scoring, search-console submission, backups, health checks, and alerting.
• Cost: the entire stack (Workers, Neon, R2, domain) runs on free and launch tiers. No per-request egress bill.
• Recovery: static-first delivery meant every database incident in 2026 was a data-staleness problem, never a downtime problem.

Figures as of Oct 2026.

What the outage settled

I keep coming back to one line from that August forensics session: the homepage said “No products” while the cache snapshot that could have built it sat committed in the repo, unused. The prebuild-data.ts snapshot had existed since the first deploy. What the incident changed is that the build now treats it as the primary source and the database as the refresh mechanism. The ~18,000 daily PriceLog observations still do the discount math, and the two-minute poll cadence with quota-aware degradation keeps that history continuous when the affiliate API throttles. But the database is a freshness concern now, not an availability concern. Blank page.

---

What I’d Do Differently

Run the catalog in a state where “the database is empty” is boring. The incidents were all traced to credentials that were too widely shared and scripts that could truncate without a second signal. That is a design problem (single-writer migrations, break-glass credentials with an expiry, a restore that cannot run twice). Not an ops problem.

And I would spend the LLM budget on fewer products with better evidence. Two thousand rows with a 60% expert-review coverage is worth less than eight hundred rows where every number is traceable to a source.

“Anyone can print a 70% off badge. The hard part is being able to prove it.”