$ cat ~/tags/data-engineering — 4 entries
The case proves the what — 2,084 products, 21 jobs, 2-minute poll. This is the how: the quota-aware poll loop, the scoring curve that unclumped 55.8% at 75, and stale-by-design edge reads.
The unglamorous plumbing behind the 18h-to-4h forecasting win: streaming a 1.8 GB Tally Prime XML export with lxml iterparse, surviving mixed encodings and damaged nodes, into an indexed SQLite frame.
How I replaced a crashing Excel-based inventory planning process for 7,800+ SKUs with a Python multiprocessing pipeline using Meta's Prophet AI — cutting weekly planning time from 18 hours to 4 hours overnight.
