Theme Park Wait-Time Forecasting: Production MLOps Pipeline
A production ML pipeline that forecasts theme park wait times a week ahead, retrained unattended every week on Databricks and served live for $0/month. Powers the Mapblazer routing engine.
Problem Statement
Mapblazer builds a guest's optimal ride itinerary from their must-visit attractions, live wait times, and walking distances. The flaw was that the optimizer routed on "current" wait times: a plan built at 9:00 AM is already wrong by the time the guest reaches their 1:00 PM attraction, and each stale estimate cascades into the next. What the solver actually needs is not a prediction endpoint but the entire cost surface, every attraction at every possible arrival time a week out, which makes this a batch forecasting problem rather than a model-serving one. Producing that surface reliably means answering the operational questions alongside the modelling ones: where the data comes from each hour, what retrains the model as crowd patterns drift, what happens when a run fails at 6am on a Sunday, and how anyone can tell whether the forecast being served right now is fresh or three weeks stale.
Results & Impact
7.01 min
Champion MAE
Prophet fleet, vs. a 9.51 min baseline
26.3%
Accuracy Gain
MAE reduction vs. per-ride mean baseline
76.6%
Within 10 min
Up from 64.5% at baseline
42.1% fewer
Severe Misses
Errors over 15 min: 23.5% to 13.6%
Weekly
Retrain Cadence
Unattended, Sunday 06:00 UTC
$0/mo
Infra Cost
Databricks Free, Supabase, GH Pages
Live Dashboard
Loading dashboard…
Rebuilt every Sunday
Regenerated from the gold _last tables two hours after the training job, then deployed to Pages.
Provenance on the page
Run id, run status and publish timestamp are printed, so a fallback week is visibly a fallback.
Static HTML, zero JS
Inline SVG charts and no external assets, so it survives a strict Content-Security-Policy.
Model Performance
Backtest on the held-out final 20% of history: 78,110 observations the models never saw, taken chronologically rather than at random. Every candidate is scored on all seven KPIs, and the champion has to clear the per-ride mean baseline before it is allowed to serve.
| Model | MAEmin | RMSEmin | Within 10m% | Severe miss% | Biasmin | High-wait MAEmin | Peak MAEmin |
|---|---|---|---|---|---|---|---|
| prophet_fleetchampionone Prophet per ride | 7.01 | 11.49 | 76.6 | 13.6 | +1.2 | 10.3 | 7.8 |
| xgb_globalone tree, ride as categorical | 8.26 | 12.88 | 71.8 | 17.6 | +3.8 | 12.1 | 9.4 |
| xgb_local_fleetone tree per ride | 8.45 | 14.70 | 71.9 | 19.3 | +3.7 | 12.9 | 10.0 |
| baseline_ride_meanper-ride historical mean | 9.51 | 14.46 | 64.5 | 23.5 | +1.0 | 14.4 | 9.1 |
Lower is better on every column except Within 10m. A severe miss is an error over 15 minutes, the size that actually breaks a route. High-wait MAE covers rides averaging over 10 minutes and Peak MAE covers 11:00 to 20:00 local, because roughly 37% of observations are exactly zero and a single average would let a model look good by predicting “short queue” everywhere.
Bias is the one column the baseline wins, and that is expected rather than surprising: predicting each ride's historical mean is unbiased by construction. It buys that with the worst error on every other measure.
The champion beats the baseline by 26.3% on MAE, lifts ±10-minute containment from 64.5% to 76.6%, and cuts severe misses by 42.1%, with the widest margin on exactly the long queues a router is most sensitive to. These are the numbers from run 20260822T203750Z; the pipeline retrains every Sunday, so the embedded dashboard above is always the current authority.
Approach & Methodology
Ingestion: A Push Agent on the Source Host
The ingestion agent runs on the same EC2 host as the upstream wait-time Postgres and pushes outbound rather than being pulled from. That was a deliberate choice: pulling would have meant exposing Postgres to a rotating set of CI runner IPs, while pushing keeps the database connection on the loopback interface, needs no security-group change, and requires only outbound HTTPS. One hourly systemd run does four things: read the watermark as MAX(wait_time_id) from bronze itself, select the next batch above it, land it as Parquet, and MERGE it in insert-only. Because the watermark comes from the destination rather than a local cursor file, an interrupted run simply re-reads the same range and the MERGE absorbs the overlap. There is no local state that can drift out of sync, and nothing to repair by hand at 3am.
Bronze to Silver: The Cleaning That Decides the Numbers
Bronze is append-only and never edited. It is the audit trail, and every correction happens downstream. Silver drops duplicate readings from overlapping batches, the source system's 900-minute sentinel, negative waits, a duplicated obsolete park, and an attraction literally named "0". Order matters more than any individual filter: timestamps are converted from UTC to America/Los_Angeles first, and only then filtered against each park's local operating hours. Done the other way round it keeps 00:00 to 15:00 local and discards the entire evening peak, the busiest hours of the day, while leaving a dataset that still looks perfectly reasonable. Every ride then resamples onto a fixed 30-minute grid through a single canonical ride-key function, and attractions with under 100 observations are dropped as too thin to model.
Features From Local Wall Time
Calendar features are derived from park-local wall time, never UTC: hour, minute, day of week, month, weekend and US-holiday flags. With under a year of history, gradient-boosted trees cannot extrapolate raw datetime values because they only split inside dates they have already seen, so hour and month are additionally encoded as sin/cos pairs, making 23:59 and 00:00 (and December and January) adjacent rather than maximally distant. Training, scoring, and the forecast grid all call the same feature builder and the same ride-key function, so a change cannot silently apply to one path and not the others.
Four Model Families, One Chronological Split
Every run trains a per-ride Prophet fleet (flat growth, daily and weekly seasonality, US-holiday regressors, 80% prediction intervals), a single global XGBoost carrying ride identity as a native categorical so it can borrow signal across attractions, a per-ride XGBoost fleet, and a per-ride historical mean as the floor every candidate must clear. The split is strictly chronological, the first 80% by time trains and the last 20% tests, never random, because random folds leak the future into validation on a time series. Hyperparameter search is deliberately absent for the same reason: a RandomizedSearchCV with random K-fold carries that leak straight into model selection, so fixed conservative parameters stand in until there is enough history for a proper rolling-origin search.
KPIs That Do Not Hide Behind an Average
About 37% of observations are exactly zero, so a single mean absolute error flatters any model that learns to predict "short queue" everywhere. Each model is scored on seven KPIs: MAE, RMSE, within-10-minute rate, severe-miss rate, bias, high-wait MAE (rides averaging over 10 minutes), and peak-hours MAE (11:00 to 20:00 local), so the segments a router actually cares about stay visible instead of averaging away. The Prophet fleet wins on every one of them, and the margin is widest exactly where it matters, on the long queues.
Champion and Challenger Promotion
A newly trained model ships only if its holdout MAE beats the incumbent's recorded MAE by at least 1%. If it does not, the incumbent is reloaded from disk and re-scored over the upcoming week, so the forecast window is fresh either way. Model artifacts are plain files, Prophet serialized to JSON per ride and XGBoost through save_model, never pickles of repository classes, which are only loadable by a process that can import the class that made them. They are written to an immutable runs/<run_id>/ directory with a manifest. A small serving.json names the run that serves, and it is written last, after the tables have been promoted. Rolling back is editing that pointer to an earlier run id.
Quality Gates and the _current / _last Discipline
Every silver and gold table exists twice. A run writes only _current, the quality gates run against _current, and only a fully passing run promotes _current into _last through atomic DEEP CLONE commits. Supabase and the dashboard read _last only, so a failed or half-finished run can never take down a working dashboard. The gates cover silver row count against last week, attraction count, zero-wait share, nulls, model presence, finite MAEs, serving-model-beats-baseline, forecast ride coverage, interval bound ordering and row-count bands, plus a domain tripwire: the local hour with the highest mean wait must fall between 11:00 and 20:00, a condition an inverted timezone conversion cannot satisfy.
Failing Without Going Dark
Any failure, whether a quality gate, a training error or unreadable bronze, drops _current, leaves _last untouched, re-scores the upcoming week with the previous serving model so the dashboard never serves a forecast window that has slid into the past, and still fails the job so the failure alerts. Both gold tables carry run_id, run_status (fresh_model, kept_previous_model, or fallback_after_failure) and generated_at, and the dashboard prints all three, so a fallback week is visibly a fallback rather than a quietly stale one. A first-ever run with nothing to fall back to changes nothing and says so.
Serving: Atomic Swap and a Dashboard With No JavaScript
Because the training job has no outbound internet, serving is pulled rather than pushed. A GitHub Actions workflow reads the gold _last tables over the Databricks SQL warehouse once, chunk-loads them into Supabase staging tables with idempotent inserts, then calls a PostgREST RPC that swaps staging and serving by table rename inside a single Postgres transaction, a catalog-only operation that is instant at any row count where the earlier DELETE/INSERT approach hit the free-tier statement timeout at 255K rows. Readers therefore always see one consistent KPI/forecast pair. The same fetch renders the public dashboard: one static HTML file with inline SVG charts and zero JavaScript, so it survives a strict Content-Security-Policy. The dashboard deploys even if the Supabase push fails, and the job still goes red so the failure is visible.
Architecture
── INGEST - AWS EC2, hourly systemd timer ────────────────────────
┌──────────────────────────────────────────────────────────────────┐
│ deploy/ec2/ingest.py runs beside the source Postgres │
│ pushes outbound by design: the database connection stays │
│ on the loopback interface, no firewall rule to change, │
│ and the only access needed is outbound HTTPS │
│ watermark = MAX(wait_time_id) read back from bronze │
│ land Parquet -> MERGE insert-only, so a killed run just │
│ re-reads the same range and the overlap is absorbed │
└────────────────────────────────┬─────────────────────────────────┘
│ Free Edition has no outbound net
↓
── PIPELINE - Databricks serverless, Sunday 06:00 UTC ────────────
┌──────────────────────────────────────────────────────────────────┐
│ bronze.wait_times_raw 842,539 rows │
│ append-only audit trail, never edited in place │
└────────────────────────────────┬─────────────────────────────────┘
↓
┌──────────────────────────────────────────────────────────────────┐
│ silver.wait_times_current ~390K rows . 109 attractions │
│ UTC -> local time first, then park operating hours │
└────────────────────────────────┬─────────────────────────────────┘
│ chronological 80/20 split
↓
┌──────────────────────────────────────────────────────────────────┐
│ train four families every run, holdout MAE in minutes │
│ │
│ Prophet fleet 7.01 <- champion │
│ XGB global 8.26 XGB per-ride 8.45 │
│ Ride mean 9.51 <- floor, not a candidate │
└────────────────────────────────┬─────────────────────────────────┘
↓
┌──────────────────────────────────────────────────────────────────┐
│ gold.kpis_current + gold.predictions_current │
│ 7 KPIs x 4 models . 7-day forecast + backtest rows │
└────────────────────────────────┬─────────────────────────────────┘
↓
┌──────────────────────────────────────────────────────────────────┐
│ 12 quality gates -> promote _current to _last │
│ atomic DEEP CLONE . serving.json pointer written last │
└────────────────────────────────┬─────────────────────────────────┘
│ pulled out, never pushed
↓
── SERVE - GitHub Actions, Sunday 08:00 UTC ──────────────────────
┌──────────────────────────────────────────────────────────────────┐
│ publish.py reads gold _last over the Databricks │
│ SQL warehouse │
│ publish_serving() RPC staging and serving renamed inside │
│ one Postgres transaction │
└────────────────────────────────┬─────────────────────────────────┘
┌──────────────────┴──────────────────┐
↓ ↓
┌───────────────────────────┐ ┌───────────────────────────┐
│ Supabase PostgREST │ │ GitHub Pages │
│ read-only API │ │ dashboard.py, zero JS │
└───────────────────────────┘ └───────────────────────────┘
└──────────────────┬──────────────────┘
↓
┌──────────────────────────────────────────────────────────────────┐
│ Mapblazer routing solver │
│ the whole cost surface: every ride x every arrival time │
└──────────────────────────────────────────────────────────────────┘