NYC taxi lakehouse & analytics
Processed 56 million trips into a tested data warehouse and three executive dashboards, with a 94.7% QA pass rate.
Python · Polars · DuckDB · dbt
The problem
Raw data is not a decision tool. Inconsistent types, questionable fares, and ambiguous definitions make even a simple revenue question hard to answer reliably.
What I built
- Bring 15 monthly Parquet files into consistent Bronze, Silver, and Gold layers, with explicit contracts between each.
- Process 56 million trips with a 94.7% QA pass rate. Quarantine anomalies and fare mismatches instead of silently fixing them.
- Use Polars, DuckDB, and dbt to create reproducible dimensions, facts, and analytics marts, backed by 60+ validation tests.
- Build three executive views: Company Pulse, Strategic Levers, and Zone Heat, connecting operational questions to explorable data.
Key tradeoff
Portability versus distributed scale. A local stack makes the project reproducible and easy to demonstrate; a shared enterprise deployment would need a different concurrency and operational design.