Data Engineering · Capstone
StudentOps Data Platform.
An end-to-end data platform that ingests six sources into a layered Postgres warehouse with dbt and Prefect, then serves a dashboard, a daily digest and a Claude assistant.
Challenge
Study life is scattered across calendars, reference managers and notes, with no single view of what is due, what is planned and what is worth reading. The goal was a real data platform, not a to-do app: ingestion, modelling, orchestration and serving, end to end.
Approach
Six sources (iCal feeds, Google Calendar, Zotero, BibTeX and RIS drops, markdown notes, OpenAlex and CrossRef) land in a bronze, silver, gold Postgres warehouse. dbt builds the typed silver views and gold facts and dimensions with SCD2 history; Prefect orchestrates the runs. Three surfaces sit on top: a Streamlit dashboard, a daily email digest, and a Claude assistant that summarises, generates flashcards and suggests citations with mandatory DOI validation.
Outcome
A single tenant platform running live for about twelve dollars a month, with observable pipelines and freshness tracking per source. It demonstrates the full data engineering loop: ingest, model, orchestrate, serve, and a grounded AI layer that refuses to invent citations.
Key decisions
- Bronze, silver, gold warehouse on Postgres with dbt, including SCD2 history on the assessments dimension.
- Six ingestion sources normalised into one model, orchestrated and scheduled with Prefect.
- A Claude assistant with a hard DOI validation gate so it never fabricates a reference.
- Daily digest by email and a Streamlit dashboard, deployed on Fly.io and Cloudflare Pages.