A post-secondary program reference dataset built from two open Parcoursup (French university admissions) datasets. Entity resolution across both sources, layered dbt modeling with 28 automated tests, full orchestration via Dagster. Solo project built to go deeper on the data engineering stack.
French post-secondary programs publish their application statistics through Parcoursup, but as raw datasets with no unified reference table. Two open-data sources each cover part of the picture: 2025 applications and responses on one side, a multi-year program directory on the other. Neither is usable alone to compute an admission rate per program. A personal project to get hands-on with a modern data engineering stack (DuckDB, dbt, Dagster) on a real public dataset.
The real problem isn't loading CSVs — it's merging two sources that don't share a 100%-reliable common key, without losing or duplicating programs, and producing an admission rate that means something rather than a naive division.
Goals set:
Layered architecture. DuckDB as the embedded analytical engine (columnar, no server), dbt Core on top for versioned SQL transformations across four layers (raw → staging → intermediate → marts), all orchestrated by Dagster as an asset graph.
Two-phase entity resolution. Exact join on the identifier shared by both sources (99.7% of programs), then an approximate fallback by name similarity (Jaro-Winkler, 0.90 threshold) for the remaining 38. Analyzing the fuzzy fallback's false positives surfaced a business rule more reliable than any threshold: exclude apprenticeship programs, which never have application statistics.
Traceable admission rate. Computed as offers over applications, with an explicit distinction between an empty rate (no statistics) and a zero rate (no offers made). The official Parcoursup rate is kept alongside it — no simple division reproduces it exactly.
28 dbt tests + 2 blocking Python checks. Key uniqueness and non-nullness, referential integrity (no program lost between layers), rate consistency, a funnel check (applications ≥ offers ≥ acceptances). A non-blocking warning flags a known source anomaly (programs accepting more candidates than their stated capacity).
Dagster orchestration. Ten assets in a single graph — two Python ingestion assets, eight dbt models — with dbt tests automatically surfaced as asset checks. One job replays the entire pipeline end to end in about a minute.
A complete, reproducible pipeline, runnable with a single command from raw data to the aggregated per-académie view.
Result. A reference table of 25,865 programs, 14,252 of them with complete application statistics, ready to query in SQL (e.g. average selectivity per académie).
Scale. 2 open-data sources · 8 dbt models · 28 tests + 2 Python checks · 10 Dagster assets · full pipeline run in ~1 minute.
Solo build. Data model design, dbt models and tests, Dagster orchestration — from the first line of SQL to a reproducible pipeline.