ProgramEvaluationPlatform
A causal estimate is only as trustworthy as the pipeline that produced it. When balance checks run in one analyst's notebook and the model lives in another, nobody can say whether last quarter's finding would survive a rerun — and in practice, it often would not.
iThe problem
Analysis code and data lived in disconnected places, so reproduction depended on one person's memory of which notebook was current.
No standard balance diagnostics: covariate imbalance could reach the modelling stage unnoticed.
Every evaluation was bespoke, which meant every evaluation had its own quiet bugs.
iiThe approach
Modelled raw enrolment and assessment data into a warehouse layer with documented, tested transformations — every field typed and traced to source.
Automated balance diagnostics that run before any estimation, failing the pipeline loudly if treated and control groups diverge beyond a defined threshold.
A specification registry: each design variant is a versioned artefact, not a copy-pasted script, so comparisons across cohorts are genuinely like-for-like.
Orchestrated end to end, so a refresh is a scheduled job rather than a person remembering to click run.
iiiWhat it produces
Producing a new evaluation went from a multi-week bespoke effort to a configuration change plus a review.
Balance failures surfaced before estimation rather than after publication.
A colleague could pick up the pipeline and extend it without a handover conversation.
Stack