Research Design Meets Market Design

Project Outcomes Statement

The goal of this NSF grant is to develop tools that can be used to leverage centralized school assignment for credible econometric evaluation of school effectiveness. As centralized market design has come to play a bigger role in school assignment across the country, the need for these methods has intensified. The core contributions are methodological. Specifically, PIs Abudulkadiroglu, Angrist, Pathak, and their coauthors aimed to derive a set of theoretical results that show how to isolate randomness in market designs of increasing complexity.

With this goal in mind, the authors successfully developed a methodology to derive unbiased measures of school quality in complex centralized assignment settings like New York City and Chicago. A key challenge to evaluating school quality in some districts comes from the use of non-random tie-breakers in admissions decisions; selective schools use information like GPA, test, scores and other information to admit students. The authors overcome this challenge by integrating a non-parametric regression discontinuity framework with the large-market matching model outlined in their earlier work, "Research Design Meets Market Design: Using Centralized Assignment for Impact Evaluation."

In their working paper, "Breaking Ties: Regression Discontinuity Design Meets Market Design," the authors use this methodology to investigate the predictive value of New York City's former school report card system. Results show that attendance at one of NYC's "Grade A" schools modestly boosts SAT math scores and may slightly increase high school graduation. These effects are smaller than the corresponding conventional value-added estimates. In addition, motivated by the ongoing debate over screened admissions policies, the authors compare attendance effects at Grade A schools that screen and Grade A schools that admit using lotteries. These estimates show no evidence of a screened school advantage. Importantly, the authors find evidence of egregious selection bias in conventional value-added estimates for screened Grade A schools.

The authors also successfully use these methods to evaluate selective exam schools in Chicago in their published paper "Regression Discontinuity in Serial Dictatorship: Achievement Effects at Chicago Exam Schools." Chicago exam schools use selective admissions criteria such as letter grades and test scores to admit students. These schools also use a neighborhood-based diversity criteria. In a third working paper, "Choice and Mismatch: Assessing Mismatch at Chicago Exam Schools," the authors find that beneficiaries of this tier-based affirmative action plan experience a decrease in their math scores. These results are explained by the applicants who do not get admitted to exam schools; exam school admittance diverts students away from the high-performing Noble Charter School Network where academic growth is higher for students.

The innovative methods supported by this grant have important implications for the field of economics and the subfields of econometrics and market design. This work also has implications for school districts that use centralized assignment systems, most often found in large high-poverty urban districts. Throughout the grant, the research team worked closely with the New York City Department of Education and Chicago Public Schools to disseminate results from the study. The team presented findings to 16 U.S. school districts, including Chicago Public Schools, Denver Public Schools, Boston Public Schools, and Indianapolis Public Schools. All of these districts use centralized assignment schemes.

In the future, these methods may be applicable outside of education within the fields of medicine and law. For example, the US National Residency Matching Program assigns medical school graduates to hospitals using a version of deferred acceptance with non-lottery tie-breakers. Research can leverage this match to answer questions about the effects of alternative medical internships, such as the value of experience in high-volume hospitals. The framework may also be useful to study the effects of resources allocated by some auction mechanisms.