What Works?

Do Scared Straight programs work as advertised? What do successful refugee resettlement programs look like? How does preschool language instruction impact reading comprehension in later schooling?

Citizens and scholars, policymakers and practitioners—everyone wants to know that programs created
to solve pressing societal problems actually work.

Researchers are on the job, producing myriad studies of such programs. When well conducted, single studies reveal much about a particular population and context, but it’s not unusual to find multiple studies on the same topic commissioned by different agencies in different settings. They can easily yield different results from one another—with confusion as the end product.

So how do we evaluate and synthesize results of multiple studies?

The answer: systematic reviews that summarize the results of a range of well-designed research studies to provide high-quality evidence of the effectiveness of interventions across diverse populations and contexts.

In 2000, a group of scholars gathered in Philadelphia to take on the challenge of research synthesis. Inspired by the example of the Cochrane Collaboration, which synthesizes empirical evidence on interventions in the health care field, they formed the Campbell Collaboration to perform systematic reviews of empirical research in the social and behavioral sciences. The new organization was named for the American sociologist Donald T. Campbell, who had envisioned an “experimenting society” comprising a “disputatious community of ‘truth seekers’” devoted to improving social policy through rigorous evaluation and analysis.

Designed to sum up the best available research on a specific question, Campbell reviews are conducted by researchers from around the world—truth-seekers who synthesize the results of multiple studies to uncover what works and what doesn’t.
To pass muster, a Campbell review must follow particular protocols: Each review uses procedures to ensure accuracy at each step in the review process. Findings are posted in Campbell’s online library—both as full reports, complete with technical details and supporting materials, and as plain-language summaries for nonspecialists.

Campbell reviews cover a wide range of topics: Scared Straight programs, welfare-to-work strategies, mindfulness-based stress reduction, conditional cash transfers in education, school-based programs to prevent bullying, parenting programs, and kinship care.

Some reviews confirm what might seem obvious. It hardly seems surprising that targeting a broad variety of language and decoding skills in preschool makes a real difference in later reading comprehension. Or that confining asylum seekers in detention centers negatively affects their mental health, both during detention and after release.

Others can be surprising. For instance, it turns out that Scared Straight programs are counter-productive. Rather than deterring young people from criminal behavior, they lead to “more offending behavior.”

And school-based programs to prevent violence in dating relationships do improve students’ knowledge and attitudes—but do little to change behavior.

Meanwhile, many point to the need for more research. We don’t have the evidence to determine whether refugee resettlement programs are working, whether interventions to reintegrate street kids are effective, or whether home-based literacy programs really do get children reading.