New Research Grants and Distinguished Fellowship to Focus on Reducing Inequality and Understanding the Use of Research Evidence
We are happy to announce five new research grants that will build stronger theory and empirical evidence in our focus areas of reducing inequality and the use of research evidence.
Three grantees will conduct research to inform responses to inequality in youth outcomes across diverse systems. One study will be the first to examine whether federal housing assistance improves children’s outcomes by reducing families’ housing cost burden. Other projects will investigate questions such as what drives teacher quality gaps between advantaged and disadvantaged schools, and how much of the racial gaps in academic and disciplinary outcomes are due to teachers’ differential treatment of students.
Two grantees focusing on understanding the use of research evidence will examine the role of intermediary organizations in leveraging research for education policy, and the news media’s role in brokering research evidence for federal policymakers, respectively.
In addition, a new Distinguished Fellow will immerse herself in two organizations that offer “makerspaces,” which provide opportunities for young people to acquire engineering knowledge and to design and create manufactured works in a collaborative community setting. The fellowship experience will support this researcher in developing an increased understanding of how day to day practice decisions affect learning opportunities for low-income young people and youth of color.
The research teams include scholars from a range of disciplines, including psychology, communication arts, economics, public policy, and education policy.
RESEARCH GRANTS: REDUCING INEQUALITY
Understanding Teacher Quality Gaps: How Did They Form, and How Can We Close Them?
What are the key sources of gaps in teacher quality, and what policies can help close these gaps?
Dan Goldhaber, American Institutes for Research; Theobald Roddy, Center For Education Data & Research, University of Washington
9/1/2015–8/31/2017, $460,433
Read More >
The Color of Emotion: Teachers’ Racialized Interpretations of Children’s Emotion and Student Outcomes
Do teachers’ racial biases and racialized interpretations of young children’s emotions and misbehaviors lead to differential treatment of students?
Amy Halberstadt, Psychology, North Carolina State University; Pamela Garner, Psychology, George Mason University; Sherick Hughes, Education, University of North Carolina
1/1/2016–12/31/2017, $349,980
Read More >
Can Housing Assistance Reduce Inequality Among Youth?
Does federally assisted housing lead to better outcomes for low income youth by making housing more affordable for their parents?
Sandra Newman, Health Policy & Management, Johns Hopkins University; Scott Holupka, Institute for Health and Social Policy, Johns Hopkins University
1/1/2016–12/31/2017,$384,424
Read More >
RESEARCH GRANTS: USE OF RESEARCH EVIDENCE
Intermediary Organizations and Education Policy: A Mixed-Methods Study of the Political Contexts of Research Utilization
How do intermediary organizations leverage research to drive education change across the country? How do intermediaries adapt to different political contexts?
Janelle Scott, Education, University of California, Berkeley; Elizabeth DeBray, Lifelong Education, Administration and Policy, University of Georgia; Christopher Lubienski, Education Policy, Organization and Leadership, University of Illinois, Urbana-Champaign
11/1/2015–10/31/2017, $443,944
Read More >
Tracking Policymakers’ Acquisition and Use of Research Evidence Regarding Childhood Obesity in the News Media
How does the news media influence the use of research evidence in policy deliberations and decisions regarding childhood obesity?
Itzhak Yanovitzky, Communication, Rutgers State University of New Jersey; Matthew Weber, Communication, Rutgers State University of New Jersey
1/1/2016–12/31/2017, $457,862
Read More >
DISTINGUISHED FELLOWSHIP
The Makerspace Movement: Sites of Possibilities for Equitable Opportunities to Engage STEM among Underrepresented Youth
Angela Calabrese Barton, Education, Michigan State University
Capacity-building and Communication Grants to Support Connections Between Research and Practice
Two new grants have been awarded to organizations working to enhance the use and usefulness of research and call attention to issues that affect young people in the United States.
American Educational Research Association
As part of its centennial celebration, the American Educational Research Association will focus its annual conference on promoting public scholarship. The Foundation’s grant will support a day-long Academic Deans Conference immediately preceding the main conference and a Strand of Public Scholarship Sessions during the conference. The Deans Conference will convene education school deans and other university leaders to focus on: 1) how to define public scholarship, 2) the criteria for what is considered meritorious scholarship, 3) metrics to measure and evaluate public scholarship, and 4) ways to align public scholarship with the missions of different academic institutions. The Strand of Public Scholarship sessions will consist of presentations designed to convey rigorous research for a broader audience, including education policymakers. The sessions will cover four major themes: the science of learning; preparing students for college, career, and lives of well-being; equitable access and opportunities for all students; and interventions, implementation, and taking to scale evidence-based ideas.
National Public Radio
Through reporting and features on a range of topics in education, the economy, health, crime, and housing, National Public Radio (NPR) has long brought attention to trends and developments that affect young people and families in the United States. The Foundation's 2016 grant represents a continuation of our ongoing support of these valuable efforts.
New Youth Services Improvement Grants to Strengthen New York City Nonprofits
Nine New York City community-based organizations have been awarded $25,000 grants to improve the quality of their youth programs.
Our Youth Services Improvement Grants program supports organizations that have demonstrated success but have also identified an area that if improved would enhance the quality of services. In addition to funding, grantees receive capacity-building consulting from the Youth Development Institute to support the implementation of projects and programs.
Sadie Nash Leadership Project
Dean Leadership Program
Sadie Nash wants to improve its Dean Leadership Program, which is a paid internship for young women (Deans) ages 19–25. Deans gain leadership skills and hands-on youth development experience while serving as role models, teachers, mentors, and advisors to Sadie Nash’s younger participants. The program offers two tracks: a summer institute and an after-school program. Discrepancies between the two tracks have resulted in different levels of competency and confidence between participants. Sadie Nash proposes to create a more consistent experience for Deans by redesigning the curriculum and expanding the training for the school-year Deans to better match that which the summer institute Deans receive.
Wave Hill
Improvements for Wave Hill’s Empowerment Programs for Teens: Forest Project and Woodland Ecology Research Mentorship
Wave Hill wants to improve its paid youth internship programs, The Forest Project and Woodland Ecology Research Mentorship. Participants in both programs do ecological restoration work in Bronx parks and woodlands while taking two related courses that are provided by Lehman College: Conservation of the Environment and Geographic Information Systems. Over time, the content of the courses has diverged from Wave Hill’s programming and has created a thematic gap between the academic and applied parts of the program. Wave Hill proposes to develop curricula for two courses that would be tailored specifically to its programs and would be taught on site at Wave Hill.
Pencil
Strengthening the Career Readiness of Teens through Improvement of the Capacity of the Fellows Program
Pencil wants to improve its Fellows Program, which provides career readiness training and a paid internship to rising and just-graduated high school seniors. The Fellows Program uses manual application and tracking processes, which have led to errors and have taken staff time that would be be better spent preparing students for internships. Pencil proposes to purchase and implement software that would automate the application and tracking systems. The new software will foster efficiency so that staff will have more time to get to know and train fellows, which will lead to better matches between student interests and job placements.
Jacob A. Riis Neighborhood Settlement House
Ravenswood College Access Program Intensive Track
Riis Academy provides college access, health, sports, and arts activities to young people drawn mostly most nearby public housing developments in Queens. Riis has had difficulty retaining its most hard-to-reach participants: 16–24 year olds who are disconnected from school and work. These youth are not on track for on-time high-school completion, or they have dropped out of high school and are unemployed. Jacob Riis proposes to reach out to older youth and provide more targeted services that will help them graduate high school or achieve its equivalency. Once the participants have graduated high school, Riis will help them pursue post-secondary education or find jobs. Intensive individualized case management along with GED/SAT prep, academic tutoring, college access workshops, job counseling and job shadowing will be some of the activities offered in this new track.
Waterwell Productions
WDP Special Education Initiative
Waterwell Productions provides the Theatrical Arts and Drama Programs for middle- and high-school students at the Professional Performing Arts School (PPAS). Department of Education mandates have significantly increased the number of special education students who attend general education classes at PPAS. Waterwell’s teaching artists are not trained to work with special education students, and are unable to meet the students’ needs in the classroom. Waterwell proposes to hire a consultant to develop a training curriculum and to create tools for teaching artists to work with students with special needs. The consultant will train Waterwell’s director of education to support teaching artists throughout the year and to provide professional development to new staff.
Chinatown Manpower Project
Project WIDE: Workplace Internship Development & Education
Chinatown Manpower Project wants to improve its Out-of-School Youth (OSY) program, which offers workforce development and education services to low-income, out-of-school Asian-American youth between the ages of 16–24. Although the OSY program has high completion rates, its participants are not successful in obtaining employment. Participants encounter barriers such as a lack of basic work skills and access to professional networks. Chinatown Manpower proposes to hire a part-time coordinator who will develop and run an internship program. The coordinator will build relationships with employers to gain contacts and to better understand the needs of the employers and their markets. The coordinator will use this knowledge during job readiness workshops to help participants improve the skills they will need to apply to and attain jobs in markets that are of interest to them.
Girls Write Now
Outreach & Enrollment Improvement Project
Girls Write Now (GWN) wants to improve its mentoring programs, which pair professional writers with girls who attend under-resourced public high schools. The goal of the program is to increase girls’ writing skills. GWN’s outreach and enrollment strategies have caused a mismatch in ethnic diversity between mentors and mentees. Also, the majority of mentors are based in Manhattan and Brooklyn, while mentees are based in all boroughs, which leads to extended travel times to get to pair meetings. Girls Write Now wants to increase the number of women of color in the mentor pool, and to redistribute mentor geographic locations to better match mentee locations in all five boroughs.
I Challenge Myself
Building Blocks Training Videos
I Challenge Myself provides physical education programs to New York City high-school students, and wants to improve its complementary X-Fit and Cycle Smarts programs. X-Fit is a crossfit exercise program that is designed to develop students’ physical strength and confidence so that they will build up to the more challenging Cycling Smarts program. ICM’s staff have observed that coaches are not able to properly model the fitness exercises for students. Students are not getting the full benefit from the programs, and are at risk of getting injured. ICM proposes to contract a filmmaker to film and edit its staff performing the exercises that make up X-Fit and Cycle Smarts. The resulting video will supplement the one-time ICM instruction, written curriculum, and still photos that are currently used to train coaches. Coaches will also be able to share the videos with students during the physical education classes.
Epic Theatre Center
Epic Remix/NEXT Service Improvement Program
Epic Theatre Center wants to improve its after-school program, Epic Remix, where high school students work with professional teaching artists to write and perform adaptations of Shakespeare plays. Although participants of Epic Remix have a 95 percent high-school graduation rate, their college attendance and retention rates are much lower. Epic proposes to partner with CATES Educational Testing Service to establish college preparatory services for all Epic Remix participants. CATES instructors will train Epic artists to deliver SAT/ACT preparation, college awareness and application preparation, while Epic artists will train CATES instructors in the Epic method of teaching. A new curriculum will be created, which will weave college prep services into Epic’s program and will be implemented by both CATES and Epic staff.
2016 Guidelines for Applying for Youth Services Improvement Grants
Updated for 2016, the application guidelines for the Youth Services Improvement Grants program outlines eligibility and selection criteria, as well as detailed instructions on preparing and submitting an effective application.
The online application will open on February 1, 2016. All applications must be received by March 11, 2016 at 3:00 p.m. EST.
Evidence at the Crossroads Pt. 8: Building an Improvement Infrastructure
By Donald J. Peurach
Now that we know these federal grantmaking initiatives will continue, how can we ensure that investments in innovation are maximized? What can the next generation of tiered evidence initiatives learn from the investments thus far? Several writers in this series have focused on program impacts. In this entry, I argue for building a complementary infrastructure focused on program improvement.
The Challenges and Learning Needs of Practicing Innovators
From 2011–2014, the William T. Grant and Spencer Foundations sponsored a learning community of program developers and practitioners funded through i3, a group I refer to as “practicing innovators.” The i3 learning community offered grantees a venue to share their experiences and insights with each other. It also brought their perspectives to bear on policy conversations about incubating educational interventions. Because of my work studying the scale up of educational programs, I was invited to participate as a consultant in these meetings and was able to witness educational innovation from the perspectives of both policy and practice.
All told, the i3 program stands as a powerful example of the possibility of using policy to shape the work of practicing innovators. It also stands as an important example of the ways in which policy can introduce new challenges for practicing innovators. By design, i3 introduced structure by requiring that practitioners develop clear intervention designs, work timelines, and budgets. They also had to establish clear goals and evaluation criteria to discipline their work. At the same time, the i3 program stretched practicing innovators to develop a much broader array of capacities and skills. This is no surprise. Such challenges are endemic in large-scale educational innovation. For i3 grantees, the challenges arose in three key areas: 1) building new collaborations, 2) working with evaluators, and 3) leading and managing complex organizations.
“All told, the i3 program stands as a powerful example of the possibility of using policy to shape the work of practicing innovators.”
1. Building new collaborations and negotiating fidelity-adaptation tensions
The i3 program has pressed grantees to collaborate with more (and more varied) districts and schools than they had in the past, often using interventions that had to be revised and extended in new ways. Scaling required managing issues of recruitment and retention while, at the same time, building open, trustful relationships that supported productive collaboration. For example, the IDEA Public Schools were awarded an i3 development grant to expand professional development to 600 new teachers, 400 instructional leaders, 24 new principals, and 160 aspiring teacher leaders to support the human-capital pipeline in the Rio Grande Valley. ASSET STEM Education, a validation grantee, proposed providing comprehensive professional development to teachers across Pennsylvania in K-6 standards-aligned STEM instruction. Reading Recovery, a scale-up project, proposed training 15 new Teacher Leaders and 3,750 new Reading Recovery teachers while also establishing new training sites in rural areas to target low performing schools and high needs students.
Scaling required balancing a deep tension between local adaptation and fidelity of implementation. On one hand, i3 grantees recognized a strong need to support local adaptation in order to manage uncertainty and increase effectiveness in fielding complex interventions across increasingly diverse arrays of schools. On the other hand, the i3 program placed a premium on fidelity of implementation in order to produce evidence of program impact, and local adaptation risked corrupting their evaluation designs.
The result was a breakdown in a fidelity-adaptation synergy that has driven continuous improvement among leading educational innovators. In the past, organizations such as Success for All and Reading Recovery (both i3 scale up grantees) navigated challenges arising through new collaborations using an approach to learning and improvement that had schools faithfully implementing practices that had been validated by research and experience while also adapting and extending these practices to address local needs and, then, propagating the new and promising practices in other sites. The i3 program design left grantees with a dilemma. Adapting their programs risked compromising their evaluations. However, not adapting them risked weak evidence of program impact as a result of continuing to implement dimensions of their programs that they recognized as problematic.
“Scaling required balancing a deep tension between local adaptation and fidelity of implementation.”
2. Working with evaluators
The i3 program also has brought innovators and evaluators into closer working relationships than has been typical in educational policy and reform. But because the two groups bring different priorities to projects, striking positive working relationships is no simple matter. For instance, innovators aim to produce replicable, effective interventions that are also responsive to local needs, opportunities, and problems. Doing so requires maintaining flexibility and adaptability in their programs. Evaluators aim to complete successful studies that yield unbiased and rigorous assessments of program effectiveness. Yet that, in turn, requires program stability and fidelity.
Crafting trustful, collaborative, and productive relationships required managing these competing priorities. Doing so was especially important in managing the tension between fidelity of implementation and local adaptation. As one grantee with a validation project explained:
“One challenge for us was teacher attrition. We had instances of teachers leaving their schools after the first year of the program. From our perspective, we needed to provide training to replacement teachers to maintain high-quality implementation in the school and to make sure that students continued to benefit. But that meant that a second year or third year school would have teachers who were only in the first year of implementation. Also, we were adapting our training every year as we learned from experience, including moving some of our online training to face-to-face coaching. So, these new, first year teachers would have slightly different training from the other teachers. But we knew that we were a validation project, and we had to make sure that these changes would be okay with our evaluators before making the final decision about how to proceed.”
Managing the tensions required that innovators and evaluators learn to work together in new ways. Logic models and measurement models became the media through which they negotiated shared understandings and struck agreements on the nature and scope of their work. Early on, some evaluators worked in pseudo-coaching roles, as they guided innovators in clarifying the logic of their program designs. Practicing innovators were then free to adapt and improve their programs in ways that did not compromise the logic and measurement models around which their evaluations were structured. These logic and measurement models did not exist at the outset of the i3 program. Rather, they emerged and evolved over the course of the program as a product of the collaborative relationships between innovators and evaluators.
3. Leading and managing complex organizations
While most of the i3 grantees had some prior experience managing educational innovation, few, if any, were formally trained or fully prepared to manage the entire scope of work and the uncertainty they encountered while working within the i3 program and scaling their interventions. Rather, all of the members of the learning community found themselves managing networks, programs, organizations, relationships, and trade-offs that were in some way new to them. For example, extending and modifying intervention designs required increasing their development staffs. Further, working with more (and more varied) districts and schools required increasing their training/coaching staffs. Working with external evaluators and with U.S. Department of Education personnel required expanding their executive and managerial staffs. While critical to the success of their programs, much of this organization-building occurred “behind the scenes” and sometimes without support.
Absent established traditions of research and professional development focused on the practice of educational innovation, most leaders found themselves addressing their learning needs either through reflective practice or through participation in communities of practice. As one participant with a development grant explained of the i3LC experience:
Especially in the early years of the grant, the i3 Learning Community was our most important resource for innovation implementation support. The opportunity to meet with like-minded and like-challenged colleagues in a place where there was no judgment was invaluable. We knew we could speak honestly, and there would always be thoughtful people who would understand and help us work through our decision making process and then reflect on the results.
Building an Improvement Infrastructure
While the i3 program has introduced more structure and discipline into the practice of educational innovation, so too has it introduced new challenges that will inhibit the program’s ultimate success. Developing system-level “improvement infrastructure” that supports practicing innovators in addressing these challenges could be a game changer.
“Balancing impact and improvement is not a matter of doing the impossible. Rather, it is a matter of duplicating success.”
After all, the structure and discipline introduced through the i3 program is very much a positive artifact of highly developed “impact infrastructure”: a system of political and policy supports emphasizing evidence as both an input to and output of the practice of developing effective, scalable educational innovations. The i3 program is one component of this impact infrastructure: a competitive grant program that structures requirements, resources, and incentives to support the use of evidence in innovation. The i3 program is supported by a web of interdependent federal policy initiatives supporting the use of evidence in innovation, including the establishment of the Institute for Education Sciences, the creation of the What Works Clearinghouse, and investment in the advancement of statistical methods and in early career professional development for researchers. It is further support by philanthropists advancing their own evidence-driven competitive grant programs, a population of private firms with capabilities for research and evaluation, and a powerful professional organization—the Society for Research on Educational Effectiveness—committed to advancing the cause.
This impact infrastructure, in turn, could provide a blueprint for building a complementary infrastructure focused on improvement.
Policy supports
A parallel improvement infrastructure could begin with a complementary policy web that promotes and supports continuous improvement. This policy web could include adapting the i3 program to support a sort of “novice–intermediate–expert” progression in creating capabilities for continuous improvement in sponsored projects. Support at the development level could focus on using design-based research to test and refine key practices and components. Further, support at the validation level could focus on developing capabilities in schools to enact evidence-driven Plan–Do–Study–Act cycles to adapt programs to address local needs. Support at the scale up level could focus on developing infrastructure linking the enterprise into a coherent, evolving learning system. This policy web could also include agencies to champion and monitor the work of continuous improvement, as well as initiatives aimed at investing in the development of formal methods of continuous improvement and in the development of researchers able to support the use of these methods.
Philanthropic, private, and professional supports
Already, elements of a complementary, private sector web are forming as key components of an improvement infrastructure: for example, the emergence of the Carnegie Foundation for the Advancement of Teaching and the SERP Institute as organizations championing the push for continuous improvement in educational innovation, matched by a community of researchers in universities and private organizations who are advancing and popularizing methods of design-based research. The growth and maturation of this private sector web could potentially be accelerated through federal resources and incentives that would draw in additional organizations, for instance by establishing requirements in competitive grant programs to incorporate formal methods of continuous learning and improvement. But it could also go further, sponsoring competitive grants programs that engage these organizations directly and provide resources and incentives to craft partnerships with practicing innovators.
Political supports
The bedrock of the impact infrastructure is political support anchored squarely in what Harvard University professor Jal Mehta aptly describes as the “allure of order”: longstanding faith among policymakers in the potential to use principles of rational management to discipline otherwise “soft” educational practices. The allure of order has deep roots in norms of rationality that have long dominated educational politics and policymaking in the U.S., and it is reinforced by longstanding appeal to both business and medicine as sources of ideas and legitimacy to support educational reform.
Perhaps the biggest challenge to building an improvement infrastructure lies in extending political discourse to include an understanding of improvement that parallels the allure of order. Indeed, ideas are emerging in both business and medicine for doing so. Possibilities include the introduction of language emphasizing “infrastructure building” (rather than “turnaround”) as an approach to improving weak schools, the use of the tech sector’s notion of “perpetual beta” as characterizing the work of educational innovation (rather than the pursuit of “What Works”), and the use of Atul Gawande’s notion of “better” (rather than “scientific knowledge”) as an outcome of improvement-focused educational innovation.
The trick will lie in understanding how these notions have been used to shape and influence broader discourse and understanding in other sectors, and ultimately adapting these strategies to education and other social sectors.
Looking forward
There is no doubt that cultivating a highly developed, coordinated improvement infrastructure will be difficult. However, the emergence of the impact infrastructure provides evidence that it is possible.
Indeed, the development of an improvement infrastructure can actually be interpreted as the essence of innovation: working iteratively to make incremental improvements to well-reasoned, promising, yet inevitably imperfect strategies and plans. In this case, such incremental improvement would center on maintaining the positive benefits that have followed from a keen, sustained focus on impact while, at the same time, developing and sustaining an equally keen focus on continuous improvement.
Viewed from this perspective, balancing impact and improvement is not a matter of doing the impossible. Rather, it is a matter of duplicating success.
Donald J. Peurach is Associate Professor at the University of Michigan School of Education.
In “Evidence at the Crossroads,“ we seek to provoke discussion and debate about the state of evidence use in policy, specifically federal efforts to build and use evidence of What Works. We start with the premise that research evidence can improve public policies and programs, but fulfilling that potential will require honest assessments of current initiatives, coming to terms with outsized expectations, and learning ways to improve social interventions and public systems. Join the conversation by tweeting #EvidenceCrossroads .
Evidence at the Crossroads Pt. 7: Using Evidence To Do the Most Good, Even When it Reveals an “Inconvenient Truth”
By Patrick T. McCarthy
Our supply of evidence about What Works, though still too small, is beginning to grow. I want to focus here on how we use evidence—how we put it to work to help more of our nation’s children and families achieve their full potential. Let’s focus on two important aspects of this question: building the bridge from programmatic evidence to population-level impact and reacting wisely to evidence that reveals an “inconvenient truth.”
Most evidence-building research zeroes in on a particular programmatic intervention and tells us whether it achieved desired results with a specific group at a given place and time. When we are lucky, we see those results replicated in different settings, giving us even firmer ground for saying that this intervention works. Such evidence can help policymakers and program managers make wise decisions about where to invest their scarce dollars so they will do the most good. That is very, very good, but if we stop there we leave unrealized a lot of the potential of good evidence to drive great outcomes for all of our nation’s children and families.
To get to population-level scale, we need to embed the best of What Works into the large public systems that serve not a few hundred children or families, but thousands upon thousands. In part, successfully embedding evidence-based programs in large systems requires them to be adapted to the way the systems work so that referral pathways are smooth, contracting provisions are appropriate and the program has secure access to a dedicated funding stream. A greater and ultimately more important challenge is to draw lessons from the evidence and apply those insights to what public systems do and how they do it. When that happens, those systems can achieve better outcomes for all of the children and families they serve.
To do this, we have to look across multiple evaluations to see patterns, clusters and commonalities. If, for instance, you look at the evidence from the multiple evaluations of Multi-Systemic Therapy, Functional Family Therapy, the Oregon Treatment Foster Care model, and similar programs, common success factors begin to emerge. They all place a lot of emphasis on training and supporting their staff to effectively engage families who often are in crisis and hostile to outsiders at the very point that staff arrive on the scene. There is no moment more critical to the outcome of an intervention than these early steps to create the conditions in which therapeutic work can succeed. And there is no one for whom getting those first relationship steps right is more critical—and more challenging—than public system workers who arrive on a family's doorstep talking about taking one of their children away or locking up one of their teenagers. If we can isolate the critical success elements common to evidence-based programs and build those into the training and supervision of frontline workers, we should begin to see every family that encounters one of these systems having much better odds for success.
The second issue related to using the evidence we are accumulating is how we act on evidence that challenges prevailing paradigms, especially when such programs don't just not work, they actually do harm. The most egregious example we have as a country are youth prisons. We have known for decades and without a shadow of a doubt that these facilities, which contradict everything we know about adolescent development, cause harm to young people and create more rather than less crime in communities. Contrary to all available evidence, we continue to lock up two to three times more young people than any other nation. We are beginning to hear calls for change from more and more quarters, and we see political will and leadership emerging. To encourage these efforts and to spark even more, in a recent TEDx talk I committed the Casey Foundation to work with any state that makes the decision to do the reform work and make the investments necessary to successfully and safely close these factories of failure and replace them with programs and facilities that help young people get back on track.
Evidence doesn’t turn itself into policy, especially when it contradicts prevailing paradigms or entrenched funding streams. If we are serious about a What Works movement, we can't allow ourselves or other decision makers to pick and choose which results we want to act upon. Philanthropy can help by investing in developing and testing alternatives, supporting evidence-driven advocacy, and underwriting technical assistance to evaluators and decision makers.
The conversation about What Works continues, thanks in part to other contributors in this series, including Ron Haskins and Gordon Berlin, as well as many others in the field. I hope this ongoing conversation includes attention to what happens as a result of the evidence. The futures of our nation’s young people depend on it.
Patrick T. McCarthy is President and CEO of The Annie E. Casey Foundation
In “Evidence at the Crossroads,“ we seek to provoke discussion and debate about the state of evidence use in policy, specifically federal efforts to build and use evidence of What Works. We start with the premise that research evidence can improve public policies and programs, but fulfilling that potential will require honest assessments of current initiatives, coming to terms with outsized expectations, and learning ways to improve social interventions and public systems. Join the conversation by tweeting #EvidenceCrossroads .
Finalists for 2016 William T. Grant Scholars Program Awards
We are proud to announce the finalists for the William T. Grant Scholars Program. These 10 early-career researchers were selected from a pool of 63 applicants after a rigorous review by our staff and selection committee.
Applicants for the Scholars Program propose five-year research and mentoring plans designed to expand their skills and knowledge in a new discipline, content area, or method. Applicants are nominated by their institutions and only one applicant can be named from any one major division (e.g., College of Arts and Sciences, medical school, etc.). Nominating institutions must formally agree to support the applicant’s research goals with sufficient resources during the grant period, so the nomination reflects the institution’s serious commitment to the applicant and his or her career.
The 10 finalists will be interviewed in February 2016 and four to six Scholars will be announced in March. Selected Scholars will each receive $350,000 over five years and participate in annual meetings.
The Scholars Program began in 1982 and has a rich history of supporting the development of early-career researchers in the social, behavioral, and health sciences.
The 2016 William T. Grant Scholars finalists are:
Matthew Desmond, Ph.D., Department of Sociology and Social Studies, Harvard University
Mesmin Destin, Ph.D., Department of Human Development and Social Policy/Psychology, Northwestern University
Laura Hamilton, Ph.D., Department of Sociology, University of California, Merced
Jacob Hibel, Ph.D., Department of Sociology, University of California, Davis
Jennifer Kubota, Ph.D., Department of Psychology and Center for the Study of Race, Politics, and Culture, University of Chicago
Kristin Long, Ph.D., Department of Psychological and Brain Sciences, Boston University
Jennifer McDermott, Ph.D., Department of Psychological and Brain Sciences, University of Massachusetts
Taryn Morrissey, Department of Public Administration and Policy, American University
Matthew Notowidigdo, Ph.D., Department of Economics, Northwestern University
Kristin Turney, Department of Sociology, University of California, Irvine
The Next Phase in our Research Focus: Improving the Use of Research Evidence
The William T. Grant Foundation has long supported research to improve the lives of young people, but we know that there is considerable distance between the research we support and youth outcomes. The stakes of omitting or misusing research evidence are high, but too often research is absent from deliberations about programs and practices for youth. And the information needs of decision makers working on behalf of youth too rarely shape research agendas.
For the past six years, the Foundation has taken up these challenges by supporting research to increase understanding of how research evidence is acquired, understood, and used, as well as the circumstances that shape its use in decision making. These projects have deepened our understanding of the users of research and their worlds and offered fresh ideas about how to improve the use of research evidence. For example, the work revealed the prominent role of intermediary organizations—think tanks, foundations, vendors, program developers, and advocacy organizations—in shaping the production and use of research evidence. Studies also underscored the importance of cultivating trusting relationships between researchers, intermediaries, and decision makers, and establishing structural supports to make sense of research evidence and produce research that is responsive to the concerns of decision makers.
A NEW DIRECTION
Understanding the problem of research use is not enough. As we embark on the next phase of this initiative, we want to support studies of how to improve the use of research evidence in ways that benefit youth.
To that end, we are shifting our focus from understanding how and under what conditions research is used to understanding how to create those conditions. This shift is significant in three ways.
First, we invite studies that identify or test actionable strategies to improve the use of existing research. This includes descriptive studies that reveal the strategies, mechanisms, or conditions for improving research use. This also includes evaluations of deliberate efforts to increase routine and beneficial uses of research in deliberations and decisions that affect young people.
Second, it invites teams to identify or test strategies for producing more useful research evidence. This includes examining incentives, structures, and relationships that facilitate the production of research in ways that respond to decision makers’ needs and optimize researchers’, decision makers’, and intermediaries’ joint work to benefit youth.
Third, it calls for projects that test the assumption that using high quality research in particular ways improves decision making and youth outcomes.
NEXT STEPS
We’ve updated our research grants application guidelines and have created new supplementary guidance for researchers who are interested in studying how to improve the use of research evidence in ways that benefit youth. Senior Program Officer Kim DuMont has also written an essay taking stock of the Foundation’s use of research evidence grants portfolio from 2009-2015, which gives a comprehensive look at the history of the initiative, highlights three themes that emerged from findings, and signals major questions that inform the next phase of the initiative.
We invite you to look over these materials, share them with your colleagues, and consider how improving the use of research evidence can positively impact the lives of young people. The next deadlines to submit a letter of inquiry for research grants are January 12, 2016 and May 5, 2016. The online application portal is open now.
The first two sections of this post are excerpted from the new supplementary guidance for studying the use of research evidence. Kim DuMont’s essay originally appeared in our 2014 annual report.
New Report: The Role of Exclusion, Social Capital, and Generic Social Processes in Upward Mobility
Despite top-down measures and informal trends that would seem to encourage integration by providing opportunities for social interactions across socioeconomic and racial divisions, social exclusion persists in the United States, particularly among the poor and disadvantaged.
What processes isolate the disadvantaged and thereby limit the accumulation of social capital that would help them get ahead? And how can research illuminate responses that would reduce inequality and improve upward mobility? Duke University’s Linda Burton and Whitney Welsh tackle these questions and more in a new report that focuses on the often unseen micro level social processes that divide our society and preserve an unequal status quo.
Burton and Welsh outline the geographic segregation of the poor across the United States in the new millennium, highlighting the ways that Americans across the country “are either exposed to, turn a blind eye toward, or are deeply embedded in lived experiences that involve micro social processes that exclude the poor.” These processes, which take place both between and within groups of haves and have-nots, stymie opportunity and prevent the poor from developing the social bonds and resources that would mitigate the negative effects of poverty and isolation.
While many macro-level policies and interventions to address poverty are essential to the survival of the poor, a closer consideration of how these policies or programs are experienced at the individual level, and how micro processes, relationships, interactions, and discourse promote or inhibit opportunity, may improve efforts to reduce inequality by identifying and ultimately removing barriers to social capital for those who need it the most.
As Burton and Welsh state, “Research designed to capture these dynamics is needed to expand our understanding of how social exclusion and the macro and micro processes associated with it are shaping the growing inequalities between and within populations in the U.S. and what factors might mitigate the deprivation of social capital among the poor.”
Download Inequality and Opportunity: The Role of Exclusion, Social Capital, and Generic Social Processes in Upward Mobility
2016 Guidelines for Applying for Research Grants and Distinguished Fellowships
Updated for 2016, our new application guides describe the Foundation’s current research focus areas and outline our funding priorities for research grants and Distinguished Fellowships.
One of the most significant updates for 2016 is the announcement of the new direction for our use of research initiative. We will now aim to identify, create, and test strategies to improve the use of research evidence in ways that benefit youth. Our shift in focus from understanding how and under what conditions research is used to understanding how to create those conditions is outlined in detail in the supplementary guidance on studying the use of research evidence.
In addition to defining key terms and providing profiles of recent grants that illustrate the types of projects we support, the guides also offer step-by-step instructions submitting a letter of inquiry or application online.
Research Grants Application Guide
Supplementary Guidance on Studying the Use of Research Evidence
Distinguished Fellowship Application Guide
Further information for applicants, including the new guides, as well as FAQs, supporting materials, and descriptions of existing grants, can be found on the Grants page. You may also find resources related to our current research interests, including blog posts, papers, presentations, and technical tools on the Focus Areas page.
Evidence at the Crossroads Pt. 6: Evidence is Only as Good as What You Do With It
By Mark Lipsey
We are living in a time of unprecedented systematic research on the effectiveness of interventions that are intended to produce better outcomes. This level of effort is producing a substantial volume of intervention research, but a critical question is what to do with these studies?
The federal government has made massive investments to stimulate and support research on intervention effectiveness. One form of this that should not be neglected is the grant programs administered through the National Institutes of Health, the Institute of Education Sciences, the National Institute of Justice, and other such agencies that support field initiated intervention research in their respective topic areas. And one of the most notable developments, as has been well described in previous posts in this series, is the launch of a number of federal initiatives that provide tiered funding to support programs differentially according to the strength of supporting evidence. Most important for present purposes is the common requirement that the funded programs be evaluated, adding still further research to our stock of evidence about what works.
These initiatives will report a range of findings on a diversity of outcomes—some positive, some inconclusive, some possibly negative. How do we extract from these findings insights that can contribute to our understanding of what works and what doesn’t? And how do we determine which findings have general applicability and thus identify programs and practices that could be effectively scaled up, and which ones are specific only to the situations and circumstances in which a particular study was done?
Drawing Conclusions
It will be tempting to try to interpret each study individually. If such a study shows positive effects, we might then conclude that the intervention it tested must be effective and we should promote its wider use. That would be a mistake. Any individual study has idiosyncrasies of method, circumstances, participants, intervention particulars, local support, and the like, along with a dose of happenstance, that make it uncertain whether the same results would occur if the study was replicated, and even more uncertain that those results would generalize to other settings, participants, etc. This is not to say that each study does not make an important contribution to knowledge. It does, but we must be careful about overinterpreting the implications of the findings of any one study.
If we are to draw conclusions with practical implications from intervention research, multiple studies are needed to provide a sufficiently broad base of evidence to support generalization beyond the idiosyncrasies of a single study. There is much talk of replication these days, and replication of the results of an intervention study is informative. But if what we want to know is how confident we can be that the findings generalize to other contexts, then multiple studies with realistic variations will be more informative than attempts at exact replication. Relatively consistent findings across different participant groups, implementation contexts, and variants of the intervention itself provide evidence that the intervention is robust to the kinds of variation likely to occur in actual practice. Inconsistent findings, on the other hand, provide clues to where the boundaries of generalization are—the circumstances under which the intervention may not work so well.
With multiple studies, of course, extracting and interpreting the information useful for practice and policy presents an even bigger challenge. One very intuitive approach would be to categorize the studies according to whether they find positive, null, or negative effects and then try to figure out what differentiates these groups. That too would be a mistake. By what criteria would we categorize the findings of each study? Individual studies universally characterize their effects according to their statistical significance, and it is that indication that is typically relied upon as an indication of whether a positive effect was found. That makes sense for a single study—it provides some protection against claiming effects that are likely to have occurred only by chance. But statistical significance is a joint function of the size of the sample and the magnitude of the intervention effect. When reviewing multiple studies, it is necessary to focus on the magnitude of the effects so that smaller studies that find large effects are not overlooked, and very large studies that find small effects are not overinterpreted.
Integrating Findings
As many readers will recognize, we have a well-developed procedure for integrating the findings of multiple intervention studies with a focus on the size of the effects rather than their statistical significance, and for exploring the characteristics of those studies that are associated with the effects they find. It is meta-analysis.
In meta-analysis, statistical metrics for the magnitude of the effects on each outcome (effect sizes) are systematically extracted from each study along with a profile of study characteristics related to methods, participant samples, intervention characteristics, implementation, setting, and the like. Analysis is then conducted to describe the distributions of effect sizes on different outcomes and, most important, to explore the relationships between the study and intervention characteristics and the nature and magnitude of the effects. The results, then, characterize the findings of a body of research, not just individual studies, and does so in a differentiated way that attempts to identify the factors associated with differential outcomes.
Meta-analysis of broader and narrower bodies of intervention research is quite common, usually undertaken by academic researchers and promoted by such organizations as the Campbell Collaboration and the Cochrane Collaboration. However, it has not been integrated into the plans of many of the government agencies sponsoring the various tiered-evidence initiatives that are underway as a method for integrating and interpreting the findings of the many studies those initiatives are producing. There are exceptions, and some indications of movement in that direction. The Office of Adolescent Health, for instance, is sponsoring a meta-analysis of the studies of teen pregnancy prevention it, and some of its sister agencies, have initiated. Similarly the Centers for Medicare and Medicaid Services is undertaking meta-analysis of the research done under its Health Care Innovation Awards initiative. Other agencies may be making similar efforts or contemplating them, but this perspective is not widespread.
Initiating a meta-analysis of the studies developed under any of these initiatives, however, is something best planned from the beginning, not decided after the studies are underway. To support the most informative meta-analysis, there are some advantageous features that can be built into the initiative at the start. For example, the distribution of studies is important. It will be more informative to have multiple studies on practical variations of each intervention in a selected set of interventions than one study on each of a set of distinctly different interventions.
Most important, however, is the opportunity to specify in advance the kind of detailed information that each local evaluator should collect and report in order to provide a rich set of variables for the meta-analysis to explore. These should include many particulars related to the implementation of the intervention, the participants who receive it, the organizational and service delivery context, and other such factors that are often not reported in sufficient detail to support the most informative meta-analysis. Agencies sponsoring (and paying for) multiple studies under a single initiative can require a level of consistency in the way those details are reported that is rarely attained when study authors are left on their own to decide what to report.
Guiding Practice
The promise of this approach is not simply that the most effective interventions will be identified in a systematic and methodologically credible way, though that will be one result. That form of knowledge allows dissemination of program and practice models expected to be effective if they are implemented with fidelity, that is, in the same way they were in the supporting research. Having a repertoire of such models is indeed a big step forward in the quest to find and scale up effective programs and practices. However, despite the evidence, there will be many reasons why providers and practitioners will not adopt those models and, if adopted, will do so with adaptations that change them from the original evidence-based version.
The larger promise from sufficient bodies of evidence and differentiated meta-analysis is identification of the principles that make the respective programs and practices effective. Knowing why something works, and not just that it works, provides an explanation—a theory if you will—that can guide effective practice in flexible ways amenable to local adaptations and practical constraints, so long as those variants preserve the underlying change mechanism that makes the intervention work.
We need a cookbook full of recipes for effective practice, but even better is knowing how to create recipes for effective practice from the ingredients on hand in the local kitchen.
Mark Lipsey is a Research Professor in the Dept. of Human & Organizational Development and Director of the Peabody Research Institute and Vanderbilt University.
In “Evidence at the Crossroads,“ we seek to provoke discussion and debate about the state of evidence use in policy, specifically federal efforts to build and use evidence of What Works. We start with the premise that research evidence can improve public policies and programs, but fulfilling that potential will require honest assessments of current initiatives, coming to terms with outsized expectations, and learning ways to improve social interventions and public systems. Join the conversation by tweeting #EvidenceCrossroads.
Evidence at the Crossroads Pt. 5: Improving Implementation Research
By Barbara D. Goodson
Over the past two decades, the education research and policy landscape has been shaped by the concept of using evidence to make policies or adopt practices. And not just any type of evidence, but high-quality evidence generated by “scientifically valid research.” The methods for assessing intervention effectiveness are becoming a matter of greater consensus, thanks to the development of agreed-upon standards of evidence for evaluating the impact of interventions. But, in an environment where effective programs are often adopted and implemented in new contexts, is it time to focus on implementation research that can support this adoption?
A Focus on Evidence of Effectiveness
Today’s definitions and standards of high-quality research and evidence are far more specific because of well-articulated systems for assessing the evidence generated by studies of intervention effectiveness. For instance, the What Works Clearinghouse (WWC), developed in 2002, seeks to inform decision making by rating evidence from individual studies and from a body of research on particular interventions or outcomes. Similar evidence rating systems now exist for other policy fields. The U.S. Department of Labor’s Clearinghouse for Labor Evaluation and Research (CLEAR) and the GRADE rating system in health care are two examples of systems that assess the strength of evidence gathered from studies of program or policy effectiveness.
This focus on effectiveness continues to grow and sustain itself as more evidence is produced. The Obama Administration, for example, has embedded a strong evalu¬ation focus into many new funding initiatives, such as the Investing in Innovation Fund (i3), which uses a tiered evidence framework to provide greater funding to programs with scientifically valid evidence of impact. If one of the goals of initiatives like i3 is to generate more rigorous evaluation of program effectiveness in a given field, this is certainly being accomplished. Over 90 percent of the 150 programs funded by i3 between 2010 and 2013—an unprecedented majority—are conducting impact evaluations with the potential of generating valid and credible evidence on effectiveness. This includes all of the larger grants and even a majority of smaller grants that were not required to conduct rigorous studies.
What About Evidence on Implementation?
More recently, though, evidence on the implementation of effective practices has received increased attention.
Decisions about which practices to adopt depend on a variety of considerations, including questions of implementation. Stakeholders need evidence that tells them about the feasibility of implementing the new practice in similar contexts or with certain populations, as well as data on the costs associated with adoption. These potential adopters also need explicit instructions on how to implement the practice, and how to provide necessary supports, such as training, coaching, or infrastructure.
If impact evidence is believed to lead to the dissemination and adoption of improved educational practices, then we need to think more systematically about how to support this adoption. We need to think about the types and quality of evidence on implementation of practices.
Concern now centers on whether standards of what constitutes high-quality implementation evidence may be developed to parallel the standards now commonly accepted for evidence on effects.
The Investing in Innovation Fund is playing a key role in promoting a focus on implementation by setting expectations that grantees conduct high-quality implementation studies that are intended to provide data to support replication and broader dissemination of effective practices. But whereas i3 uses WWC standards as the metric against which to measure the strength of impact evidence generated by grantees, it does not have a comparable set of standards to gauge for strength of implementation evidence.
The absence of commonly accepted standards for implementation studies led i3 to ask its national evaluator, Abt Associates Inc. and its partners, to propose a set of evidence standards for measurement of implementation for programs funded through the initiative. The standards that have been developed are based on a comprehensive logic model, a system for measuring the fidelity of implementation of the key inputs identified in the logic model, and annual assessment and reporting of the extent to which fidelity of implementation of the key inputs has been achieved.
In light of the i3 experience, and the increasing expectations among funders that high-quality implementation evaluations be conducted in concert with rigorous impact studies, the field needs to take the next step toward systematic standards for research evidence on implementation. Even if it is too early to think about standards such as those delineated by the WWC, is it time to develop guidance for researchers that would, at a minimum, lay out types of information on implementation that they should collect in different evaluation contexts (formative, efficacy, effectiveness, or scale-up studies, for example), implementation measures, and ways to analyze and report data on implementation. As part of the evolution of standards for implementation studies, we can provide signals to researchers about best practices in measuring implementation.
Ultimately, if our search for effective reforms for educational practice is successful, having strong and reliable evidence on implementation will be crucial for enacting real reform in our schools.
Barbara D. Goodson Ph.D. is a principal scientist at Abt Associates, where she is the principal investigator on two large studies being conducted for the Institute for Education Sciences in the Department of Education. One of these studies is the National Evaluation Technical Assistance for the i3 Investment and Innovation Grants, where she leads the technical assistance to 150 grantees conducting randomized and quasi-experimental impact and implementation studies of a variety of educational innovations.
In “Evidence at the Crossroads,“ we seek to provoke discussion and debate about the state of evidence use in policy, specifically federal efforts to build and use evidence of What Works. We start with the premise that research evidence can improve public policies and programs, but fulfilling that potential will require honest assessments of current initiatives, coming to terms with outsized expectations, and learning ways to improve social interventions and public systems. Join the conversation by tweeting #EvidenceCrossroads .
Evidence at the Crossroads Pt. 4: The Obama Behavioral Insights Team, an Important Addition to Evidence-based Policy
By Ron Haskins
Although there are numerous signs at both the federal and state levels that evidence-based policy is influencing policymakers, a lurking danger in any movement is that it will fall short of its potential unless it continues to expand. Concern that the evidence-based policy movement is merely a flash in the pan should be at least somewhat assuaged by the breadth the movement has already achieved.
Perhaps the most far-reaching element of the movement are the six evidence-based initiatives started by the Obama administration, which are now supporting well over 1,400 programs at the local level in teen pregnancy prevention, home visiting, preschool and k-12 education, community-based programs, and employment and training. The Teen Pregnancy Prevention Initiative and the Maternal, Infant and Early Childhood Home Visiting Program are especially notable. These two initiatives demonstrate several of the characteristics that seem certain to become classic features of evidence-based policy. Notably, both illustrate the advantages of tiered funding initiatives in which most of the federal grant funds (75 percent in both cases) go toward programs with strong evidence of producing impacts on important outcomes, while smaller grants go toward promising and innovative programs that are comparatively untested. This arrangement ensures that most funds are spent on programs that maximize the chances of producing impacts, while simultaneously providing adequate funds to develop new programs that may prove their worth in the future. The pipeline to innovation must remain open because new and more effective programs are always needed.
But the distinct approaches that are sustaining the evidence-based movement do not end with the Obama initiatives. Joining the parade are the reforms of Head Start, based in large part on observational measures of teacher performance in the classroom; the Pew-MacArthur Results First initiative, in which 19 states have agreed to review the evidence on which their social programs rest and, based on this assessment, make recommendations to their respective state legislatures about shifting funds to programs that work; the recent focus on Pay for Success programs (also called “Social Impact Bonds); and others.
To this impressive list we can now add the recent work by the White House’s Social and Behavioral Sciences Team (SBST). Founded just last year, and under the leadership of wunderkind Maya Shankar, the team has just published the results of 17 studies, all but one based on random-assignment designs, to influence people’s choices or improve government efficiency. Based in large part on the work of Richard Thaler of the University of Chicago and Cass Sunstein of Harvard, SBST is teaching government to “nudge” people into making better decisions.
The team is using the results of behavioral research which, roughly speaking, shows subtle ways that various messages can influence what people do. If people are reminded of obligations in timely fashion, they are more likely to fulfill those obligations. When filling out a form, if people sign at the beginning stating they will provide honest and accurate information in their answers, they are likely to give more accurate answers than if they sign at the end when they have already provided their answers. When employees specifying withholdings are defaulted to depositing withheld funds in a retirement savings account or taking an action to opt out, rather than having to take an action to opt in, the enrollment rate will increase and people will save more for retirement.
There are several reasons the SBST behavioral science initiative is such a stellar entry into the flooding tributaries of the evidence-based movement. First, the energy of the evidence-based movement must be rekindled by new initiatives that strengthen the claim that evidence-based policy can improve program impacts. Second, the findings from all but one of the administration’s studies are based on random assignment designs, the gold standard of program evaluation. It follows that their results are likely to be reliable and replicable. Third, all the studies have immediate application to government policy. We learn how to write letters that increase compliance, ways to increase contributions to retirement plans, how to increase the likelihood that government workers will increase use of two-sided copying, how to increase college entry, how to increase the call-in rate for programs that need information from account holders, and many other lessons that improve individual choices and promote government efficiency. Fourth, it seems reasonable to conclude from the entirety of the SBST’s results that if scaled up, their behavioral innovations would save the government billions of dollars. A more subtle and difficult to measure outcome of scaling up the SBST discoveries is the effects on health and on well-being in old age. It may be difficult to measure the benefits of millions of Americans saving more for retirement, but who doubts that a savings account is a key to peace of mind and greater choices in spending by retirees who have more in their savings account, not to mention the indirect benefits generated from less dependence of retirees on their children and other relatives.
The field of application of nudges based on behavioral science is limited only by the imagination of program designers and government officials. Equally important, the early success of applying behavioral principles to government policy illustrates yet again the soundness of basing policy choice on evidence of outcomes. Evidence-based policy is expanding its reach, this time by showing new ways to influence behavior, improve the efficiency of government programs, and save money. Evidence-based policy is on a roll.
Ron Haskins is a senior fellow in the Economic Studies program and co-director of the Center on Children and Families at the Brookings Institution and senior consultant at the Annie E. Casey Foundation in Baltimore. From February to December of 2002 he was the senior advisor to the president for welfare policy at the White House.
In “Evidence at the Crossroads,“ we seek to provoke discussion and debate about the state of evidence use in policy, specifically federal efforts to build and use evidence of What Works. We start with the premise that research evidence can improve public policies and programs, but fulfilling that potential will require honest assessments of current initiatives, coming to terms with outsized expectations, and learning ways to improve social interventions and public systems. Join the conversation by tweeting #EvidenceCrossroads.
Evidence at the Crossroads Pt. 3: Research-Practice Partnerships, the Future of the Evidence Movement
By Gordon L. Berlin and Rekha Balu
One of the most unheralded aspects of the recent federal investments in scaling evidence-based programs is the novel partnerships formed between researchers and program developers. Both the U.S. Department of Education’s Investing in Innovation (i3) grants and the Corporation for National and Community Service’s Social Innovation Fund (SIF), for example, supported expansions that have more than doubled the participants served in funded programs, while also funding evaluations involving new partnership between researchers and program developers. These alliances have provided new ways for researchers and program staff to work together and have supported programs in their efforts to continuously improve and refine their interventions.
In both the SIF and i3, MDRC worked in concert with program leaders to think about how to use evidence to improve or even re-design interventions. We sometimes learned as much from disappointing findings as we did from programs that “worked.”
BELL (Building Educated Leaders for Life), a recipient of a SIF grant, runs a successful elementary school program and wanted to test its application in a middle school setting. MDRC’s evaluation of their summer program for middle-schoolers found encouraging results for math but not reading. Both organizations then launched a joint diagnostic process to identify possible programmatic solutions for middle school. As Tiffany Gueye, the Executive Director of BELL noted in an Education Week piece, BELL leaders “plugged findings from the study into our continuous-assessment process” and decided to focus enhancement efforts on staff training, curriculum, and student assessment, as well as a greater focus on social and emotional learning. MDRC and BELL are continuing to work together to learn how these efforts are playing out in the programs and to design a learning agenda for the future.
The story was a little different for the Center for Employment Opportunities (CEO), another SIF grantee. An earlier MDRC study of CEO’s transitional employment program in New York City for recently released offenders re-entering the community found that, although the program did not yield long-term employment effects, it did reduce recidivism among program participants. Using what it learned from the evaluation, CEO restructured its employment programming to place more emphasis on post-program employment retention services as it expanded to other jurisdictions. MDRC is working with CEO in an initiative to pilot a cognitive behavioral education component with the hope that it will further improve employment outcomes, particularly for younger offenders.
This type of collaborative approach to building reliable evidence also can be applied to improving systems, not just specific programs or interventions. Outside of the SIF or i3 structure, MDRC is working with New Visions for Public Schools, which supports a network of 77 New York City public schools, to promote evidence-based practices while embedding ongoing evaluation and evidence-building into the system of support it offers schools. In a constantly changing operational environment, a support organization or system of schools finds as much benefit from using evidence to build systems for identifying and supporting students at risk as from introducing specific evidence-based interventions. This next generation of research–practice partnerships is employing a mix of advanced analytic techniques—including using predictive analytics, exploiting variation across schools, and conducting quick turnaround experiments—to inform decision making and improve school and student performance.
As the evidence movement matures, it is increasingly clear that we need to build on lessons not only from clear successes, but also from interventions that have not worked. Neither program developers nor researchers can tackle this task in isolation. Research–practice partnerships are a strategic way to ensure we learn from what does and does not work as we actively translate evidence into practice. These next generation alliances between social policy programs and evaluators also have the potential to tackle more complex challenges by capitalizing on the availability of the data housed in public systems and the opportunity to embed random assignment and other rigorous methods inside organizations’ continuous improvement cycles, heralding a new age in evidence-based practice.
Gordon L. Berlin is President of MDRC, a nonprofit, nonpartisan education and social policy research organization dedicated to learning what works to improve programs and policies that affect the poor. Rekha Balu is a Senior Research Associate focused on K-12 program evaluation.
In “Evidence at the Crossroads,“ we seek to provoke discussion and debate about the state of evidence use in policy, specifically federal efforts to build and use evidence of What Works. We start with the premise that research evidence can improve public policies and programs, but fulfilling that potential will require honest assessments of current initiatives, coming to terms with outsized expectations, and learning ways to improve social interventions and public systems. Join the conversation by tweeting #EvidenceCrossroads.
Research Grants Webinar: Applying for Research Grants on Reducing Inequality
In this video, Program Officer Vivian Louie leads a discussion and Q&A on strategies for applicants to develop strong letters of inquiry for research grants in our reducing inequality focus area.
The goal of this webinar is for potential applicants to gain a better understanding of our interests and develop strong letters of inquiry, specifically in the domains of immigration and mental health.
Also, read Vivian Louie's posts on for research grants on reducing inequality.
New Report: Insights on the Funding Landscape for Research on Inequality Among Young People in the United States
If we are to reduce the effects of today’s inequality on tomorrow’s society, then research on inequality among young people is essential.
But a new report from Sarah Bruch of the Univerity of Iowa finds that, despite increased attention to the topic of inequality, the funding landscape for research on youth inequality in the U.S. is still relatively small.
In Investing in Knowledge: Insights on the Funding Landscape for Research on Inequality Among Young People in the United States, Bruch maps the current funding environment for research on understanding and addressing social and economic inequality among young people in the U.S. The insights that make up the report emerged from interviews with social science researchers and staff from foundations and government agencies, and an examination of materials from 300 funding organizations that have either an explicitly expressed interest in inequality and young people or youth-serving systems, or a reputation for funding inequality research that affects young people as a population or in institutions.
Bruch highlights the complexity of inequality by highlighting its different distinctions and definitions, and outlines the dominant approaches of funding organizations who support research on youth inequality in the U.S. Finally, drawing on an understanding of the current landscape, Bruch offers three strategies for advancing efforts to understand and address youth inequality.
Download:
Investing in Knowledge: Insights on the Funding Landscape for Research on Inequality Among Young People in the United States
President’s Comment: To Reduce Inequality, We Need More Answers
Whether compared to other countries or to our own history, it is clear that inequality in the United States today is excessive and harmful to our economy and society. Yet the evidence is also clear that social policy can reduce inequality, and that the challenges facing young people and families are not inevitable.
Private foundations like ours can help reduce inequality in several ways. Of course, they can help by contributing to evidence-based programs and services that improve opportunities for the disadvantaged. But how do we know which programs and services go farthest in reducing inequality? And what combination of efforts is likely to provide the most leverage on the problem in specific contexts? In response to these concerns, another important role for private foundations is to support research that addresses key questions about reducing inequality. Indeed, the enormous scope of the challenge of inequality means that even the largest contributions to programs and services will make little headway unless they are applied in the most powerful ways possible.
But what is the current state of research funding for studies of inequality in the U.S.? Recently, we commissioned Sarah K. Bruch, a sociologist at the University of Iowa, to scan the funding landscape (public and private) for inequality research. Today we are releasing Professor Bruch’s report, Investing in Knowledge: Insights on the Funding Landscape for Inequality Among Young People in the United States, which draws on an examination of materials from 300 funding organizations, as well as interviews with researchers and with staff from foundations and government agencies.
The analysis reveals a scarcity of funding for research on inequality, which is especially troubling in light of the importance of the problem. Funding scarcity may be one source of the slow growth of research on inequality, particularly research focused on inequality among young people. Professor Bruch follows her scan of the landscape with three recommendations for strategies to advance efforts to understand and reduce youth inequality:
More resources specifically targeted to supporting research on inequality among young people. This is the critical lens if we are to reduce inequality in the next generation.
More diverse scholarship on inequality. A more inclusive range of researchers would broaden the perspectives on inequality and ways to reduce it.
More integrated approaches in response to inequality. There is no single solution to this huge challenge, and research that draws on a broad range of intellectual traditions and methodological tools and pulls them together in cohesive responses will increase the likelihood of success in the long run.
This analysis and accompanying recommendations gives us much to ponder and, we hope, provides a new stimulus for action on the part of funders and researchers alike.
Read Investing in Knowledge: Insights on the Funding Landscape for Research on Inequality Among Young People in the United States
