Teacher Teams and School Processes in Scaling-Up a Content Literacy Innovation in High Schools - Final Report: The Evaluation of the Scale-Up of Reading Apprenticeship Through the Reading Apprenticeship Improving Secondary Education (Raise) Project

Abstract

We report on the scaling up of a high school content literacy intervention, Reading Apprenticeship, over a period of four years as part of the independent evaluation of an Investing in Innovation grant from the U.S. Department of Education to WestEd’s Strategic Literacy Institute (SLI). The goal of our scale-up study was to understand the school processes that support successful implementation and promote scaling of an innovation. We focused on teachers, principals and other school-level program leadership who were also the focus for SLI’s innovation designed to support scale. The logic model that our work was guided by built on research literature that provided insights into scale-up as increasing local ownership and depth of commitment, as well as growth in numbers. We provided formative feedback to SLI while contributing to the empirical methods and evidence for studying scale-up processes.

Study Design: The scale-up study was conducted in parallel to a RCT in which 22 treatment and 20 control schools from Pennsylvania and California participated for three years. In the scale-up portion of the evaluation, a total of 239 schools in four cohorts adopted the program in Pennsylvania, Michigan, Indiana, and Utah. Our study focused on the scale-up teachers and schools but also compared these to teachers and schools in the RCT treatment group. Surveys of teachers and principals, as well as records of participation in project activities provided a rich longitudinal dataset. In addition, four schools served as case studies where school leaders and program support staff were interviewed over two years to offer a concrete picture of decision processes and resources. We used descriptive statistics from surveys of the larger sample to track changes over time in features of program implementation, as well as resulting changes in attitudes and commitment among teachers. Regression methods were used to identify predictors of increased or decreased participation. We used these predictors in comparing the scale-up and RCT as contexts for the implementation and impact of Reading Apprenticeship.

Findings: Over the five years of the project, the scale-up of RAISE reached approximately 533,000 students, and 1,719 teachers in 239 schools in four states. During the first year of RAISE implementation, teachers and administrators in the Scale-up schools reported high levels of buy-in and commitment to the initiative. Teachers were successfully integrating Reading Apprenticeship practices into their instruction and cited collaboration and support from other teachers as the most effective means at building their capacity to implement what they learned during the RAISE professional development. Teachers and administrators identified competing initiatives as a primary challenge to implementing and sustaining RAISE long term and, as we found through our case study work, schools faced unique and localized challenges as they navigated their way through the scale-up. In the second and third year of implementation we found that trends in the uptake of RAISE activities and enthusiasm for the initiative decreased or leveled off by the third year. We also found a greater spread in responses across schools by the third year, calling for further investigation into school-level processes and the characteristics of schools that may be associated with the gain or loss of RAISE participants over time.

Through this investigation, we found that teacher participation in team meetings during the first year and school-wide commitment (rather than individual commitment and actual classroom usage by the teachers) predicted the increase in number of teachers participating in a school. Such increases were less likely where the school did not see regular participation in team meetings in the first year and there was less commitment among teachers to school-wide success. Increases were unrelated to demographics and resources available to the school and unrelated to many characteristics and actions of administrators. We hypothesized that a process, consistent with SLI’s innovation for scaling Reading Apprenticeship, which involved a cross-disciplinary teacher team within the school, led to increased and sustained program participation. As a final step, we compared the characteristics of Scale-up and RCT treatment schools. Scale-up schools had more of the characteristics associated with growth suggesting greater long term sustainability than in RCT schools. We raise the issue of whether the implementation under the constraints of the RCT is a good model for measuring impact in the context of scaling up an innovation.

Keywords: Reading Apprenticeship, reading, literacy, Empirical Education, Strategic Literacy Institute, scale-up study, RAISE