Using Multiple Evaluation Measures to Improve Teacher Effectiveness

Consensus is elusive when it comes to figuring out exactly what it takes to improve our nation’s public schools. When the quest is to ensure that our children achieve academically, there just aren’t many certainties. Except one: The quality of teaching matters.

Research shows that an effective teacher is key to student success. But determining what evidence best reflects teacher effectiveness and how this information can be used to improve the quality of teaching are among the significant issues facing public education today.

The impetus for meaningful teacher evaluation reform from many sectors set the stage for the major changes we are now witnessing in the direction and scope of teacher performance evaluation. Some of the factors leading to this reform include:

• The 2009 seminal report, “The Widget Effect,” exposed the reigning indifference to instructional effectiveness in our schools and in our policies—an indifference that ignores variations in the effectiveness of our teachers, treating them as if they were all the same, and that does little to address the problem.
• Advocates are decrying the lack of state guidance and requirements for teacher evaluations. For too many school and district leaders, formal evaluation is a compliance activity instead of an opportunity to provide meaningful feedback to teachers for improvement.
• Academics pronounce that the state of teacher performance evaluation is a nonsystem in need of major reform.
• Many sectors—governors and mayors of different political parties, state legislatures, businesses, and educators and their unions—are calling for meaningful reforms in the way we evaluate and support our teachers.

Dynamic reforms effecting teacher evaluation and support are now happening in states and school districts. These reforms are inspired in part by the U.S. Department of Education’s competitive grant programs, including Race to the Top, which require new standards and assessments in our public schools, data systems capable of measuring student growth, and human capital systems designed to recruit, develop, and retain effective teachers. This effort is matched by recent priorities of the Teacher Incentive Fund supporting district-wide evaluation systems that reward teacher success. The Education Department’s decision to provide waivers from key provisions of or flexibility within the Elementary and Secondary Education Act—also known as No Child Left Behind—offers a further boost and a framework for states to make these long overdue reforms in a coherent way.

On February 28, 2012, 26 states and the District of Columbia submitted requests to the Department of Education for waivers. Twenty-three states were ultimately approved; two states (Idaho and Illinois) have pending applications; one state (Vermont) withdrew; and one state (Iowa) was rejected. (Note: Idaho’s application was approved on October 17, 2012, while this paper was drafted and is therefore not a part of this analysis.) Eleven other states received waiver approvals in an earlier round. As part of the second round of requests, all states presented plans to raise standards, improve accountability, and support reforms to improve principal and teacher effectiveness. These plans provide an important view into the decisions and actions of states as they design, build on, or perfect the systems for these new reforms.

Many states are now actively building or implementing educator workforce systems with meaningful evaluation and support systems that are linked to improvements in classroom practices and student achievement. No longer is teacher evaluation expected to be merely perfunctory or used exclusively as the basis of personnel decisions. State leaders are rethinking the underlying assumptions and policies of teacher evaluation systems and, together with critical stakeholders, are planning the implementation of new systems.

The focus of this report is on one piece of this very large set of transformations: the multiple measures and multiple methods used in new teacher evaluation systems, including the weighting of these measures, to determine a composite score of teacher effectiveness. The data source for our analysis is the plans of 23 second-round waiver applicants approved by the U.S. Department of Education as of August 2012. These include the plans received and approved for Arizona, Arkansas, Connecticut, Delaware, the District of Columbia, Kansas, Louisiana, Maryland, Michigan, Mississippi, Missouri, Nevada, New York, North Carolina, Ohio, Oregon, Rhode Island, South Carolina, South Dakota, Utah, Virginia, Washington, and Wisconsin.

Our review of these various reform plans indicates that the design and implementation of new systems of evaluation and support are truly works in progress. It’s clear that this work will be an iterative process and that it should be open to review

and adjustment as new research and the results of pilot implementations surface. For now, the state efforts and the waiver process both represent a rich laboratory of exploration and reform that bears watching for lessons to be learned, as well as for necessary corrections to be made. A few findings have already emerged from this initial review. They include the following:

• This is hard work that is being approached differently by states while they implement multiple reforms.
• It is difficult to legislate, regulate, and provide guidance for change within an environment of multiple simultaneous reforms. These reforms include the implementation of new college and career-ready standards, statewide data systems, new assessments, and new state responsibilities for these new systems, to name a few. The new educator evaluation systems must align with and be a part of these other reforms.
• Each state approach, including that of the District of Columbia, is different, and each is at a different stage of development and implementation. Evaluation designs are influenced by factors such as the characteristics of local school districts, laws governing charter school autonomy, and a state’s history for local control and collective bargaining agreements related to educator evaluation.
• Measures used to assess educator effectiveness are diverse and cannot be captured by only one or two indicators.
• Waiver winners rely on a range of measures and methods for assessing teacher professional practice, including classroom observations, self-assessments and reflection, teaching artifacts, student-learning measures, and surveys of students and parents.
• States are using both student-achievement measures (measures of student learning at a specific point in time) and growth measures (changes in student learning over time), including value-added estimates based on state assessments when available, to capture measures of student success aligned with individual teachers or teams of teachers. Some states are still considering the types of student-growth measures to use, and some are piloting multiple models before recommending a particular approach.
• States are also looking to more personalized and school-appropriate measures for determining teacher impact on student learning and vesting teachers more directly in monitoring student progress through approaches such as student-achievement goal setting, student-learning objectives, student-learning targets, teacher goal setting, and unit work samples. These measures are used to actively engage the teacher and the evaluator in a goal-setting process for student learning that is customized for the teaching assignment and for the students.
• States give different weights to component measures devoted to indicators of student achievement and indicators of professional practice; they also rely on different measures. Some states have specific percentages of components spelled out in state law. Others do not. In some cases a certain amount of discretion is given to local districts for insertion of components they value in the evaluation.
• States are expanding the measures used to determine teacher effectiveness for nontested grades and subjects.
• Though some states are in the beginning stages, all are determining or developing assessments applicable to teachers of grades and subjects that are not part of statewide, standardized assessments for the purpose of determining student growth.
• Typically this involves expanding the portfolio of state assessments to provide growth data in all grades and subjects or expanding the portfolio of nationally or locally approved assessment tools that can be validly used such as classroom-based assessments, unit tests, end-of-course assessments, student-learning objectives, and portfolios.
• Systems have diverse purposes.
• Waiver applicants were responsive to the application requirements making these systems as much about differentiating educators on their levels of effectiveness and for use in making personnel decisions as about using the evaluation process to identify areas for overall educator improvement.
• Successful systems need an infrastructure of support.
• The work of the states is not just about creating new systems of teacher evaluation, but also about putting an infrastructure in place to ensure the success of these systems. This means that teachers and principals must receive orientation to the new systems; evaluators must receive appropriate training (for example, in collecting evidence, rating against a professional standard, and providing feedback); rubrics and protocols for observation must be identified and tested; strong teacher-student data links must be in place that verify that the teacher of record is tied to the right students for purposes of assessing teacher impact; and management systems must be devised that allow teachers to track their progress toward learning goals. Just as importantly, supports and interventions must be in place to move teachers toward higher levels of effectiveness in line with the information provided through evaluation.

Against this evolving backdrop we offer the following policy recommendations:

• The U.S. Department of Education should closely monitor the successes and problems experienced by these states and the District of Columbia as they implement these new systems of evaluation and support them going forward.
• The states and the District of Columbia should continue to heed emerging findings from research and evaluation and seek feedback from their own efforts to ensure continuous improvements.
• The U.S. Department of Education and philanthropic organizations should continue to support improvements in the tools and infrastructure necessary for the development and sustainability of these new evaluation systems.
• Lessons learned from these efforts must inform the future direction of education reform through the reauthorization of the Elementary and Secondary Education Act.