Implementation Considerations

Effective implementation of a well-designed strategic evaluation system drives improvement in teacher practice and student learning. This section highlights important details for implementing:

Core Components Selection, Calculation and Weight

When implementing the core components of a strategic evaluation system, there are a variety of considerations related to each component. These considerations require a district to reflect on the context of their schools, educators and community along with the district structure and existing evaluation system.

When determining the weight of each component that will be used to evaluate educators, districts should consider the following questions:

  1. What will be the overall size of the student achievement component?
  2. What will be the overall size of the student survey component? (if applicable)
  3. What will be the overall size of the observation component?
  4. Will a school-wide measure be included for every teacher?
  5. How will the inclusion and weights of the components vary for different teachers, i.e., tested vs non-tested subjects?
  6. Is there a minimum number of days for a teacher to receive an evaluation?

LESSONS LEARNED

Research indicates that student achievement should be at minimum 30%, with a target of 35% of an educator's total evaluation score.

Districts that include student survey or student voice in the overall evaluation typically weight this component from 5 to 15% of the overall evaluation. Research suggests that evaluation systems with student surveys are strong and more reliable than those that neglect incorporating student voice. (The Widget Effect; Making a Difference: Six Places Where Teacher Evaluation Systems are Getting Results). The following breakdown is recommended:

  • State Assessment: 10-15%
  • District Assessment: 15%
  • Other (Portfolio of student work, Student Learning Objectives (SLO), etc.): 5%

When determining what student growth measure will be used to evaluate student achievement, districts should consider the following questions:

  1. What tests/data will be included in the student achievement component and will this require a change to protocol to include different assessments?
  2. How can the student growth measure align with the district’s philosophy on assessment?
  3. What student achievement and/or growth data does the district already collect?
  4. What is the capacity of the district to collect, analyze and track student growth measures?
  5. Is there a minimum start date or minimum attendance rate required for a student to be included in the calculations for a teacher?

LESSONS LEARNED

The specific measure used and weight of each measure typically varies by the category a teacher falls within.

For example, a teacher in a non-tested subject area or grade may have a certain measure weighted less or more heavily. Additionally, a teacher that teaches a specific sub-set of students may also have differing weights and/or measures.

When determining the student perception survey and implementation process, districts should consider the following questions:

  1. Will all grades receive the survey or will it be limited to grades 3 and higher?
  2. Will students complete a survey for all teachers or will they submit feedback for a limited number?
  3. Will every class be surveyed, for example, would a student complete a survey for their advisory period or a study hall?
  4. Will surveys be administered to students in special education units?
  5. What languages will you offer the survey in?
  6. Will the survey be conducted online or paper? How many times a year will you conduct the survey? Will you average the scores?
  7. Will there be a minimum length of time the student will need to have been assigned to the teacher’s class?
  8. Will results be compared differentiated by school type (elementary, middle, high school) and/or by subject (core subject, elective)?
  9. Will questions on the survey be available in advance?

LESSONS LEARNED

Districts should put systems in place so a student does not complete a survey for every teacher.

For example, a district could have students only complete two student perception surveys a year and use a randomly generated list to select the students that will complete a survey for a teacher.

When determining the calculation approach to teacher's ratings, districts should consider the following questions:

  1. Will the calculation of the overall scorecard calculation include a targeted distribution or have fixed cut points?
  2. Will the calculation of individual components follow a targeted distribution or have set fixed cut points?

Target Distribution vs. Fixed Cut Points

Target Distribution: Target distribution sets a fixed percentage of teachers for each rating or point level and assigns cut scores for assessments based on this distribution. The use of a target distribution can help promote equity across grades and content areas because the cut score can be set independently for each assessment, mitigating for any differences in rigor of assessments. Additionally, once set, the distribution does not have to be adjusted if the test is changed. While the target distribution makes the evaluation process fair and sustainable, it limits how many teachers can be rated at the top, which can feel unfair to teachers and outside stakeholders.

A target distribution can also promote budget sustainability, if strategic compensation is included in the system design. A target distribution allows for budgeting models to predict the number of teachers who could receive salary increases, based on potential increase in performance rating.

Fixed Cut Points: Fixed cut points use pre-determined cut scores for each assessment and each overall performance level. While this method is more easily understood by teachers, providing a goal or a target for their individual performance, it does not allow for a district to adjust those targets if the outcomes of the student achievement component is different than expected. For example, if standards change for a STAAR exam, it may result in a harder exam and fewer than expected teachers reaching the higher performance levels.

Fixed cut points at the performance level can present a challenge for budget forecasting if strategic compensation is included in the system design strategic compensation. With fixed cut points, it is mathematically possible for all teachers to achieve the highest performance and therefore highest compensation level. If fixed cut points are used, other implementation parameters may be used to help ensure fiscal sustainability, such as a salary cap increase or a limit on the number of performance levels a teacher may advance each year.

Calibration of Observers

Calibration is one of the most important and challenging elements of implementation.

When determining the administrator calibration methods, districts should consider the following questions:

  1. What is the district’s calibration/observation best practice recommendations?
  2. How aligned are the outcomes of the observation component to the District’s definition of Excellence?
  3. How aligned are the outcomes of the observation component to the student growth components?
  4. How can inter-rater reliability be included as a part of the evaluator’s own observation?

LESSONS LEARNED

Due to the large part observations play in many evaluation systems, and the tendency for scores to be inflated over time, a district needs to provide initial training to ensure calibration as well as take ongoing action to maintain consistent scoring.

One example of ongoing training are calibration exercises (e.g., watching and rating videos together or observing teachers and debriefing) with principals multiple times during the year. In addition to training principals, principal supervisors should be provided calibration training at the beginning of the year and throughout the year. Finally, campus calibration walks throughout the year with people from outside the campus can help ensure principal calibration. Research shows that outside observers are least likely to bring bias to observation ratings. Districts should monitor and audit observation data regular to check for observers that may not be normed and investigate for biases or inflation.

Differentiated Compensation

Developing a differentiated compensation system based on the evaluation results requires careful consideration and preparation so that employees understand and embrace the transition to a new pay structure.

Increasing Base Salary or Providing a Stipend: A district may approach differentiated compensation by offering a change in base salary structure or as a stipend paid in addition to a traditional pay structure. A change to base salary signals a long-term commitment by a district, but may require implementation rules such as a targeted distribution to ensure it is fiscally able to sustain the proposed changes. A stipend allows a district to adjust the amounts each year based on the district’s available budget. This approach may increase the feeling among teachers that this is a temporary program, similar to other "pay for performance" stipends in the past.

Setting a Salary Floor: A district can set a salary floor so that an employee who was with district prior to the change in compensation will not see their salary decrease. This decision will impact how teachers who have prior salaries based on years of experience and/or degrees are hired and paid as they transition into a new district’s pay structure. A district may also consider providing a salary floor based on their starting salary with the district or may choose to not offer a salary floor, meaning the teacher’s salary will adjust based on the first year of performance in the district.

Setting a Salary Increase Cap: As the district transitions from a traditional pay scale, a cap on increase in salary each year can help ensure financial stability and predictability. Some districts have used a $5,000 cap on an increase each year for the first two years of a new compensation system.

Compensation Level Movement: Implementing rules that control for fluctuation in effectiveness level will provide stability to a teacher’s annual salary. With the addition of more complex student achievement measures, the final evaluation scorecards may not be available until late summer/early September, meaning teachers will need to be able to anticipate the salary for the coming year based on implementation parameters. Here are some examples of parameters:

  • Limits on compensation levels for early years teachers: Early years teachers may advance one level per year, and must begin at the first level.
  • Limits on number of compensation levels advance in one year: Teachers may advance one level per year. Teachers who are hired with three or more years of teaching will not have a level assigned until they have completed their first year in the district.
  • Limits on how many years of lower performance to have compensation level lowered: Teachers with multiple years of lower performance, will be lowered one level.
  • Averaging of two years performance to produce compensation level: When multiple years of evaluation data is available, an average of the last two years will be used to determine compensation level.

Change Management and Communication

One of the biggest mistakes leaders can make is assuming that teachers, school leaders and parents will quickly get on board with changes to evaluation systems. Change can trigger anxiety and the growth measures in particular can be very hard to understand. It is important to remember that teachers and principals will be the ones implementing the evaluation system every day and so district leaders need to engage them continuously to build buy-in, understanding and get feedback on what’s working and what could be improved. Districts that have led change successfully have done so after significant stakeholder engagement and can honestly say that stakeholder input drove the changes.

Change management and communication is critical for districts implementing a value-added measure (VAM). Educators may already have some assumptions, or misconceptions, around VAM so it is important that district’s provide clear communication and information to leaders and educators.

LESSONS LEARNED

The framing and messaging around changing the evaluation system and the evaluation system in general should never focus on judging, sorting or getting rid of the worst teachers.

The highest and best purpose of an evaluation system is supporting teacher growth and ensuring that students are taught by high quality teachers. It is important for district leadership to give a detailed presentation highlighting the “WHY” for the strategic evaluation initiative to its senior leaders, and how they envision implementing the changes to the evaluation system within the district; allowing plenty of time for questions and mutual understanding. Doing this on the front end will avoid conflicts down the road when implementation of program components begins.

Financial Information

There are a number of ways to "find" existing dollars in a district’s general operating budget to fund a strategic evaluation system. For many districts that may have used a flat increase for all employees, it may consider using those dollars to fund the new strategic compensation approach. Some additional examples used by districts include:

  1. Prioritizing the funding of strategic initiatives through cuts across the board by department (i.e., have each department prioritize their spending, and then as a district leadership team, determine what spending is not necessary or not a priority).
  2. Analyzing all existing professional development contracts, particularly those under the umbrella "approved vendor" list to determine which vendors have programs that can show positive data for their performance and which cannot. This exercise has uncovered many expenditures that had zero to no known impact on students/campuses.

LESSONS LEARNED

A district should not rely on philanthropy or grants to fund increases to teacher salaries.

It is important to make a concrete commitment to the educators that their salary is funded out of the general fund, and can continue to be funded by the district.

TEA- Teacher Incentive Allotment

The Texas legislature passed a sweeping bill for school finance reform in the spring of 2019, including unlocking funds districts for a Teacher Incentive Allotment. For additional information on TIA and other components of House Bill 3, please visit the TEA webpage.

Organizational Capacity

As with any district initiative, the capacity of district departments to execute this work is critical for success. The performance evaluation team is frequently housed within Human Resources, supporting current evaluation system(s). As changes to the system are considered, the team supporting may need to grow and will need to expand beyond the Human Resources department to include the following roles:

  1. Evaluation System Director: This person will serve as the champion for a strong multi-measure evaluation system, ensuring close-coordination of district departments involved in the work. They will oversee execution of evaluation cycle and lead continuous improvement cycle for program.
  2. Evaluation & Assessment Data Director: This person oversees the calculation of student achievement and growth measures, the student survey administration and the production of overall evaluation scores. A district may choose to out-source this role and responsibilities, dependent on their capacity, to places such as ERG or SASS.
  3. LESSONS LEARNED

    Prior to embarking on strengthening or creating a new evaluation system a district should assess its current capacity to collect, analyze and track the data necessary to determine accurate and understandable evaluations.

    Many districts use partners to support them. For example, Lubbock ISD partners with SAS EVAAS and Batelle for Kids to aid in their value-added model and stipend awards.
  4. Compensation/Budget Director: This person creates budget models for compensation scenarios. This is an existing role within districts.
  5. Communications: This person assist with stakeholder engagement and creates resources and forums to share information; this is a critical component in successful implementation. This is typically an existing department within a district. It is recommended to include this department to assist with the efforts.
  6. School Leadership: These individuals serve as a champion for the evaluation system. They help communicate the "WHY" to Principal and Teacher stakeholders and oversee implementation of the observation rubric, specifically the calibration of observers. Additionally, they participate in the continuous improvement process.
  7. LESSONS LEARNED

    Engaging with school leadership early on in the process is important to ensure their input is included in the design and that they understand and support the evaluation system.

    Many districts gather principals early in the process. The principals are than able to be advocates for change.
  8. Legal/Employee Relations: These individuals participate in the modifications or the design of a new system to ensure compliance with local and state regulations. They provide guidance on implementation and individual components and update district policies related to staffing decisions (if applicable). This is a decision point for a district to decide how performance evaluations will be used in hiring decision. This can be an opportunity to change policies from using seniority as a determining factor in compensation to how include performance on evaluation system when considering staff reductions and in leveling decisions (when expected enrollment is not aligned to actual enrollment).
  9. Employee Relations: Any change to a performance evaluation system should include close coordination with both the employee relations department and the district’s legal department to ensure compliance with district policy and state education code. Changes could also result in an initial increase in grievances or lawsuits. Including employee relations and legal in the design and implementation can mitigate the exposure of the district.
  10. Implementation Leadership Team

    In addition to broad engagement with most departments in the district, a subset of leaders, the implementation leadership team, will be critical to identify and to charge with ensuring a coherent and smooth rollout of changes. The implementation leadership team should include the following members:

    • Project Lead – a person with the authority (either by existing role or Superintendent designee) to make decisions to ensure the process moves thoughtfully and at the appropriate speed.
    • Superintendent of Schools – It is imperative that the Superintendent is the public champion of the initiative to change the evaluation system, repeatedly reinforcing why multi-measure evaluation and strategic compensation is a priority for the district.
    • Superintendent Cabinet (Chiefs of every department) – The evaluation system has implications for every aspect of teaching and learning and therefore, each department chief, particularly those in charge of human capital, school leadership, assessment, communications and operations/finance will play a critical role in rolling it out and in reiterating the reasons for implementing changes. Aligned efforts and a united front will trickle down to all staff/teachers and pave the way for smoother implementation.
    • Campus Experts/ Representatives – Because evaluation systems are implemented by principals and teachers, it is valuable to have a representative or champion at each campus who understands and believes in the evaluation system and can support his or her colleagues to implement it with fidelity and use the system to improve feedback and professional development. These representatives can also provide valuable feedback to district leaders about how the system is or is not working well and what tweaks might improve the experience at the campus-level. This can be a multi-year, stipend position.

    LESSONS LEARNED

    Bringing in data and research team members early on is critical for making sure the data generated by the evaluation system can be translated into accessible information and acted upon at the school and district level.

    The evaluation data should be analyzed every year to surface systemic weaknesses or biases in the system. In Dallas, the evaluation team includes the Office of Institutional Research who provide the score cards for teachers and leaders to understand the data.