Many variables taken into account in value-added systems

On Board Online • June 10, 2013

By Cathy Woodruff
Senior writer

The list of factors that can affect how well a student performs on a single state standardized test is long, and only one item on that list is how well he or she has mastered the material being tested.

Virtually all of the other variables are beyond the control of his teacher or his school principal, who both are to be rated, in part, on that student’s performance under New York’s fledgling Annual Professional Performance Review (APPR) program.

Some variables are impossible to predict in any reliable, systematic way. Perhaps pre-test jitters kept this student awake the night before the test. Maybe a sick parent, or even a sick pet, is weighing on his mind.

But testing experts say other influences that can harm a student’s performance are predictable and quantifiable. Those influences, they say, can be recognized through a formula or “algorithm.”

The practice of compensating for those recognized variables often is referred to, generically, as “value-added.” In New York, a handful of factors – including poverty, prior test scores, disability and limited English proficiency – already are applied to the test scores through a system the State Education Department calls a “growth” model.

Now, New York’s Education Department staff and consultants have blended all of those growth factors, and more, into a richer New York-specific recipe that they are giving the value-added label.

The enhancement of the formula makes it more reliable and more effective, Education Commissioner John B. King Jr. told the Board of Regents in April. He said expansion of factors related to students’ previous performance will be among the most significant changes when New York shifts to its value-added approach.

“Once you control for prior academic history, you’ve controlled for a lot of factors,” King said. “You can never get to an algorithm that controls for every possible variable … but you can get closer.”

New York is unique in making a distinction between a growth model and a value-added model, according to Amy McIntosh, a senior fellow for teacher and leader effectiveness with the Regents Research Fund. She is part of a team  that has been putting together the new value-added model that the Regents are expected to consider at their June meeting.

The addition of more factors will make it even less likely that a teacher will see her APPR rating – Highly Effective, Effective, Developing or Ineffective – affected by factors unrelated to the quality of her work, McIntosh said.

“It is more likely that it will be just about the instruction in the classroom, rather than the characteristics of the students, which the teacher can’t control,” McIntosh told On Board.

Once New York adopts a value-added system, the weight of student improvement on tests will go up to 25 percent with APPR evaluations. Student improvement on state tests counts for just 20 percent under the growth model, which accounts for fewer factors.

The value-added approach also can lead to fairer ratings for teachers whose students already are high achievers and whose already strong test scores might have little room for improvement, McIntosh said.

But no matter what model is used, McIntosh said a critical step toward protecting the quality of the information that goes into the ratings is taking place right now. Friday, June 14, is the deadline for school districts to submit electronic data connecting individual students to individual teachers for the purposes of APPR, she said.

Broadly, the goal of this process is to ensure that any growth or value-added information used in a teacher’s evaluation truly was based on tests taken by that teacher’s students, rather than those in a different teacher’s class.

“Teacher-student data linkage is a critically important process for the credibility of these measures with teachers,” McIntosh said. “Having teachers verify that data will go a long way to helping them understand and accept the results because they will know these are their kids.”