(by Jacques Arsenault)

Jay Matthews writes in this week’s Washington Post about the predisposition of schools to place a strong emphasis, in the NCLB era, on the performance of “on-the-bubble” students. He refers to a March 13 Ed Week commentary by Cesar Chavez Public Charter School teacher Sarah Fine (subscription required) in which she describes the focus placed on students at the cusp of proficiency, often to the detriment of other students who are considered either “safe bets” to cross the bar, or those who are seen as unlikely to reach the passing standards.

Matthews writes:

I have been hearing for some time about this practice of devoting special attention to what are called the “on-the-bubble” kids. They are close to scoring proficient on the annual test, which affects the school’s rating under the No Child Left Behind law. Some schools give them extra teacher time, leaving less help for lower-performing students, such as Shawn, who have no chance of increasing the passing rate. I sometimes shrugged this off as just one more sign of poorly led schools. A good principal, I said, would put an end to such nonsense.

But Fine’s story surprised me, because she is working at one of the city’s best-led public schools.

This pattern, while unfortunate and unfair to most students, is an inevitable result of an accountability system that puts its emphasis on one performance mark. In the case of No Child Left Behind, the system is set up on a “percent-proficient basis, where schools are graded on the percentage of students in several groups that achieve proficiency on particular standardized tests.

Accountability systems in education and other fields are predictable, in that people will respond to the incentives given and the standards proposed. At worst, the incentives can have lead to actions such as the widespread cheating on Chicago Public Schools testing uncovered by Steven Levitt in 2005. Even when the rules are followed, teachers and schools will change their behavior to focus on the specific measures on which they will be graded.

In a percent-proficient structure, e.g., if a score of 70 is considered proficient, then a teacher (A) who has students scoring:

95, 85, 82, 78, 75, 71, 53, 46, 34, 25 (60% proficient)

will be considered more successful than another teacher (B) whose students score

100, 95, 89, 78, 69, 68, 68, 67, 66, 65 (50% proficent)

Even though most of the students in Teacher B’s classroom are performing at a higher standard than their individual counterparts, Teacher A has more students passing the 70 mark, and is thus rewarded for the work she put in with her “on-the-bubble” students to get them over the hump.

This type of system can result in schools devoting full class periods to “bubble sheet skills” or teachers spending an inordinate amount of time with “bubble students” while ignoring the needs of students on the ends of the spectrum.

—

Several approaches have been proposed or instituted by schools or systems to address the shortcomings of the “percent-proficient” based testing system, and they generally fall into three categories:

1. Value-add: shifts the basis of measurement from “standard” to “progress.” As Mathews writes,

The best solution, just about everyone agrees, would be to accelerate the expected change of No Child Left Behind to a value-added assessment. Once states and the District improve their computer systems, they can rate each school by how much each child improves, rather than the current method of recognizing schools that reach a designated percentage of passing scores.

Value add measures introduce some questions themselves, including whether it is enough that students are making progress in a grade if they come in one or more grade levels below, (e.g., if a 6th grader begins the year reading at a 4th grade level and finishes it reading at a 5th grade level, how should we evaluate that progress?). Are we closing the achievement gap by doing this, or is it enough that we are not allowing the gap to increase further?
On a logistical basis, are there ways to either combine value-add and proficiency scores, or to report them separately but to treat each as valuable in its own right?

2. Multiple score standards: rather than simply having a cut-off at proficient, this idea would measure the percentage of students performing at proficient, as well as, for example, those 10+ percentile points above proficient, and 10 percentile points below. In this way, teachers and schools would be incentivized not just to focus on those students on the cusp, but also to bring the bottom-performing students closer to the proficient mark, and to move the accelerated students beyond just clearing the proficiency bar.

3. Alternative methods of assessing student performance and knowledge: While the first two options listed here focus on how test data is collected and analyzed, this third option seeks to change the structure of the assessments themselves. In many cases, this is a call for adding to, or replacing, the standardized test with more qualitative or “performance-based” measures. Mathews calls for some version of this in his column, saying:

There should also be a way to honor Fine’s request for an extra dimension, such as reporting a rise in students doing scientific experiments or writing analytical papers.

But there are already examples of measures such as these being used, in the International Baccalaureate curriculum (as Mathews notes), and in new, performance-based systems such as Rhode Island’s Portfolio graduation option, which several school districts are beginning to role out, through a program designed by Brown University and the Rhode Island Department of Education, with funding from the Gates Foundation.

—

Each of these alternatives to the “percent proficient” measurement system requires more effort or resources being put into the assessment process, but we must remember that teachers and schools will respond to the accountability measures that they are given. Just as we must continue to improve the quality of content on assessments given to students, there is an equal imperative to continue to improve the way we collect and assess the outcomes that we desire for our children.

The aims and requirements of providing high-quality education to all students are not as simple as a single arbitrary line-score on a single exam, and so our testing regimes probably should not be either.

(Cross-posted at Jacques of All Trades)