Standardized testing and the necessity of evaluation of merit
I picked up a thing or two about standardized testing while working for Pearson, one of the largest providers of these assessments. Pearson owns the GMAT, hundreds of occupational licensing exams, and many state standardized assessments. In the UK, they own Edexcel, which accredits GCSEs (vocational certifications). Pearson had a roster of the best and brightest of the world's psychometricians on staff, and I got to spend time with them as part of designing and rolling out our Pearson's efficacy strategy (i.e., how to evaluate learning outcomes). They are jovial and practical people, used to running robust statistical analysis to demonstrate the credibility of the tests.
Distilling from many conversations, I find that the problem rarely lies with the test itself but rather how the outcome is used and what it is interrupted to mean. Often those evaluating these scores have little appreciation of what you can and cannot deduce from them. For example, exams are generally either 'norm' or 'standard' referenced. The GMAT is a norm-referenced test. That means that the result tells you how well the test-taker performs relative to the pool of those who also took the test. Advanced Placement exams are 'standard-referenced', which tell you how a test-taker performs relative to the defined standard.
The difference can be confusing if you don't think about it actively.
The GMAT can't tell you, objectively, if one is proficient enough at arithmetic to conduct a Discounted Cash Flow analysis. That's not what it measures. It's also not the point of the exam. It measures how one performs, quantitatively, relative to the cohort who is taking the exam. It is meant to differentiate relatively. Not to determine absolute potential. But I get why people can be confused.
The overall point here is that data, measurement, and testing are all important to the process of learning, placement, and advancement.
Using these tools requires judgment. As my colleague and co-author Michael Fullan would say, "a fool with a tool is still a fool". We need to invest in the judgment and collective wisdom of those who are in a position to use the tools in our education system.
I recommend this take about what rolling back the ACT and SAT might mean for education in California. Checker is one of the most respected education policy thinkers in the U.S. At 75, his depth of insight and pace of writing are unparalleled:
What’s cockeyed here is that it’s not the SAT and ACT that are “flawed.” There’s no denying that scores tend to correlate with test-takers’ socioeconomic circumstances and often with their race, but that’s not because the tests are biased. It’s because in California, as elsewhere in the United States, the K-12 education system has shown itself incapable of producing remotely equitable academic outcomes among racially and socioeconomically diverse students.
Consider, for example, that on the National Assessment of Educational Progress—the “Nation’s Report Card”—among the Golden State’s eighth graders in 2019, an overall 30 percent were “proficient” (or better) in reading, somewhat worse than 33 percent for the nation as a whole. But when we look at group performance, we find that 45 percent of California’s white students and 57 percent of Asian students made it to that level, but just 10 percent of black pupils and 19 percent of Hispanics did so. Which is to say, fewer than one in five students in those two huge minority groups was truly literate when they entered high school. Is it any wonder that their SAT and ACT scores, a few years later, average far below those of their white and Asian classmates?
Most professors want academically prepared students in their classrooms, young people ready to succeed in bona fide college-level academics. That means “proficient” (or better) in all the core skills one should possess at the end of high school and a solid knowledge base in subjects they will study further. Yet the regents chose to shoot the messenger rather than push hard on the unwelcome message: that California’s schools are failing to educate their students of color well enough to place them on a level playing field when the time comes for college.
This is a complicated topic, with endless more depth. But for now, I leave you here.