Monday, August 17, 2015

Why I Am No Longer a Measurement Specialist

I was introduced to psychometrics in 1959. I thought it was really neat.

By 1960, I was programming a computer on a psychometrics research project funded by the Office of Naval Research. In 1962, I entered graduate school to study educational measurement under the top scholars in the field.

My mentors – both those I spoke with daily and those whose works I read – had served in WWII. Many did research on human factors — measuring aptitudes and talents and matching them to jobs. Assessments showed who were the best candidates to be pilots or navigators or marksmen. We were told that psychometrics had won the war; and of course, we believed it.

The next wars that psychometrics promised it could win were the wars on poverty and ignorance. The man who led the Army Air Corps effort in psychometrics started a private research center. (It exists today, and is a beneficiary of the millions of dollars spent on Common Core testing.) My dissertation won the 1966 prize in Psychometrics awarded by that man’s organization. And I was hired to fill the slot recently vacated by the world’s leading psychometrician at the University of Illinois. Psychometrics was flying high, and so was I.

Psychologists of the 1960s & 1970s were saying that just measuring talent wasn’t enough. Talents had to be matched with the demands of tasks to optimize performance. Measure a learning style, say, and match it to the way a child is taught. If Jimmy is a visual learner, then teach Jimmy in a visual way. Psychometrics promised to help build a better world. But twenty years later, the promises were still unfulfilled. Both talent and tasks were too complex to yield to this simple plan. Instead, psychometricians grew enthralled with mathematical niceties. Testing in schools became a ritual without any real purpose other than picking a few children for special attention.

Around 1980, I served for a time on the committee that made most of the important decisions about the National Assessment of Educational Progress. The project was under increasing pressure to “grade” the NAEP results: Pass/Fail; A/B/C/D/F; Advanced/Proficient/Basic. Our committee held firm: such grading was purely arbitrary, and worse, would only be used politically. The contract was eventually taken from our organization and given to another that promised it could give the nation a grade, free of politics. It couldn’t.

Measurement has changed along with the nation. In the last three decades, the public has largely withdrawn its commitment to public education. The reasons are multiple: those who pay for public schools have less money, and those served by the public schools look less and less like those paying taxes.

The degrading of public education has involved impugning its effectiveness, cutting its budget, and busting its unions. Educational measurement has been the perfect tool for accomplishing all three: cheap and scientific looking.

International tests have purported to prove that America’s schools are inefficient or run by lazy incompetents. Paper-and-pencil tests seemingly show that kids in private schools – funded by parents – are smarter than kids in public schools. We’ll get to the top, so the story goes, if we test a teacher’s students in September and June and fire that teacher if the gains aren’t great enough.

There has been resistance, of course. Teachers and many parents understand that children’s development is far too complex to capture with an hour or two taking a standardized test. So resistance has been met with legislated mandates. The test company lobbyists convince politicians that grading teachers and schools is as easy as grading cuts of meat. A huge publishing company from the UK has spent $8 million in the past decade lobbying Congress. Politicians believe that testing must be the cornerstone of any education policy.

The results of this cronyism between corporations and politicians have been chaotic. Parents see the stress placed on their children and report them sick on test day. Educators, under pressure they see as illegitimate, break the rules imposed on them by governments. Many teachers put their best judgment and best lessons aside and drill children on how to score high on multiple-choice tests. And too many of the best teachers exit the profession.

When measurement became the instrument of accountability, testing companies prospered and schools suffered. I have watched this happen for several years now. I have slowly withdrawn my intellectual commitment to the field of measurement. Recently I asked my dean to switch my affiliation from the measurement program to the policy program. I am no longer comfortable being associated with the discipline of educational measurement.

Gene V Glass
Arizona State University
National Education Policy Center
University of Colorado Boulder

The opinions expressed here are those of the authors and do not represent the official position of the National Education Policy Center, Arizona State University, nor the University of Colorado Boulder.


  1. " ... those served by the public schools look less and less like those paying taxes." This is a ridiculous assertion. That's like saying those served by public highways and clean water systems look less and less like those paying taxes. Everyone (except perhaps the most extreme plutocrats, who don't pay much in taxes, anyway) benefits from a strong public education system.

    1. The writer is fully aware that everyone benefits. He is saying that the wealthy will always send their children to private schools and do not care to continue supporting public schools. It's part of the much larger reform movement to kill public schools. This writer is politely saying why he can no longer be a part of using psychometric measures to finish the job. I hope he writes more.

    2. I think you missed the point of the commentary

  2. Thank you, Dr. Glass. It takes an extraordinary person to make such a dramatic change. I assume you have read S.J. Gould's brilliant book, The Mismeasurement of Man. In fact, I'd bet a penny or two that it might have influenced your decision a tiny bit.

    Based on your biographical musings, you are just a few years older than I, maybe two or three. As we age, the strong connections among our neurons become less encumbered with spurious signals, and we become wiser. As the song goes, "I can see clearly, now, the rain has gone". But, of course, that 'clarity' has a price.

    I was so struck with your story that, as an (unpaid) editor of a weekly op-ed column in a small paper, I wanted to see if you might consider being a 'guest columnist' and allow me to post this in the paper. I'm not sure it could work (for one thing, most people have no idea what 'psychometrics' even means), but with a tweak, here and there, and a shortening to under 600 words, I think it could be worth a try. I would, of course, consult and allow you total control over the final decision.


  3. Thank you for explaining so very clearly the reasons you are acting on the courage of your convictions. Bravo!

  4. Excellent decision! Now if only AERA would follow in your footsteps.

  5. Bravo! Welcome back from the dark side! I hope you will not only disassociate yourself from the field of psychometrics, but will join the fight to break the stranglehold that the test developing companies have on the public school system. Robey Clark, Portland OR

  6. "[C]heap and scientific looking." in a single phrase Professor Glass sums up the entire neoliberal corporate education reform project.

    The full quote made the SJQA page.

  7. “You can take the boy out of the discipline of educational measurement, but you can’t take the discipline of educational measurement out of the boy.”

    One could argue that a departmental ID change from “Research & Evaluation Methodology” to “Educational Foundations, Policy & Practice” is not much of a gain, but the discipline of educational measurement is buried in each.
    It doesn’t seem reasonable to me to blame the “sins” of current standardized achievement testing and its detrimental impact on pre-collegiate schooling on psychometric pioneers of earlier years or to demean your personal contributions to the field. I’d chalk up the educational policy failures to psychometric ignorance rather than to psychometric science and technology. Irrespective of the blame, the inherent self-corrective mechanism of science and technology works to fix such errors, but sometimes it takes longer than “it should.”

    Anyway, whatever your departmental ID, you’re an educational measurement specialist.

  8. Dick;
    Self-corrective? The pragmatics needed for self-correction would require looking at testing as a practice, not as a technical problem. This was Messick's advocacy for consequential validity, for using multiple sources of data for important decisions, and Meehl's advocacy for a clear eyed view of scientific evaluation. Looking at practice (Pragmatism) is something close to a paradigm change, not the self-correction of normal science. Do you disagree?

    1. I dunno. As I understand the constructs, "inherent self-corrective mechanism" and "paradigm shift," pretty much go hand-in-hand. I'd settle for either or both, insofar as current standardized achievement practice in the US is concerned.

      Messick, Meehl, and their contemporaries and fore bearers, would, I think, be as appalled by the current "educational testing" milieu as Glass is. What to do to address the situation is a whole nother story.

  9. For me this is huge. Dr. Glass is perhaps the foremost educational psychometrician in the US if not the world . For him to see the deleterious effects that the improper use of his discipline is having on students is monumental. It gives validity to all of those students, teachers, and parents who have experienced the fallout of high stakes testing in the US.

  10. When I was learning research statistics, the standard cautionary tale on construct validity featured the foibles of Phrenology, the onetime pseudoscience that sought to psych out a person’s aptitude and character by measuring the bumps on his or her head to the last decimal place.

    It appears that Phrenology never dies, it just fads away in ever new fashions.