The Unfairness of Accuracy
Ladies and Gentlemen, we now have the greatest example ever of why quantified, standardized tests should not be used as the way of certifying students’ achievements. A novelist couldn’t have come up with a better one.
In Massachusetts, we’ve introduced the Massachusetts Comprehensive Assessment System (MCAS) that all students must pass in order to get a high school diploma. Fail the MCAS, no diploma.
So, along comes Jennifer Mueller, a senior at the Whitman-Hanson Regional High School in (Go Panthers!) who took the MCAS last spring and got a score of 218, two points short of passing. So, she looked over the results and thought there was one particular answer marked wrong that she was convinced she’d gotten right. So, she took it up with the State Department of Education and convinced them that her answer was acceptable. So, the Department upgraded her and everyone with the same answer. As a result, 557 students who had failed, now passed.
You can read the question in question at the Boston Globe site for the next couple of days before the article gets moved to the Ash Heap of Broken Links. It represents the numbers from 0-10 in binary by showing each as a set of 4 circles, either white or black. Then it gives a multiple choice answer asking you to find the set of 4 circles that express 11.
Jennifer Mueller looked at the sets of white and black circles in row after row and observed a social pattern: The first white circle has dinner alone. Then the second white circle has dinner alone. Then they have dinner together. Then a third white circle has dinner alone. Then the three of them have dinner together. That pattern holds: a white circle has to have dinner alone before it can have dinner with the rest. Jennifer used this to correctly predict the next in the sequence. Unfortunately, the next line in her pattern expresses the number 12 in computer language, not the number 11. The pattern she found is indeed there. Her answer is right. And she deserves credit for it.
Jennifer’s discovery hits all the bases. She came up with a pattern that took a particular type of brilliance to see. Furthermore, to make this a textbook example, the pattern she came up with depended on her thinking about things in terms of pictures, not numbers. And she thought this through in terms of sociality — those dots were having dinner together. Where’s that type of intelligence being measured and valued?
The biggest point is that 500 students who yesterday were not worthy of a high school diploma today are worthy, without anything about them changing. I understand that these tests are standardized and quantified in order to make them fair, but here’s a case where accuracy actually works against fairness. A score of 220 passes, 218 fails: because it’s so precise, who passes and fails is arbitrary. It has to be, because being educated doesn’t have a sharp edge: “Yes you are, no you’re not, and here’s the dividing line.” It doesn’t work that way, no matter how much comfort we get from thinking that tests can sort us into two piles. “Accurate and unfair.” It’s not a junction we’re comfortable with, but it’s there. Fairness is all about our ability to make exceptions, to look at context, to dwell on what the numbers don’t say…Just like Jennifer Mueller, who almost failed because she is so much smarter than the test she took.
Categories: Uncategorized dw
Am I missing something here?
First, cheers for Jennifer Mueller for showing us again how right Carol Gilligan is. The writers of her question (I’m guessing male) saw it in terms of math, she instinctively saw it in terms of relationships.
Second, more cheers to Jennifer for showing such creative visual intelligence.
Third, I don’t know which I hate more, the idea of the MCAS or the test itself.
But….
The lead-in to her question talks about “off/on switches used to represent numbers” and the question itself asks explicitly “which of the following represents the number 11?”
Now if the question had been “which is a logical next item in this sequence of dot patterns?” Jennifer’s answer and answer ‘C’ would work equally well. But I don’t see how Jennifer’s answer can be construed as a “represent[ation of] the number 11” given the pattern examples representing numbers 1 through 10 that precede it. The 11th item in a series, yes; the *number* 11, no.
In what sense is her answer right then? What is it that the judges have caved in to here? Are they really as inept as they appear to be?
Stu, I’d thought about the issue you raise. Here’s why I think they were right to accept her answer.
Suppose the puzzle were posed in terms of an alien counting system instead of in terms of how computers count. You would assume (I think) that you weren’t supposed to crack the code, just figure out the next in the sequence, and then JM’s answer would clearly be right. Yes?
If so, then by couching it in computer terms, the test is biased towards those who have been taught base 2 because they have an additional way to figure it out; in fact, if you know binary, this question doesn’t require the 11 examples they give you. Since the examples are there, the test is assuming you DON’T know base 2. And in that case, it’s like looking at the alien counting system where you’re simply trying to find a pattern.
Now, there’d be nothing wrong with the test testing you on your understanding of base 2 if that were part of the curriculum. But, since they give the 11 examples (and based on the way they explain the issue), clearly it’s not testing you on your knowledge of base 2 but on your ability to figure out the equivalent of an alien counting system.
I think.
You missed something David, Jennifer still failed the test:
—————-
Despite making the discovery in an MCAS tutorial for students who had failed the test, Mueller was chagrined to find that her own score of 218 still fell short of the 220 required to pass. ”I want to graduate. I want to be able to walk down the stadium with my class,” Mueller said yesterday. Still, she said, ”I’m pretty excited about it, because 500 other students passed.”
—————-
On the plus side, she has 500 very good friends who didn’t know her name last week…
I was the teacher that was helping this student during a tutorial review. I got a copy of the exam that she took and I looked at the answers that she gave and compared them to the correct answers. As we did each question hoping to reinforce the material that she knew and to locate her weaknesses I asked the student to explain how she got the answer that she did to the famous question. Other students that got the answer correct said that they looked at the problem vertically and went down each column and said what came next in the pattern. Only one student attacked the problem with the binary way. Since the others got it right by a pattern I thought it well to have Jennifer explain her answer and question it. In so doing I took the student to the Superintendents office where Ruth Whitner listened and had me write up the students explanation. Never was any dinner party method ever mentioned. She saw a pattern . From there it went on and on to this point. …How wonderful that we listen to students and try to help them Isn’t that what education is all about. Without getting too philosophical and arguing about this let’s put it to rest.
Yup, Jonathan, you’re right.
But, in some ways it makes it a better story since someone who came up with an original way of answering the question still failed the test.
Regina,
Great to hear from you.
The “dinner party” story came from the Globe.
But, why do you think it should be “put to rest”? She does not seem to mind having her story told. What am I missing?
Too bad that even “MCAS critics” (at least the single one quoted in the article) characterize this as an error in the test. When you treat it that way, you’re buying the underlying assumption that testing works and is a good idea. After all, errors in detail can be fixed. I think the real point is that the MCAS situation is an outcome of rhetoric overcoming logical thinking. You have to be able to clearly identify something in order to measure it. However, it seems to me that they’re saying “this test measures education” and “education is the score on this test”. A meaningless tautology that’s clouded by the “source credibility” associated with the governmental agencies who instituted the test. In other words, it looked like a good idea to many people not because of the idea, but because of whose idea it was.
The problem here is a misunderstanding of binary. My (home-schooled) sons aged 5 and 7 understand binary, and can do arithmetic in it; my wife taught them as an aside while doing place value for base 10, and they play the ‘think of a number game’ with me using a binary search.
What this shows is how Mass. public schools have failed to teach an idea as readily grasped as binary.
It also shows that having a test with a 1-bit result (pass/fail) is a mistake, though sadly common.
Standardised tests are a good idea. Discarding their information conten by reducing the results to one bit is not.
In response to Kevin Marks, but to other defenders of the test as well, I believe this student shows the absurdity of trying to measure intelligence and learning of a socially constructed and socially situated knowledge base.
The test attempts to gauge universals. The student’s far more logical response shows exactly how socially situated all knowledge is. Reminds me of the recording sent into space on one of our deep probes, or the radio waves, because the assumption is that those who would encounter such recordings or radio waves would have the receptors to decipher them.
Aristotle didn’t “invent” the syllogism and binary mathematics were not just sitting around on the ground waiting for someone to “receive” the knowledge. They were socially constructed by a particular mindset in a very particular milieu.
My favorite philosophical view on this matter comes from Henry David Thoreau:
“Why level downward to our dullest perception always, and praise that as common sense? … Sometimes we are inclined to class those who are once-and-a-half witted with the half-witted, because we appreciate only a third part of their wit.” (Thoreau, 373) Walden.
There’s an essay with this quote, a meditation on gifted and talented education and the absurdities of what currently passes for school at http://WWW.NUTBALL.COM/moveablefeast/archives/000297.htm
Miasma
There was a similar incident with (i think) the SAT some years ago.
The question was: If you stick these two solids together so, how many faces will the resulting solid have?
A student noticed that two of the component faces would be coplanar in the compound, and so his answer was one less than the expected answer.
Hi jen it’s chris from maine great job on the MCAS discoverery. When i lived in MASS and took the test I had a hard time and when i would have been in high school i would have had to take thye test (p.s.)i’ve gotten ten times better in pool so when you come up next time your going down