June 21, 2010
A new study may revive arguments that the average test scores of black students trail those of white students not just because of economic disadvantages, but because some parts of the test result in differential scores by race for students of equal academic prowess.Stop and think for a moment about what the fallacy might be.
The finding — already being questioned by the College Board — could be extremely significant as many colleges that continue to rely on the SAT may be less comfortable doing so amid allegations that it is biased against black test-takers.
"The confirmation of unfair test results throws into question the validity of the test and, consequently, all decisions based on its results. All admissions decisions based exclusively or predominantly on SAT performance — and therefore access to higher education institutions and subsequent job placement and professional success — appear to be biased against the African American minority group and could be exposed to legal challenge," says the study, which has just appeared in Harvard Educational Review (abstract available here).
The existence of racial patterns on SAT scores is hardly new. The average score on the reading part of the SAT was 429 for black students last year — 99 points behind the average for white students. And while white students' scores were flat, the average score for black students fell by one. Statistics like these are debated every year when SAT data are released, and when similar breakdowns are offered on other standardized tests.
The standard explanation offered by defenders of the tests is that the large gaps reflect the inequities in American society — since black students are less likely than white students to attend well-financed, generously-staffed elementary and secondary schools, their scores lag.
In other words, the College Board says that American society is unfair, but the SAT is fair. And while many educators question that fairness of using a test on which wealthier students do consistently better than less wealthy students, research findings that directly isolate race as a factor in the fairness of individual SAT questions have, of late, been few.
The new paper in fact is based on a study that set out to replicate one of the last major studies to do so — a paper published in the Harvard Educational Review in 2003, strongly attacked by the College Board — and the new paper confirms those results (but using more recent SAT exams). The new paper is by Maria Santelices, assistant professor of education at the Catholic University of Chile, and Mark Wilson, professor of education at the University of California at Berkeley. The earlier study was by Roy Freedle of the Educational Testing Service.
The focus of both studies is on questions that show "differential item functioning," known by its acronym DIF. A DIF question is one on which students "matched by proficiency" and other factors have variable scores, predictably by race, on selected questions. A DIF question has notable differences between black and white (or, in theory, other subsets of students) whose educational background and skill set suggest that they should get similar scores. The 2003 study and this year's found no DIF issues in the mathematics section.
But what Freedle found in 2003 has now been confirmed independently by the new study: that some kinds of verbal questions have a DIF for black and white students. On some of the easier verbal questions, the two studies found that a DIF favored white students. On some of the most difficult verbal questions, the DIF favored black students. Freedle's theory about why this would be the case was that easier questions are likely reflected in the cultural expressions that are used commonly in the dominant (white) society, so white students have an edge based not on education or study skills or aptitude, but because they are most likely growing up around white people. The more difficult words are more likely to be learned, not just absorbed.
While the studies found gains for both black and white students on parts of the SAT, the white advantage is larger such that the studies suggest scores for black students are being held down by the way the test is scored and that a shift to favor the more difficult questions would benefit black test-takers.
Ready? Here goes:
By definition, blacks and whites are equally good at randomly guessing on multiple choice questions. So, the more difficult the question and thus the higher the percentage of students who randomly guess, the narrower the white-black differential.
If you made all the questions impossible esoteric, the white-black gap would disappear. If you made them all unbelievably easy, the white-black gap would also disappear. But when you make them a reasonable mix of difficulty in order to maximize the predictive value of the SAT, you wind up with a white-black gap — because there is also a white-black gap in real world performance.