Bias in Mental Testing

Bias in Mental Testing - Jensen

Broken comb question, attacked because a poor child may perceive broken comb as normal, was 8th easiest out of 161 for black students, 10th for white, p.24

early 1900s England: intelligence tests doubled percentage of secondary school scholarship winners from working class backgrounds (compared to teachers' grades/recommendations) p.50

A bimodal distribution nearly always means that the sample of people represents two different groups

children of Terman gifted subjects had IQs following a normal distribution with mean 132 and standard deviation 15, p.83

20-25% of below-70 IQs are abnormal (Down's, birth trauma, etc.), concentrated below 50. Majority between 50-70 are biologically unremarkable, p.84

in situations that involve mainly physical or social skills, IQ is largely irrelevant until it falls below 60 or 70, p.107

authentic child prodigies are found only in chess, music, and mathematics, p.111

average single item on intelligence test is close to .3, or 15% better than chance, p.129

correlational analyses do not distinguish between breadth and altitude of intellect because the two are so closely correlated, typically .8 to .9, p.134

pattern for individually administered tests: start at relatively easy point, not base. if missed, lower until subject obtains five-ish consecutive correct. proceed until subject gets five-ish consecutive wrong. lower correctness and higher wrongness are then assumed, p.134

speed factor, e.g. making x's in rows of 300 boxes within 3 minuets, had about a .2 correlation with general intelligence for children between 9-12 years. small but present correlation

galton-cattel tests: late 1800s, battery requiring specialized lab equipment and testing wide range, first attempt at intelligence tests. flawed and limited but important, p.140

binet test: early 1900s, different tasks by age, French, modified by Terman into Stanford-Binet and given IQ scaling

vocabulary has a high relation with g, because intelligent people are capable of educing more meaning from single encounters with words and discriminating subtle differences between similar words, p.146

starting on p.148, lots of test item examples

classification items include group or individually administered, verbal nonverbal or performance, speed or power, breadth or altitude, culture-loaded or culture-reduced

animals can be tested in intelligence in speed of learning, complexity of what can be learned, integration of sensory information to achieve goals, flexibility of behavior against obstacles, insightful over trial-and-error problem solving, transfer of learning, etc. p.175

in the purest forms of trial-and-error learning, such as blind maze running, humans do not perform significantly better than lower animals, p.176

reverse discrimination: black button reward, white button none. after learned and mastered, buttons are reversed. see how long it takes animals. they don't differ so much in time to learn first discrimination, but in speed and possibility of learning reversals. fish show no sign of "catching on". monkeys learn it completely. 2-5 year old children take half the time of rhesus monkeys, p.178

oddity problem: 2+ identical objects (eg circles), 1 different (eg square). rewarded for picking up odd one. when learned, given new set of stimulus objects. reduced time to master each new oddity problem indicates learning oddity. no animals below primates mastered, young children also not mastered. can be made more complex by e.g. if background is green pick odd, if background is red pick normal, p.178

in memory tests to find hidden rewards, chimpanzees did better than 8 year olds and almost as well as adults--memory may not be great as measure of human intelligence differences (Harlow and Harlow, 1962, p.34) p.179

getting food: chickens in u-shaped barrier: run back and forth frantically rushing the food until they chance around the barrier. cats/dogs: examine, then quickly go around. four-sided barrier with pull string to open: cats/dogs: no clue. monkeys: ok cool i'll open it

normal and institutionalized children have been given same battery of tests Kohler used with chimpanzees and other investigators used with monkeys. Exactly the same rank order of difficulty of these problems emerged for human children as for chimpanzees and other apes, suggesting that these tests reflect similar functions across species, p.182

intelligences a, b, c: a - genotype, b - combination of genes/environment, where you're at now. phenotype, c - results of any one test, p.185

causes of correlation between measurable psychological variables:
1. sensory-motor skill (negligible for mental ability unless individual is handicapped)
2. part-whole relationship: shifting automobile gears is a subset of a driving test. also: clarinet and saxophone are more correlated than clarinet and violin.
3. functional relationship - one is prerequisite for other. digit-span is prerequisite for verbal arithmetic test
4. environmental correlation: boxing and hockey are not related skillwise but someone who knows about hockey is more likely to know about boxing than opera
5. genetic correlation: correlated because of common or correlated genetic determinants, p.195

restriction of range: height and weight correlation among people is between 0.6 and 0.7, but drops to 0.1-0.2 among pro basketball players. correlation between ability test and gpa will be lower in college sample than high school sample because of restricted range, p.196

factor analysis: get lots of correlations, then build 5-dimensional venn diagrams out of them. subtract areas that overlap between different skills until you are left with however many factors go into things. yeah I need to study more here. p.209

>>>>> Thurstone's primary mental abilities: number, word fluency, verbal meaning, memory, reasoning, space, perceptual speed. <<<<< (from Bischof, 1954, p.14), p.215
Major second-order group factors: v:ed verbal-educational (numerical, verbal, logical reasoning), k:m spatial, metchanical, "practical" (spatial visualization and understanding of physical and mechanical principles) (Vernon 1950), p.215

it proved impossible to devise tests involving complex cognitive functions that does not have a considerable loading on g, p.215

another: g, verbal-literary, verbal-linguistic, arithmetical, visuo-spatial, classification, memory, relational, audio-rhythmic

another! general, spatial-perceptual-motor, educational aptitude, verbal factor, numerical factor, p.219

another! sorted by g loading: relational .84, associative .54, perceptual .33, sensory-motor .23 taken from diverse british tests

finally we arrive at jensen's sorting from Hakstian and Cattell: fluid intelligence, artistic ability, crystalized intelligence, mechanical knowledge

"no one has ever assembled or factor analyzed such a comprehensive matrix of intercorrelated measures of all human characteristics as described in the preceding discussion" p.227 <- why not? also is this still accurate

highly g-loaded tests: raven's progressive matrices, verbal analogies (cut is to sharp as burn is to hot), series completion (1, 2, 4, 7, 11, __, __)
arithmetic problem reasoning: "John is 6 years old or half as old as his sister. what will be John's age when his sister is 40?"
paragraph comprehension
figure analogies and figure classifications, p.230

moderately g-loaded tests:
direct perception and manipulation
sentence completion
figure recognition
handwriting speed
counting speed
paired-associate learning
pitch, delayed discrimination

low g-loaded tests:
simple addition (speed)
counting groups of dots
crossing out designated letters or numbers
recognition memory of words and numbers
rote memory tasks
tapping speed
dotting speed

reaction time has essentially no g loading, but choice reaction time has a slight but significant g loading

backwards digit span has a significantly higher g loading than forward digit span p.231

crystallized and fluid intelligence: fluid peaks late teens early 20s, drops more after 55. crystallized slows growth in late teens but continues through 60-70. fluid intelligence has higher standard deviation than crystallized, p.236

if musical achievements were treated with the same uniformity and persistence of instruction throughout the school years that we see in the case of reading or arithmetic, we can be fairly sure that IQ tests would show a much higher correlation (though not a perfect correlation, because of special talents) with musical accomplishment than is now the case, p.240

tests of crystalized intelligence are usually better predictions of future achievement than fluid ones, since they reflect not only effects of g(f) but some of personality, motivational, and past opportunity factors involved in achievement, p.241

wrt testing: Conditional probability statements based on empirical research simply do not include or require the notions of capacity or potential. p.243

Reuning, 1972, p.179: Kalahari Bushmen, when given g-loaded cognitive performance tests, "accepted as a matter of fact that the 'clever ones' would do well on them." tribe members recommended as guides or opinion-givers tended to have above average scores, p.248

g factor is manifested in tests to the degree they involve mental manipulation of inputs, choice, decision, invention over reproduction, reproduction over selection, meaningful over rote memory, long-term over short-term memory, and distinguishing relevant info in complex problems, p.250

with driving tests: ease and quickness of learning correlates with g, but with prolonged practice driving becomes much easier. final level of performance will be more related to personality and very specific aptitudes, p.313

>>>>> in one training program, changes in methods of instruction reduced validity of math aptitude test in predicting training success from >.50 to ~0, but low math aptitude took 50% longer to complete in modified training. most would have failed course under old method, Ford & Meyer 1966, look this up or I swear I will smite you, p.313

no other single fact that we can determine about a child after age 5 better predicts future educational progress and attainments than IQ, p.317

from Hollingworth and Cobb: 146IQ cluster and 165 IQ cluster had achievement differences comparable to 100IQ & 120IQ cluster, do not differ in the least between each other in terms of quality of home background, p.320

achievement in foreign languages shows low or intermediate correlations with IQ as compared with other academic subjects, p.320

these tests predict performance in algebra much more highly than in shorthand.  The
same is true of achievement in English composition as compared with spelling or arithmetic concepts and reasoning problems as compared with arithmetic computation or so-called mechanical  arithmetic.

IQ  predicts  achievement  better  in  subjects that  are  hierarchically  ordered  in  complexity and in the sequence of cognitive skills and knowledge that are prerequisite to more advanced achievement, as in mathematics, the physical sciences, and engineering, than in less hierarchical subjects such as history and the social sciences.  The biological sciences are intermediate in this respect, p.321

when teachers "teach to the test" it most likely increases correlation between IQ and achievement test scores, because IQ correlates best with achievement test scores in the case of uniformity of exposure, p.322

predictive validity of IQ in reading is higher in lower-SES group than in higher-SES group

The Woodrow studies showed two main findings. (1) Measures of performance on a large variety of rather simple learning tasks showed only meager intercorrelations among the learning tasks, and between learning tasks and IQ. Factor analysis did not reveal a general factor of learning ability. (2) Rate of improvement with practice, or gains in proficiency as measured by the difference between initial and final performance levels, showed little or no correlation among various learning tasks or with IQ. Even short-term pretest-posttest gains, reflecting improvement with practice, in certain school subjects showed little or no correlation with IQ. Speed of learning of simple skills and associative rote learning, and rate of improvement with practice, seem to be something rather different from the g of intelligence tests. Performance on simple learning tasks and the effects of practice as reflected in gain scores (or final performance scores statistically controlled for initial level of performance) are not highly g loaded. p.327

POSSIBLY THE MOST IMPORTANT PART OF THE BOOK, PRINT IT, FRAME IT, ASK FOR MORE OF EXACTLY THAT
learning is more highly correlated with IQ in these situations:
1. when it is intentional, the tasks calls forth conscious mental effort, and is paced to permit "thinking" (simple repetition correlates only slightly with IQ)
2. when it is hierarchical and later learning depends on earlier mastery
3. when the material is meaningful (learning the order of three-letter nonsense syllables shows little correlation with IQ)
4. when the task permits transfer from different but related past learning
5. when it is insightful (proving pythagorean theorem vs learning capital cities)
6. when the material is of moderate difficulty and complexity (if too complex, everyone falls back on things like trial and error)
7. when amount of time for learning is fixed
8. when material is more age-related (some things can be learned almost as easily by a 9-year-old child as by an 18-year-old--WHICH THINGS PLEASE TELL ME THIS ONE'S IMPORTANT)
9. at an early stage of learning something "new" (related more to rate of acquisition of new skills than proficiency and improvement at later learning stages)
p. 328

Samuels and Dahl, 1973: "If we wish to reduce the correlation between IQ and achievement, the job facing the educator entails simplifying the task, ensuring that prerequisite skills are mastered, developing motivational procedures to keep the student on the task, and allocating a sufficient amount of time to the student so that he can master the task." p.329

median correlation of US Employment Service's General Aptitude Test Battery (good measure of g) and college grades: .40, p.330

IQ and field of study. why the differences? 1. hierarchical nature of math and science
2. greater objectivity and reliability of achievement criteria in math and science than humanities and arts
3. important role of special talents in the arts, p.331

self-dependence is greater in college and so lowers validity of IQ and aptitude tests in predicting grades (self-discipline, study habits, etc. become more important), p.331

ratio of males to females in college majors, attempt at explanation in terms of sex difference in math aptitude: correlations betwee nsex and major field (.15 to .28) are reduced (to .07-.17) when males and females are statistically equated on verbal and math SATs, p.332

> again, the mention of selection problems, where because of selection from grades and aptitude scores, you end up with many places selecting for either relative high GPA and low aptitude or low GPA and high aptitude--a perpetuation of "smart but lazy" phenomenon, p.333

considering the wide margin of discrepancy between objectively tested attainments and years of education, diplomas, and credentials, it seems an obvious conclusion that most employers and our social institutions in general have put far too much stock in sheer amount of schooling and formal credentials and not enough in objectively assessed actual achievement, p.334

IQ not stand-in for socioeconomic status: still correlation of .3 to .35 between IQ and years of schooling among brothers reared together, when IQ is measured in elementary school, p.335
> when IQ is partialled out, correlation between SES and scholastic achievement drops to .30; when SES is partialled out; correlation drops only to .62, p.336
childhood IQ determines about three times more of the variance in adult educational level than fother's education and occupational levels combined, p.336

more intelligence school children are usually better accepted socially by their classmates than are less intellectually favored, p.337
> note: it's time to drop my whole "social acceptance is a problem for smart people" thing, mildly at least. it may hold at very high ends.

ok, weird study. Schmidt, unpublished manuscript: foreman trainees rated peers for future probability of success. whites rating whites correlated .385, blacks rating blacks correlated .421, whites rating blacks was not significantly correlated (.088), p.338

not judgment of "average" person, but averaged judgments of many persons can show lots of consistency, such as when people judge occupational "level", which ends up correlating highly with average tested IQ in professions, p.339

a certain threshold level of intelligence is necessary but not sufficient for success in most occupations, p.344

inequality in occuational status between brothers is about 82% of status inequality in general population, correlation between brothers' occupational statuses is about .30 (Jencks, 1972, pp. 198,343), p.346

in recent years, the popular psychological and educational literature has promulgated the notion of "creativity" as a psychological trait quite distinct from, or even opposed to, general intelligence. the belief is probably born of the hope that, if a person is deficient in intelligence, there is a chance that he may possess an abundance of something at least equally valuable--creativity. However, there is no sound scientific basis for this hope, p.353

critical reviews of attempts to measure creativity (e.g. how many uses can you think for a brick?) have concluded that they show hardly any higher correlations with one another than with standard tests of intelligence--no g for creativity p.354

suggestion of threshold relationship of intelligence to creativity: probably around one standard deviation above mean IQ, there is little relationship between IQ and rated creativity; below threshold there is little creativity in culturally significant sense and creativity tests are hard to distinguish from general intelligence tests, p.355

"in a highly organized social system of a technological bent that tends to sort out people according to their abilities, and rewards them more or less accordingly, it seems not surprising to find that those traits of personality and temperament that complement and reinforce the development of intellectual skils... should become genetically assorted and segregated" p.356

Harvard psychologist Jerome Kagan was the first to report the observation of a negative correlation between hyperkinesis in young children 3-6 and their intellectual level at maturity, p.358
question: could this have something to do with ADHD and high iq kids, underdiagnosis, whatever? high iq kids are less on the H which makes the AD harder to notice?

inverse relationship between obesity and IQ: 41.4% lower quartile women obese, 10.7% upper quartile (men: 17.0%, 9.3%), p.363

core definition of unbiased test: major and minor groups share same regression line, and if they differ on test they differ on criterion as well, p.380

interesting selection model: expected utility model, where you use an unbiased test, then subjectively rate the comparative value of accepting different groups (accepting one might give +2, another +3) then choose based on that. p.410

item characteristic curve: curvy line representing percent of people at each raw score who got item right. should be an up diagonal. if it bumps up in the middle that's a problem, p.443

"all the foregoing methods of internal analysis for detecting culture bias in tests depend on two basic assumptions. The first basic assumption underlying the foregoing methods of internal analysis is that cultural differences between groups will interact with item content or item types and that group differences in cultural bacground should not produce equal effects on all of the items in a heterogeneous test. Rejection of this assumption can be cogent only if evidence can be adduced for the presence of some cultural factor that is hypothesized to have a uniformly enhancing or depressing effect across all the items in the test" p.451 (I believe this is the current standard steelman for tests being biased)

People who miss items do not pick distractors at random. a person's choice of distractors usually has as much test-retestability as their choice of correct responses, p.452

regarding predictive, concurrent, and postdictive measures of aptitude: it doesn't make much difference when you do the test, p.468

green and farquhar 1965, low correlation with GPA among non-WM students: "we can put little stock in the results and conclusions of this poorly designed and inadequately reported study", p.475

in cases where predictive bias is found, the use of the majority regression line or the common regression line almost invariably favors blacks relative to whites, p.476

biographical inventory of 300 items: 49 significantly differentiated race for males, females, or both sexes. when racially differentiating biographical factors are partialed out of correlation between biography and IQ, it drops from .45 to .25, indicating IQ measures racial difference independent of assessed life experience differences, p.477

in Coleman report (1966 report of hundreds of thousands of test scores), aptitude results overestimated achievement scores for minorities, p.480

high school gpa predicts college gpa better than sat scores for white students, reverse for black students, p.485

spuck & stout 1969, 32 low SES mexican-american students, another strange study people may cite, p.489

AFQT does show some validity differences between white and black testers-- .34 for white, .26 for black, possibly because of relative presence of rote learning in military, p.491

AFQT predicts job knowledge better than it does job performance, p.498

Boehm 1972 meta-analysis of 13 studies: 62% predictor tests nonsignificant in both, 17% significant in both, 4% significant differential (5% would be expected by chance), so no evidence of bias, p.500

from Ruch's 1972 review: no evidence in error of estimates or slopes of which selection effects would consistently favor group, but significant and consistent bias for white intercept being higher than black. selection procedures using same regression equation for both will be biased against white and in favor of back, p.505

piaget is good and worth studying more, cool stuff about tilted water in bottles and pouring coke from full can into basin and half full can into tall cylinder and stretching 6 pennies out to be longer than 8

jensen's views on uses of iq tests are intriguing, particularly in that he rejects universal testing in favor of achievement tests, seems largely uninterested in gifted programs except in cases of underachieving and particularly minority underachieving children.

"because of these limitations, differential "patterns of ability" (between groups on aptitude tests) are more artifacts of the particular groups, tests, and methods used than they are "facts of nature." (p.731)

"fewer than one fourth of the individuals in the four ethnic samples of the Lesser, Fifer, and Clark study can be correctly sorted into their respective ethnic groups on the basis of the resemblance of their individual profiles to the groups' profiles (Feldman, 1973)" (p.731)

it's worth paying attention to which less g-loaded aptitudes correlate with which jobs, as those aptitudes can allow more intellectually diverse groups to flourish in those jobs, 739ish

"For many years now it has been practically  axiomatic  among people in the testing field  that  the  fact  of  statistical  differences  between  racial  populations  should  not  be permitted  to  influence  the  treatment  accorded to individuals  of any race—in  education, employment, legal justice, and political and civil rights. The well-established finding of a wide  range  of  individual  differences  in  IQ  and  other  abilities  within  all  major  racial populations,  and the great amount of overlap of their frequency  distributions,  absolutely contradicts  the  racist  philosophy  that  persons  of  different  races  should  be  treated  differently,  one and all, only by reason of their racial origins.  Those who would accord any treatment to individuals solely by virtue of their race will find no rational support in any of the  scientific  findings  from  psychological  testing  or present-day  theories  of differential psychology.  That much seems certain.  Righting the past wrongs of racial discrimination cannot be furthered by blaming the mental tests (which we admittedly should continue to improve and to use more wisely), but by prohibiting racial discrimination in any form, by legal sanctions when necessary, and by seeking equal educational opportunities for members of those minority groups that have been denied them in the past, so they can compete fairly, as individuals, in selection for employment, technical training, or higher education, without condescending dispensations." (p.737-738, conclusion of book)