21 - Chapter 21: Intuitions vs. Formulas

Chapter 21: Intuitions vs. Formulas

·         Paul Meehl, Clinical vs. Statistical Prediction: A Theoretical Analysis and a Review of the Evidence, predict’ns based on simple stats (algorithm, formula) more accurate than predict’ns of experts who have lots of data
·         rarely good idea to substitute judg’t for formula — of course, formula shd be ignored in certain cases, e.g. formula to predict  whether a person will go to movies tonight, ignore formula upon learning he has broken leg (“broken-leg rule”)
·         Why is expert judg’t so poor ?

• o   experts can’t account for many factors (even tho they believe they can)
• o   judg’ts are inconsistent when based on complex info, asked to re-evaluate same info, give different judg’ts — we are infl’d by many factors, some imperceptible (see Priming)

·          we shd use only formulas for final decisions (e.g. not interviewers, who are too confident of their intuitions), esp. in low validity envir’ts
·         complex stat’l algorithms, such as weightings (“multiple regression”) not result in more accurate predict’ns — equal weighting as good or better — an algorithm constructed on back of envelope often good enough to compete w. an optimally weighted formula, certainly will outdo expert judg’t — Robyn Dawes, “The Robust Beauty of Improper Linear Models in Decision Making” — See Atul Gawande, A Checklist Manifesto,  many examples of virtues of checklists & simple rules  to deal with complex situ’ns w. many factors

 The Hostility to Algorithms

·         demystification of expertise
·         clinical vs. stat’l predict’n — clinicians not accept Meehl, know their predictive judg’ts are quite accurate — but clinicians’ accuracy is valid only for short-term predict’ns — long-term predict’ns not accurate, less than formula
·         we prefer human over mechanical — prefer human error (e.g. causing death) to mechanical error

Learning from Meehl

·          e.g. story of interviewing new recruits in Israeliy Army in 1955, poor results w. “psych’l” tests & interviewer’s summary impression, better w. easier checklist of facts abt recruit’s life to assess certain qualities, then formula — interesting that after interviewers used checklist, their intuitive judg’ts as good as formula, because it came after disciplined evalu’n + scoring of objective facts

 Do It Yourself

·          e.g. instructions for hiring a new saleman

Speaking of Judges vs. Formulas

·         “Whenever we can replace human judgment by a formula, we should at least consider it.”
·         “He thinks his judgments are complex and subtle, but a simple combination of scores could probably do better.”
·         “Let’s decide in advance what weight to give to the data we have on the candidates’ past performance. Otherwise we will give too much weight to our impression from the interviews.”