AI summary by TwinMind
Research Bias: Sort It Out, Science | STUFF YOU SHOULD KNOW
Stuff You Should Know
53m 4s
Introduction & Context Setting
- Hosts Josh Clark and Charles W Chuck Bryant discuss erosion of expertise/trust in science
- Major problem: general public dismissing expert knowledge in favor of YouTube videos, social media
- Contributing factors: sloppy/fraudulent science by some experts undermining faith in profession
- Science allowing too much bias to creep in, producing useless/trash papers
- Need for scientific community leadership to address bias problems
Confirmation Bias
- Most well-known bias affecting both scientists and general public
- Definition: seeking information that supports hypothesis while avoiding contradictory evidence
- Historical example: René Blondlot's "N-rays" (1903) - French scientist claimed discovery of new rays
- American Robert Wood secretly removed crystals during experiment
- Blondlot still "saw" N-rays, proving confirmation bias
- Other French scientists published papers supporting false discovery
- Occurs in study design (methodology shaped to confirm hypothesis) and interpretation phases
Sampling Bias (Selection Bias Subtype)
- Pre-study bias in participant selection
- Problems with small-scale studies using only university students (WEIRD population)
- Online survey limitations - excludes non-internet-savvy populations
- Self-selection bias: volunteers for wellness studies already health-conscious
- Historical example: 1936 Roosevelt vs Landon polling disaster
- Polls conducted at country clubs, car owners (luxury at time)
- Predicted Landon landslide, Roosevelt actually won
- Survivorship bias: studying only successful businesses, ignoring failures
- WWII example: reinforcing plane sections that survived hits vs. studying downed planes
- Channeling bias: patients with different illness severity channeled into different treatment groups
- Example: new hardcore antacid given to sicker patients, making drug appear dangerous
Question Order Bias
- During-study bias affecting survey/interview responses
- Brain organizes information into patterns, pays more attention early on
- 1984 General Social Survey example: honesty picked 66% when listed high, 48% when listed low
- Priming effects: earlier questions influence later responses
- Solutions: randomize question order, ask questions in multiple ways
Interviewer Bias
- During-study bias where researcher influences participant responses
- Occurs through tone of voice, body language, facial expressions
- Example: medical questionnaire where interviewer knows patient has disease
- May probe more intensely: "Are you sure you've never smoked? Not even one?"
- Solutions: internet surveys, robot/AI questioners, proper interviewer training
Recall and Acquiescence Bias
- Participant-related biases during study phase
- Recall bias: faulty memory when recalling past events
- People remember things that make them look bad less accurately
- False correlations created when participants "remember" connections
- Acquiescence bias: tendency to agree/answer affirmatively
- More common among less educated participants
- Solutions: reframe questions to provide comfortable "back door" responses
- Example: Instead of "Did you vote?" ask "Did things come up that prevented you from voting, or were you able to vote?"
Publication Bias
- Academic publishing industry has "stranglehold" on science
- Similar to 24-hour cable news effect on journalism - voracious beast needing constant content
- Publishers prefer "positive results" (studies showing correlations/effects)
- 2010 study: social science papers 2.3x more likely to show positive results vs. physical sciences
- Creates pressure for researchers to cherry-pick data or use questionable statistical methods
- "Publish or perish" mentality forces scientists to produce publishable results
- High-quality studies with negative outcomes ignored in favor of low-quality positive studies
File Drawer Bias
- Negative results literally filed away, never published
- Prevents building comprehensive scientific knowledge base
- Wastes resources - other researchers repeat failed experiments unknowingly
- Proteus effect: 70% of initial studies cannot be reproduced by follow-up studies
- Legal/ethical issues with burying negative drug trial results
- 2007 Congressional law requiring reporting of human experimental treatment studies
- 2016 strengthening of requirements
Solutions and Improvements
- The Lancet's innovative approach: accepting papers based on study design/methodology before results known
- Commits to publishing regardless of positive/negative outcomes
- Prevents low-quality studies from being conducted
- Scientists need to acknowledge bias exists and actively work to counteract it
- Transparency in reporting potential biases in published studies
- Better training for researchers on bias recognition and prevention
