AI summary by TwinMind

Research Bias: Sort It Out, Science | STUFF YOU SHOULD KNOW

Stuff You Should Know

53m 4s

Introduction & Context Setting

  • Hosts Josh Clark and Charles W Chuck Bryant discuss erosion of expertise/trust in science
  • Major problem: general public dismissing expert knowledge in favor of YouTube videos, social media
  • Contributing factors: sloppy/fraudulent science by some experts undermining faith in profession
  • Science allowing too much bias to creep in, producing useless/trash papers
  • Need for scientific community leadership to address bias problems

Confirmation Bias

  • Most well-known bias affecting both scientists and general public
  • Definition: seeking information that supports hypothesis while avoiding contradictory evidence
  • Historical example: René Blondlot's "N-rays" (1903) - French scientist claimed discovery of new rays
    • American Robert Wood secretly removed crystals during experiment
    • Blondlot still "saw" N-rays, proving confirmation bias
    • Other French scientists published papers supporting false discovery
  • Occurs in study design (methodology shaped to confirm hypothesis) and interpretation phases

Sampling Bias (Selection Bias Subtype)

  • Pre-study bias in participant selection
  • Problems with small-scale studies using only university students (WEIRD population)
  • Online survey limitations - excludes non-internet-savvy populations
  • Self-selection bias: volunteers for wellness studies already health-conscious
  • Historical example: 1936 Roosevelt vs Landon polling disaster
    • Polls conducted at country clubs, car owners (luxury at time)
    • Predicted Landon landslide, Roosevelt actually won
  • Survivorship bias: studying only successful businesses, ignoring failures
    • WWII example: reinforcing plane sections that survived hits vs. studying downed planes
  • Channeling bias: patients with different illness severity channeled into different treatment groups
    • Example: new hardcore antacid given to sicker patients, making drug appear dangerous

Question Order Bias

  • During-study bias affecting survey/interview responses
  • Brain organizes information into patterns, pays more attention early on
  • 1984 General Social Survey example: honesty picked 66% when listed high, 48% when listed low
  • Priming effects: earlier questions influence later responses
  • Solutions: randomize question order, ask questions in multiple ways

Interviewer Bias

  • During-study bias where researcher influences participant responses
  • Occurs through tone of voice, body language, facial expressions
  • Example: medical questionnaire where interviewer knows patient has disease
    • May probe more intensely: "Are you sure you've never smoked? Not even one?"
  • Solutions: internet surveys, robot/AI questioners, proper interviewer training

Recall and Acquiescence Bias

  • Participant-related biases during study phase
  • Recall bias: faulty memory when recalling past events
    • People remember things that make them look bad less accurately
    • False correlations created when participants "remember" connections
  • Acquiescence bias: tendency to agree/answer affirmatively
    • More common among less educated participants
    • Solutions: reframe questions to provide comfortable "back door" responses
    • Example: Instead of "Did you vote?" ask "Did things come up that prevented you from voting, or were you able to vote?"

Publication Bias

  • Academic publishing industry has "stranglehold" on science
  • Similar to 24-hour cable news effect on journalism - voracious beast needing constant content
  • Publishers prefer "positive results" (studies showing correlations/effects)
  • 2010 study: social science papers 2.3x more likely to show positive results vs. physical sciences
  • Creates pressure for researchers to cherry-pick data or use questionable statistical methods
  • "Publish or perish" mentality forces scientists to produce publishable results
  • High-quality studies with negative outcomes ignored in favor of low-quality positive studies

File Drawer Bias

  • Negative results literally filed away, never published
  • Prevents building comprehensive scientific knowledge base
  • Wastes resources - other researchers repeat failed experiments unknowingly
  • Proteus effect: 70% of initial studies cannot be reproduced by follow-up studies
  • Legal/ethical issues with burying negative drug trial results
  • 2007 Congressional law requiring reporting of human experimental treatment studies
  • 2016 strengthening of requirements

Solutions and Improvements

  • The Lancet's innovative approach: accepting papers based on study design/methodology before results known
    • Commits to publishing regardless of positive/negative outcomes
    • Prevents low-quality studies from being conducted
  • Scientists need to acknowledge bias exists and actively work to counteract it
  • Transparency in reporting potential biases in published studies
  • Better training for researchers on bias recognition and prevention

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome