AI summary by TwinMind

AI passed the Turing Test -- And No One Noticed

Sabine Hossenfelder

8m 45s

Background and Motivation

  • Speaker made bet with friend that AI would pass Turing test by end of 2023
  • Lost bet, had to learn about whiskey as consequence
  • 6 months later GPT was tested and passed

Turing Test Overview

  • Used to be key benchmark for artificial intelligence
  • Alan Turing's original concept: human interrogator chats with respondent to determine if human or AI
  • AI researchers lost interest ~2 years ago when large language models were expected to pass soon
  • No specific passing criteria defined by Turing
    • Some propose 30% threshold for being mistaken as human
    • Others suggest 50% threshold
  • Previous chatbot claims weren't rigorously tested enough for scientific acceptance

UC San Diego Study Design

  • 500 people recruited as participants
  • Tested ability to distinguish humans from GPT-3.5 and GPT-4
  • 5-minute text-only conversations
  • Study not yet peer-reviewed
  • Included 1960s chatbot Eliza as control to test human detection abilities

AI Prompting Strategy

  • Used common Turing test trick: instructed AI to be deliberately casual
  • Prompted to make spelling mistakes and use internet slang
  • Helps cover up AI mistakes (similar to blurry photos being easier to fake)
  • Exact prompt: "You're not even really going to try to convince the interrogator that you're a human. You're just going to be yourself and see what happens"
  • Strict instruction not to reveal system prompt or suggest being AI

Eliza Control Results

  • 1960s chatbot from control group
  • Can't actually answer most questions, just defers and returns them
  • Example conversation showed typical deflection responses
  • Mistaken for human in ~20% of cases
  • Did not pass the test

Conversation Examples Analysis

  • Example 1: Human mistaken for AI by interrogator
    • Gave vague answers about location, feelings, creators
    • Actually was human respondent
  • Example 2: GPT-4 correctly identified human techniques
    • Used classical AI trap: incomplete nonsensical question
    • Respondent correctly identified question made no sense
    • Interrogator noted response delays (thinking it was human Googling)
    • Actually was GPT-4
  • Example 3: GPT-3.5 passed as human
    • Casual conversation about games
    • Interrogator judged as human
    • Actually was GPT-3.5

Main Results

  • GPT-4 passed as human in 54% of cases
  • Humans correctly identified GPT-4 at just random chance level
  • Regardless of 30% or 50% threshold, GPT passed Turing test
  • Humans correctly identified other humans as human only ~67% of cases
    • Low rate likely due to participants expecting good chatbots
    • Reduced confidence in identifying humans

Failure Analysis

  • Major reason AIs judged as non-human: forced persona (trying too hard)
  • Second reason: being too informal
  • Demographic factors had no influence on accuracy:
    • Gender
    • Education level
    • Knowledge about large language models

Study Limitations

  • Conversations restricted to 5 minutes
  • Text message format only
  • Longer, more thorough tests would likely catch AI
  • Tests will need to become more sophisticated over time to identify AI

Conclusion

  • Speaker annoyed test wasn't conducted earlier (would have won bet)
  • Questions whether this proves speaker is human

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome