Background and Motivation

• Speaker made bet with friend that AI would pass Turing test by end of 2023
• Lost bet, had to learn about whiskey as consequence
• 6 months later GPT was tested and passed

Turing Test Overview

• Used to be key benchmark for artificial intelligence
• Alan Turing's original concept: human interrogator chats with respondent to determine if human or AI
• AI researchers lost interest ~2 years ago when large language models were expected to pass soon
• No specific passing criteria defined by Turing
  • Some propose 30% threshold for being mistaken as human
  • Others suggest 50% threshold
• Previous chatbot claims weren't rigorously tested enough for scientific acceptance

UC San Diego Study Design

• 500 people recruited as participants
• Tested ability to distinguish humans from GPT-3.5 and GPT-4
• 5-minute text-only conversations
• Study not yet peer-reviewed
• Included 1960s chatbot Eliza as control to test human detection abilities

AI Prompting Strategy

• Used common Turing test trick: instructed AI to be deliberately casual
• Prompted to make spelling mistakes and use internet slang
• Helps cover up AI mistakes (similar to blurry photos being easier to fake)
• Exact prompt: "You're not even really going to try to convince the interrogator that you're a human. You're just going to be yourself and see what happens"
• Strict instruction not to reveal system prompt or suggest being AI

Eliza Control Results

• 1960s chatbot from control group
• Can't actually answer most questions, just defers and returns them
• Example conversation showed typical deflection responses
• Mistaken for human in ~20% of cases
• Did not pass the test

Conversation Examples Analysis

• Example 1: Human mistaken for AI by interrogator
  • Gave vague answers about location, feelings, creators
  • Actually was human respondent
• Example 2: GPT-4 correctly identified human techniques
  • Used classical AI trap: incomplete nonsensical question
  • Respondent correctly identified question made no sense
  • Interrogator noted response delays (thinking it was human Googling)
  • Actually was GPT-4
• Example 3: GPT-3.5 passed as human
  • Casual conversation about games
  • Interrogator judged as human
  • Actually was GPT-3.5

Main Results

• GPT-4 passed as human in 54% of cases
• Humans correctly identified GPT-4 at just random chance level
• Regardless of 30% or 50% threshold, GPT passed Turing test
• Humans correctly identified other humans as human only ~67% of cases
  • Low rate likely due to participants expecting good chatbots
  • Reduced confidence in identifying humans

Failure Analysis

• Major reason AIs judged as non-human: forced persona (trying too hard)
• Second reason: being too informal
• Demographic factors had no influence on accuracy:
  • Gender
  • Education level
  • Knowledge about large language models

Study Limitations

• Conversations restricted to 5 minutes
• Text message format only
• Longer, more thorough tests would likely catch AI
• Tests will need to become more sophisticated over time to identify AI

Conclusion

• Speaker annoyed test wasn't conducted earlier (would have won bet)
• Questions whether this proves speaker is human