AI summary by TwinMind

Why AI doesn't speak every language

Vox

10m 15s

Large Language Model Fundamentals

  • ChatGPT is app sitting on top of large language model (version of GPT)
  • Models perform natural language processing for customer service, auto completion
  • GPT analogy: used bookstore that never left room, only learned through books in store
  • Process: scan text, learn language, check by covering answers, use knowledge for sentiment analysis, summarization, translation, response generation

Training Data Language Distribution Problem

  • Common Crawl data 2008-present: massive language bias
    • English: >40% of every crawl
    • German (DEU): ~6% consistently
    • Finnish (FIN): 0.4% in 2023 despite many pages
  • Research shows: 7,000 languages spoken globally, only ~20 languages make up bulk of NLP research
  • Languages categorized as high-resource vs low-resource
    • High-resource: English, German, Chinese (like popular books in bookstore)
    • Low-resource: don't appear as text online, don't make it into datasets, become unintelligible to AI

Jamaican Patois Research Case Study

  • Ruth-Ann Armstrong: researcher creating new datasets for low-resource languages
  • Jamaican patois: Creole language spoken in Jamaica alongside English
  • Dataset creation process:
    • 650 examples of Jamaican patois statements
    • Two-column format with entailment labeling (entailed/agreed, contradicted, neutral)
    • Extremely labor-intensive manual work
  • Jamaican patois not listed in Common Crawl languages

Catalan Language Analysis

  • Catalan: most spoken language in autonomous community of Spain
  • GPT-3 training data composition:
    • English words: 92%
    • German words: 1.4%
    • Spanish: 0.7%
    • Catalan: 0.01% of entire training set
  • Common Crawl reports Catalan as 0.2335% of survey
  • Performance testing:
    • GPT-3 generated 3 Catalan sentences mixed with real sentences
    • 3 native speakers evaluated results
    • Results were "very good for the machine"
  • Key issues:
    • Performance reasonably good but language-specific models still preferable
    • Transparency problem: GPT-3 claims only ~140 pages of Catalan (novella-length)
    • Dependency risk on few institutions/companies
    • Example: Google cutting Catalan links in searches

Corporate Control and Transparency Issues

  • Major AI companies control training data: Meta, Microsoft, Baidu, OpenAI, Google
  • Companies decide which content included without disclosure of sources/authors
  • Common Crawl represents only fraction of GPT-3 training data
  • GPT-4 training details unknown
  • Risk: companies could arbitrarily cut out languages

Open-Source Alternative: BLOOM Project

  • Location: Paris supercomputer with underutilized GPU capacity
  • Thomas Wolf: co-founder of Hugging Face (AI research hub)
  • Big Science's BLOOM: open-source multilingual model project
  • Approach:
    • Started from Wikipedia list of most spoken languages
    • Added low-resource languages when possible
    • Focus on African languages for very low-resource inclusion
    • Partnered with local communities to identify good data sources
    • Transparent data sourcing and acquisition methods
  • Key advantage: complete knowledge of data sources ("know the books in library")

Benefits for High-Resource Language Speakers

  • English speakers are primary target audience (>40% Common Crawl)
  • Personal example: Jamaican accent initially caused Siri comprehension issues
  • Expanding training datasets to include more accents proved helpful for voice assistants
  • Broader language representation benefits everyone through improved trust and accessibility
  • Catalan speakers' perspective: "It's our language" - ownership and representation matter

Digital Footprint vs Real-World Usage Gap

  • Languages may have sizable real-world speaker populations but minimal digital presence
  • Risk: languages with small digital footprints "bound to just disappear" from AI systems

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome