AI summary by TwinMind
Why AI doesn't speak every language
Vox
10m 15s
Large Language Model Fundamentals
- ChatGPT is app sitting on top of large language model (version of GPT)
- Models perform natural language processing for customer service, auto completion
- GPT analogy: used bookstore that never left room, only learned through books in store
- Process: scan text, learn language, check by covering answers, use knowledge for sentiment analysis, summarization, translation, response generation
Training Data Language Distribution Problem
- Common Crawl data 2008-present: massive language bias
- English: >40% of every crawl
- German (DEU): ~6% consistently
- Finnish (FIN): 0.4% in 2023 despite many pages
- Research shows: 7,000 languages spoken globally, only ~20 languages make up bulk of NLP research
- Languages categorized as high-resource vs low-resource
- High-resource: English, German, Chinese (like popular books in bookstore)
- Low-resource: don't appear as text online, don't make it into datasets, become unintelligible to AI
Jamaican Patois Research Case Study
- Ruth-Ann Armstrong: researcher creating new datasets for low-resource languages
- Jamaican patois: Creole language spoken in Jamaica alongside English
- Dataset creation process:
- 650 examples of Jamaican patois statements
- Two-column format with entailment labeling (entailed/agreed, contradicted, neutral)
- Extremely labor-intensive manual work
- Jamaican patois not listed in Common Crawl languages
Catalan Language Analysis
- Catalan: most spoken language in autonomous community of Spain
- GPT-3 training data composition:
- English words: 92%
- German words: 1.4%
- Spanish: 0.7%
- Catalan: 0.01% of entire training set
- Common Crawl reports Catalan as 0.2335% of survey
- Performance testing:
- GPT-3 generated 3 Catalan sentences mixed with real sentences
- 3 native speakers evaluated results
- Results were "very good for the machine"
- Key issues:
- Performance reasonably good but language-specific models still preferable
- Transparency problem: GPT-3 claims only ~140 pages of Catalan (novella-length)
- Dependency risk on few institutions/companies
- Example: Google cutting Catalan links in searches
Corporate Control and Transparency Issues
- Major AI companies control training data: Meta, Microsoft, Baidu, OpenAI, Google
- Companies decide which content included without disclosure of sources/authors
- Common Crawl represents only fraction of GPT-3 training data
- GPT-4 training details unknown
- Risk: companies could arbitrarily cut out languages
Open-Source Alternative: BLOOM Project
- Location: Paris supercomputer with underutilized GPU capacity
- Thomas Wolf: co-founder of Hugging Face (AI research hub)
- Big Science's BLOOM: open-source multilingual model project
- Approach:
- Started from Wikipedia list of most spoken languages
- Added low-resource languages when possible
- Focus on African languages for very low-resource inclusion
- Partnered with local communities to identify good data sources
- Transparent data sourcing and acquisition methods
- Key advantage: complete knowledge of data sources ("know the books in library")
Benefits for High-Resource Language Speakers
- English speakers are primary target audience (>40% Common Crawl)
- Personal example: Jamaican accent initially caused Siri comprehension issues
- Expanding training datasets to include more accents proved helpful for voice assistants
- Broader language representation benefits everyone through improved trust and accessibility
- Catalan speakers' perspective: "It's our language" - ownership and representation matter
Digital Footprint vs Real-World Usage Gap
- Languages may have sizable real-world speaker populations but minimal digital presence
- Risk: languages with small digital footprints "bound to just disappear" from AI systems
