AI summary by TwinMind
Lecture 04: Gathering and Collecting Data
MIT OpenCourseWare
1h 23m 53s
Course Overview and Plan
- Today's lecture lighter than previous two lectures
- Three main topics: where to find data, how to analyze it for distribution patterns, extracting insights beyond raw data
- Tomorrow will continue uncovered material, then return to probability (marginal/conditional distributions, functions of random variables)
Finding Existing Data Sources
- Much more data available now than during instructors' PhD/undergraduate years
- Three broad categories: existing data libraries, internet-based data collection, collecting own data
Academic and Government Data Sources
- MIT Libraries: Katherine McEl (social science librarian) helps find/purchase data, including GIS data
- Data.gov: Executive branch government data on tariffs, national parks protection, etc.
- IPUMS website: Census data (full old censuses, 1% extracts of recent ones), Current Population Survey (quarterly US population survey with wages/hours/jobs data), American Time Use Survey (time allocation data showing declining student study hours)
- International IPUMS: Censuses from many countries (Kenya example), started ~10 years ago, buying/cleaning censuses from statistical offices progressively
Data Repositories and Archives
- ICPSR (University of Michigan): First major data repository, researchers deposit cleaned datasets, excellent documentation/anonymization, uniform codebooks across datasets
- Harvard-MIT Data Center: Open to MIT community and general public, similar anonymization/distribution services
- Amazon Data Repository: Public data available with registration
- American Economic Association: Journals require posting research data unless confidential, available on AEA website
International and Specialized Sources
- Demographic and Health Surveys (DHS): Available for most developing countries, uniform questionnaires, demographic/health focus, asset data (not income), maternal/child health, education data, GPS coordinates available with permission
- World Bank Living Standard Measurement Surveys (LSMS): Rich household surveys in developing countries, 2-4 hour interviews, panels available in some countries, comprehensive consumption/earnings data
- RAND datasets: Household panels for Indonesia (98% followup rate over 20 years), Malaysia, Mexico with excellent followup
Modern Internet-Based Data Sources
- FiveThirtyEight: Polls and diverse topics, GitHub repository with story data, sports data (now ESPN-owned)
- Yahoo Data Dump: 13 terabytes of anonymous user interaction data on news feeds, largest existing dataset
- Open Source Sports: UK researchers' sports data aggregation
- NBA Stats: NBA-specific data collection
- Wayback Machine: Historical internet data storage
Data Access and Permissions
Restricted Access Data
- Some data free but requires agreements due to confidentiality
- Administrative data requires various permissions
- J-PAL North America provides tutorials for accessing administrative data
- Company partnerships (Facebook example) require data security compliance, human subjects approval, physical location restrictions
Freedom of Information Act (FOIA)
- Government-collected data should be publicly available unless confidential
- Applies to federal, state, local governments
- Cumbersome process requiring legal format letters
- Example: Drug Enforcement Administration data (Angela's thesis experience - extremely slow, don't recommend for class)
- International equivalents exist (India example)
Web Scraping Fundamentals
Basic Approaches
- Pull data from single page
- Crawl entire website
- Access background forms
- Ongoing collection for time series/panels
- Limitations: time-consuming, computational power requirements, website restrictions (Google Scholar example: 500 people/day limit, 6-month process)
Technical Implementation
- R options: XML library with readHTMLtable for simple tables, Rvest package (Beautiful Soup equivalent, unverified)
- Python approach: More conventional, extensive tutorials, requires Python + pip, Beautiful Soup library for pre-programmed scraping functions
Practical Example: Used Books Price Data
- Sarah and Glen's project: internet impact on used book prices
- Need panel data: same book titles, physical vs online prices over time
- Website inspection reveals price class tags for easy Beautiful Soup extraction
- Repeated collection with potential website blocking issues
Social Media APIs
- Twitter, Facebook have Application Program Interfaces (APIs)
- Require registration, user authorization for following
- APIs sometimes discontinued or poorly maintained
- Website scraping often more reliable than APIs
- Facebook data example: Sierra Leone social networks extracted via Facebook API
Collecting Original Data
Digital Collection Methods
- Survey tools like Survey Monkey
- Mobile apps for tracking (movement, commuting patterns)
- Experimental interventions with data collection
- Digital collection now cheaper than paper even in resource-poor settings
- Tablets cost-effective, eliminate double-entry/reconciliation problems
Data Collection Process Steps
- Obtain funding (may be zero for simple projects)
- Prepare data collection and management plan (encryption, safety protocols)
- Obtain human subjects approval (IRB/COUHES at MIT)
- Pilot data collection instrument
- Implement collection
Human Subjects Protection
- Research definition: Production of generalizable knowledge
- Three principles: Justice, informed consent/freedom, beneficence
- Firms vs researchers: Firms can do much more via user agreements, different regulations
- Facebook controversy: Company experiments vs research projects distinction
Data Visualization: Histograms
Definition and Construction
- Function counting observations in specified bins
- n (sample size) = sum of observations across all bins
- Can plot raw counts or densities (divided by total observations)
- Example: Female height data from J-PAL Bihar experiment
R Implementation
# Basic histogram
hist(data$height)
# Improved version
hist(data$height[data$height > 1.2 & data$height < 2],
breaks = 20,
xlab = "Height (meters)",
ylab = "Density",
col = "blue")
Bin Selection Impact
- Fewer bins: less precision, smoother appearance
- More bins: more precision, potentially choppy appearance
- Trade-off between detail and readability
Kernel Density Estimation
Concept and Motivation
- Nonparametric probability density function estimation
- Extension of histogram with continuous function
- Instead of vertical bars, uses weighted averages with kernel functions
- Creates smoother, more continuous representation
Technical Details
- Kernel choice: Bell-shaped functions preferred (Epanechnikov common), uniform possible but less smooth
- Bandwidth selection: Critical parameter determining smoothness vs detail capture
- Default in R: Uses conventional bandwidth selection algorithm
Bandwidth Trade-offs
- Too small: Jagged, high variance function
- Too large: May miss important data features, introduces bias
- Optimal bandwidth: Balances variance and bias through established recipes
- R's default generally suitable for most applications
