AI summary by TwinMind

Lecture 04: Gathering and Collecting Data

MIT OpenCourseWare

1h 23m 53s

Course Overview and Plan

  • Today's lecture lighter than previous two lectures
  • Three main topics: where to find data, how to analyze it for distribution patterns, extracting insights beyond raw data
  • Tomorrow will continue uncovered material, then return to probability (marginal/conditional distributions, functions of random variables)

Finding Existing Data Sources

  • Much more data available now than during instructors' PhD/undergraduate years
  • Three broad categories: existing data libraries, internet-based data collection, collecting own data

Academic and Government Data Sources

  • MIT Libraries: Katherine McEl (social science librarian) helps find/purchase data, including GIS data
  • Data.gov: Executive branch government data on tariffs, national parks protection, etc.
  • IPUMS website: Census data (full old censuses, 1% extracts of recent ones), Current Population Survey (quarterly US population survey with wages/hours/jobs data), American Time Use Survey (time allocation data showing declining student study hours)
  • International IPUMS: Censuses from many countries (Kenya example), started ~10 years ago, buying/cleaning censuses from statistical offices progressively

Data Repositories and Archives

  • ICPSR (University of Michigan): First major data repository, researchers deposit cleaned datasets, excellent documentation/anonymization, uniform codebooks across datasets
  • Harvard-MIT Data Center: Open to MIT community and general public, similar anonymization/distribution services
  • Amazon Data Repository: Public data available with registration
  • American Economic Association: Journals require posting research data unless confidential, available on AEA website

International and Specialized Sources

  • Demographic and Health Surveys (DHS): Available for most developing countries, uniform questionnaires, demographic/health focus, asset data (not income), maternal/child health, education data, GPS coordinates available with permission
  • World Bank Living Standard Measurement Surveys (LSMS): Rich household surveys in developing countries, 2-4 hour interviews, panels available in some countries, comprehensive consumption/earnings data
  • RAND datasets: Household panels for Indonesia (98% followup rate over 20 years), Malaysia, Mexico with excellent followup

Modern Internet-Based Data Sources

  • FiveThirtyEight: Polls and diverse topics, GitHub repository with story data, sports data (now ESPN-owned)
  • Yahoo Data Dump: 13 terabytes of anonymous user interaction data on news feeds, largest existing dataset
  • Open Source Sports: UK researchers' sports data aggregation
  • NBA Stats: NBA-specific data collection
  • Wayback Machine: Historical internet data storage

Data Access and Permissions

Restricted Access Data

  • Some data free but requires agreements due to confidentiality
  • Administrative data requires various permissions
  • J-PAL North America provides tutorials for accessing administrative data
  • Company partnerships (Facebook example) require data security compliance, human subjects approval, physical location restrictions

Freedom of Information Act (FOIA)

  • Government-collected data should be publicly available unless confidential
  • Applies to federal, state, local governments
  • Cumbersome process requiring legal format letters
  • Example: Drug Enforcement Administration data (Angela's thesis experience - extremely slow, don't recommend for class)
  • International equivalents exist (India example)

Web Scraping Fundamentals

Basic Approaches

  • Pull data from single page
  • Crawl entire website
  • Access background forms
  • Ongoing collection for time series/panels
  • Limitations: time-consuming, computational power requirements, website restrictions (Google Scholar example: 500 people/day limit, 6-month process)

Technical Implementation

  • R options: XML library with readHTMLtable for simple tables, Rvest package (Beautiful Soup equivalent, unverified)
  • Python approach: More conventional, extensive tutorials, requires Python + pip, Beautiful Soup library for pre-programmed scraping functions

Practical Example: Used Books Price Data

  • Sarah and Glen's project: internet impact on used book prices
  • Need panel data: same book titles, physical vs online prices over time
  • Website inspection reveals price class tags for easy Beautiful Soup extraction
  • Repeated collection with potential website blocking issues

Social Media APIs

  • Twitter, Facebook have Application Program Interfaces (APIs)
  • Require registration, user authorization for following
  • APIs sometimes discontinued or poorly maintained
  • Website scraping often more reliable than APIs
  • Facebook data example: Sierra Leone social networks extracted via Facebook API

Collecting Original Data

Digital Collection Methods

  • Survey tools like Survey Monkey
  • Mobile apps for tracking (movement, commuting patterns)
  • Experimental interventions with data collection
  • Digital collection now cheaper than paper even in resource-poor settings
  • Tablets cost-effective, eliminate double-entry/reconciliation problems

Data Collection Process Steps

  1. Obtain funding (may be zero for simple projects)
  2. Prepare data collection and management plan (encryption, safety protocols)
  3. Obtain human subjects approval (IRB/COUHES at MIT)
  4. Pilot data collection instrument
  5. Implement collection

Human Subjects Protection

  • Research definition: Production of generalizable knowledge
  • Three principles: Justice, informed consent/freedom, beneficence
  • Firms vs researchers: Firms can do much more via user agreements, different regulations
  • Facebook controversy: Company experiments vs research projects distinction

Data Visualization: Histograms

Definition and Construction

  • Function counting observations in specified bins
  • n (sample size) = sum of observations across all bins
  • Can plot raw counts or densities (divided by total observations)
  • Example: Female height data from J-PAL Bihar experiment

R Implementation

# Basic histogram
hist(data$height)

# Improved version
hist(data$height[data$height > 1.2 & data$height < 2], 
     breaks = 20, 
     xlab = "Height (meters)", 
     ylab = "Density",
     col = "blue")

Bin Selection Impact

  • Fewer bins: less precision, smoother appearance
  • More bins: more precision, potentially choppy appearance
  • Trade-off between detail and readability

Kernel Density Estimation

Concept and Motivation

  • Nonparametric probability density function estimation
  • Extension of histogram with continuous function
  • Instead of vertical bars, uses weighted averages with kernel functions
  • Creates smoother, more continuous representation

Technical Details

  • Kernel choice: Bell-shaped functions preferred (Epanechnikov common), uniform possible but less smooth
  • Bandwidth selection: Critical parameter determining smoothness vs detail capture
  • Default in R: Uses conventional bandwidth selection algorithm

Bandwidth Trade-offs

  • Too small: Jagged, high variance function
  • Too large: May miss important data features, introduces bias
  • Optimal bandwidth: Balances variance and bias through established recipes
  • R's default generally suitable for most applications

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome