Telephone polling, which arose in the 1970s, replaced in-person area probability sampling because in-person interviewing (door-to-door) was the dominant method from the 1930s until then.

Roughly 80-90% of polling errors come from sample skews rather than from sampling (random) error.

Random digit dialing became the dominant phone-polling method because roughly 30% of the population had unlisted numbers, making listing-based sampling infeasible; it worked well for 20-30 years but broke down in the last 10-15 years as marketing calls, cell phones, and distrust drove cooperation rates from ~70% down to ~20% or less, hurting accuracy.

Pollsters typically select 10 to 20 times as many phone numbers as the number of completed interviews they want, meaning the people who actually respond are a small, non-random slice that skews toward more women, higher education, and higher income, requiring weighting and adjustment.

A truly random sample of about 1,000 people gives a margin of error of about plus or minus 3% with 95% confidence regardless of total population size, so a tiny fraction (one in 300,000 Americans) suffices; quadrupling to ~4,000 tightens it to about 1% at 99% confidence.

The clean statistical results (law of large numbers and central limit theorem) only hold for perfectly executed random sampling; in practice sampling plans are rarely executed perfectly because you never get near 100% cooperation, and the non-responders differ systematically from responders in unobservable ways, producing skews rather than mere random noise.

Weighting underrepresented groups by large factors (5-10x for some groups) drastically inflates variability: with a weight of 10 in a sample of 1,000, a single miscoded respondent shifts the whole sample by 1%, so a single error can destroy a nominal 3% margin of error.

Weighting creates two distinct problems: it roughly doubles the real variability compared to a simple random sample, and it leaves residual skews (unobserved or uncorrected biases) that do NOT shrink as sample size grows, so conventional sampling-error estimates tell you how the procedure varies sample-to-sample but not how systematically wrong the procedure is.

In the 2008 New Hampshire primary, all roughly 30 pre-election polls failed to show Hillary Clinton ahead of Obama, which she won, indicating something systematically wrong affecting all polls rather than random chance.

Likely the New Hampshire 2008 polling miss came from over-representation of college- and graduate-degree voters (who favored Obama, over-represented by 2-3x and under-corrected) and from unreliable self-reports of voting intention, rather than from racism (the Bradley effect).

Self-reported voting intention is unreliable: about 90% of people say they will vote in a primary but actual turnout is much lower (US presidential turnout has hovered in the mid-50% range, congressional under 40%), and likely-voter screens cannot fully fix this because people answer based on what they think they should say.

Of about 1,300 polls Rivers examined from the 2008 presidential primaries, fewer than 50 reported anything other than the margin of error for a simple random sample with no weighting—meaning the reported margins of error are systematically misleading, which Rivers calls scandalous.

Typical media polls correct only for age, race, and gender (sometimes education, rarely income) using weighting methods about 60 years old, well behind modern statistical theory, partly from a long-standing belief that random digit dialing was good enough.

Final pre-election polls tend to be too similar to each other relative to their known sampling variability, suggesting pollsters herd—choosing among multiple defensible weighting schemes the one that places them in the pack rather than the one that yields a divergent answer—though Rivers attributes this to risk-aversion and the art of weighting rather than dishonesty.

There would seem to be an incentive for a pollster to trust divergent weights and look like a genius (as the lone correct caller of an upset), but in practice small subsamples (e.g. a NH poll with ~100 Republicans) make divergent results look like noise that gets discounted, which is why having 30 polls all wrong in New Hampshire was an extraordinary wake-up call.

The fraction of people with internet access at home, school, or work now probably exceeds the number with landline access once you exclude cell-phone-only households and those with no phone, making internet-based polling a viable alternative to telephone polling.

Massive consumer and voter databases that did not exist 25 years ago now provide detailed information (income, home value, etc.) on most people, allowing pollsters to draw a randomly selected target sample from a voter list and then match closely-similar respondents from a large opt-in internet panel, creating a sample that mimics a random sample across many dimensions and removes skews that simple demographic weighting cannot.

In 2006, the matched internet-panel method produced election forecasts whose average error was substantially less than the average reported telephone poll, mainly because it removed biases better (not because sampling variability was smaller), and Rivers argues telephone polling could be improved by similarly substituting a respondent who resembles the missed person rather than drawing another number from the same skewed population.

Interactive voice response (robocall) polls make up roughly 80-90% of polling done in 2008 because newspapers cut back on expensive live-interviewer polling, and despite low response rates and not knowing who is answering, their record is not bad—largely because IVR organizations pay closer attention to weighting than traditional phone pollsters.

Some organizations (Associated Press, New York Times) refuse to report polls using nonprobability sampling—respondents selected without known probabilities of selection—but Rivers argues they are in denial because their own low-response telephone polls also lack known selection probabilities.

Averaging polls (as RealClearPolitics does) does not necessarily wash out error: if pollsters make the same systematic errors or herd toward each other, the average is no more reliable than a single poll; averaging only clearly helps when polls use the same methodology, in which case combining them is equivalent to a larger sample, and ideally polls should be weighted by the reciprocal of their variability.

Weighting pollsters by their historical accuracy (as fivethirtyeight.com does) is questionable because past success may be partly random luck and because pollsters who erred likely changed their methodology afterward, so historical outcomes don't reliably predict future accuracy—analogous to picking mutual funds by past manager returns.

Apparent poll movements late in a campaign are often illusory: a methodology shift (e.g. Gallup switching from registered-voter to likely-voter weighting) creates a bump unrelated to real opinion change, and pollsters who err have a ready ex-post story of a late undetected surge—what Rivers calls the last refuge of scoundrels in polling—when in fact the same poll repeated would have produced the same wrong answer.

The Bradley/Wilder effect—the claim that voters tell pollsters they will vote for a black candidate but then don't—was a real overstatement of the black-candidate vote in the early 1980s but disappeared by around the late 1990s; a study Rivers cites found no systematic over-reporting across black congressional and Senate candidates and no comparable 'Whitman effect' for women candidates.

Experiments Rivers ran with Shanto Iyengar (lightening/darkening Obama's skin, and swapping black vs white candidate photos in a hypothetical congressional race) found essentially no candidate-race effect on voting, though about 15% of Democratic primary voters—even outside the South—held racial views considered conservative or racist.

Network exit polls (high-quality, ~50% response rate) have consistently overstated the Democratic share of the vote in presidential elections (2000, 2004), but because this bias appeared uniformly nationwide—including areas controlled by neither party—it indicates a systematic polling tendency, not the voter fraud that naive observers inferred.

The 2000 Florida election-night debacle—calling it for Gore, retracting, calling for Bush, retracting—was caused by a combination of erroneous data feeds, a data-entry error behind the Bush call, and an exit poll too close to have been called; since then networks have staffed decision desks with inside and outside experts and made no state-level miscall.

Calling a 46-40 poll lead (±3% margin of error) a 'statistical dead heat' or 'statistical tie' is incorrect: the best estimate is a 6-point lead, not zero, and while the 95% confidence standard means you can't formally declare a winner, it is far more likely the lead is real than that the true gap is zero, since extreme errors are unlikely.

The convention bounce—a candidate gaining several points (about five per party, net swing ~ten) during/after their party's convention—is partly an artifact of differential response: supporters of the convening party are energized and answer polls while opposition partisans, irritated by the 'phony love fest,' decline to participate, so YouGov controls for prior party identification to neutralize the effect.

Internet panel respondents are compensated with a reward they are not told in advance, and are not told what answers qualify them for a poll, in order to avoid biasing results; this contrasts with the phone tradition of not compensating, where rising marketing-call fatigue may eventually require incentives that introduce their own biases.

Party identification shifted dramatically: Democrats had an ~18-point lead from the New Deal through the 1960s, Republicans closed it to a few points by 2004, but 80% of those Republican gains were erased since 2004 (half of them in fall 2007/spring 2008), which in principle should make 2008 a terrible year for Republicans.

Obama's extraordinarily high favorability in fall 2007 faded by spring 2008 as Hillary Clinton ramped up rhetoric, leaving him looking like a run-of-the-mill (not weak) candidate without the magical appeal he had even among Democratic core constituencies; black Democrats, who initially didn't strongly favor him because of the Clintons' ties to blacks, swung to ~90% support once they recognized he was black.

Rivers agrees with Brian Caplan that the average voter couldn't pass Econ 101—reacting to problems by passing a law to ban them without thinking about consequences and blaming convenient targets—as illustrated by a YouGov poll where many people blamed 'speculators' for gas price increases despite probably not knowing what a speculator is, and a sample of Congress would likely respond similarly.

Welcome to EconTalk and the closing sign-off acknowledging the sound engineer.