THE EXIT POLL SCANDAL OF 2004
THE EXIT POLL SCANDAL OF 2004
Randy M. Mott1
@All rights reserved.
To many voters, watching the coverage of the 2004 election on television or over the Internet, the early mood of the evening would abruptly change as actual results begin to be posted. Many of us saw leaked exit polls that portrayed a nationwide Kerry lead of 3 percent and surprisingly strong initial estimates from virtually all of the background states. When the Bush campaign commented on the results, they expressed their view about their inaccuracy and inconsistency with the Bush campaign’s internal polling and tracking. The early actual vote counts began to dispel the exit polling leaks, but some networks hung on to the exit polling data to delay announcing state projections later in the night.
How could scientifically conducted exit polls, presumably sharpened in their methodology since the 2000 debacle [miscalling Florida for Gore] and the 2002 implosion [crash of the computer] be consistently wrong in one direction? The short answer is that they cannot be. A statistical sample of over 12,000 voters randomly selected by interviewing every tenth or twentieth person leaving carefully selected precinct could be subject to error, but the error would be random. My view is that the exit polling sample in the morning of the election, and perhaps in the second round of exit polling as well, was skewed toward Kerry voters. This was not a result of errors by the firms doing the polling, one of their principals literally wrote the book on exit polling, but was likely due to a systematic campaign to pack the exit polls by Democratic organizations.
This view has been suggested by both Dick Morris2 and Michael Barone.3
This article will review the background and purpose of exit polling, discuss how it was done in 2004 and summarize the evidence of irregularities that point to “poll packing,” including some scenarios that could explain the results.
Background of Exit Polling
“During the day the exit pollers collect the data from bellweather precincts. A single bellweather precinct can represent many other similar precincts.”4 Morton Halperin, ABC News Director, explained the uses the networks make of exit polling:
“They have basically two purposes: First of all, they serve to allow us to understand who voted on Election Day, how they voted, and why. It's an incredible experience for America every two years, going to the polls; we're able to tell who the electorate was made up of. It's a wonderful way to learn about America and to look at America on Election Day. The second purpose they serve is to allow us to project races, who the winners are in individual races, and that is something that the country likes and it's something the media likes, but it's not a gimmick; it's an important function for the press, a free press, to be able to tell people as quickly as possible who has won national elections. And the exit polls, by surveying voters as they leave the polls around the country, allow us to do that.” Lehrer News Hour, PBS, http://www.pbs.org/newshour/bb/media/july-dec02/vns_11-07.html.
“Exit poll results, the least accurate of the three, come in three times during the day,” Alicia C. Shepard , American Journalism Review online, Jan-Feb. 2001.5
History of Exit Polling
“After 1980, each network conducted its own exit polls on a massive scale. But costs mounted and network news executives, far more focused on the bottom line than in the past, scrambled to save money. In 1990 they created Voter Research & Surveys to conduct exit polls, offer analysis and make projections for all networks. Mitofsky was named to head the new operation. By joining forces, each network would save $9 million over a four-year period, according to a report cited in David W. Moore's book, "The Superpollsters." It also made VRS research affordable for CNN. Philip Meyer, a pioneer in what is sometimes called "precision journalism," spoke out against a network consortium at a 1991 meeting of the American Association for Public Opinion Research, calling it "a bad idea." If VRS makes a mistake, Meyer said then, there could be "terrible consequences" because there would be no other exit polls to serve as a counterweight. If VRS was wrong, he reasoned, everyone would be wrong.” 6
Going into the 2000 Presidential election, the networks and subscribers had formed the Voter News Service to serve this collective function. The use of exit polls in the 2000 election in Florida proved to be problematic at say the least. “In Florida, VNS-trained interviewers conducted scientific exit polls in 45 precincts with 4,356 people after they voted and staffed 120 sample precincts and 67 county election offices.”7
“As votes arrived from sample precincts carefully chosen to represent voters across the state, the model predicted a 5.4 percent lead for Gore. It indicated Gore needed a "critical value"--a statistical degree of certainty--of 2.6 or higher before any network could comfortably hand the vice president Florida. At 7:50 p.m., the "critical value" showed 3.2 for Gore. The Voter News Service model was more than 99.5 percent sure Gore would carry the state.” 8
As a result, very early in the evening to those of us eagerly watching the returns, the Florida race was called for Gore by all of the major netwroks. This actually occurred while voters were still in line in the Florida panhandle, a heavily Republican area. Subsequent analysis indicated that this discouraged about 7,500 voters from continuing to wait in line in Florida. The results were also felt in other tight races in Western time zones where the polls would be open for hours. Close races in several other states went Democratic with Republicans blaming the Florida miscall.9
As a result of the 2000 election, the networks went to back to improve the VNS methodology. The Associated Press had its own exit poll data in 2000 and was a key part of a new methodology in the 2002 Congressional election. “The networks blamed Voter News Service, the company they had formed to count votes and conduct exit polls, for faulty data that led to the wrong calls in 2000. VNS tried to rebuild its system, but it broke down on election night 2002 and failed to provide usable exit polling information. VNS was then disbanded.”10
"Determined to avoid a repeat of high-profile failures in 2000 and 2002, television networks will rely on new systems on Nov. 2 to help project election winners and analyze why voters made their choices. And they have turned to The Associated Press to count the vote for them. The six news organizations that have formed the National Election Pool - ABC, CBS, NBC, CNN, Fox News Channel and the AP - say they're confident things will go better this time, based on test runs and the experience of people involved." 11
The 2004 Exit Polling Approach
“This time, the news organizations contracted with two veteran polling companies - Mitofsky International and Edison Media Research - to conduct exit polls. They agreed that the AP - which has been tallying votes in elections since 1848 - would be their sole source for vote counts, and the news cooperative has significantly beefed up its system in response. Each of the organizations will use data provided by NEP to make its own projections election night. The organizations also have promised, for the first time in a presidential election, not to call states that span two time zones until all of the polling places have closed. One flaw exposed in 2000 - the failure of VNS to account for the increased use of absentee ballots - has been corrected, said Linda Mason of CBS News, an NEP spokeswoman. Telephone surveys of people voting by absentee ballots will be conducted in 13 states this year, instead of just three.” 12
The NEP organization started with efforts to be more transparent in its methodology , even posting a FAQ on the Internet. "The things that clearly went wrong four years ago, it's hard to imagine them going wrong again with what they've done with this system," said David Bohrman, CNN's Washington bureau chief.13 That statement was made before the eventful day of November 2, 2004.
The Initial Exit Polling Errors
Experts, including the architects of the 2004 methodology, are quick to point out that exit polling has its proper uses and its limitations. Its margin of error is about +/- 3 percent with all of the results tabulated. An early exit poll will normally only be +/- 7 percent.14 The final exit polls used to tell us why voters made their decisions and other demographics are adjusted after a review of the actual voting results. These final results are normally used to analyze the voters’ reasons for voting and the performance of the candidates among various demographic groups. Obviously, the final exit polls have significantly more reliability for their intended use.15
Early in the day of the election a Democratic blogger reported that: “Democratic base precincts are performing 15% higher than GOP base precincts.” (Daily Cos website). In Florida, he reported: “Dem base precincts are performing 14% better than Bush base precincts. In precincts that went for Gore, they are doing 6% better than those that went for Bush. African American precincts at 109%, Hispanic precincts at 106%.”16 Full initial results ended up being posted on the Drudge Report and Slate magazine online. The Daily Kos web site expressed the view widely shared at that point in the election: “Before anyone worries that we may be complacent, consider this -- if these turnout numbers check out nationally, even to much smaller extent, we could be looking at serious gains in the House and Senate.”17 Other bloggers seized on comments by GOP pollster Frank Luntz interpreting the exit polls and predicting a big Kerry win. These comments were taken, allegedly, from an internal memorandum in Luntz’ firm.18
“Meanwhile, there were concerns among television executives about early exit polls that indicated John Kerry would do much better than he appeared to be faring as actual vote counts came in,” according to David Bauder of the Associated Press, supra. “At Fox News during the afternoon and early evening we assumed, based on the exit poll results, that Kerry was overwhelmingly likely to win.”19 Actually, “the national survey showed the senator with a statistically significant lead, one falling outside the survey's margin of error.” New York Times, November 5, 2004.
“The early polls seemed to verify the anticipated surge in the number of newly registered 18- to 29-year-old voters casting ballots and heavy turnout among women, key ingredients in Kerry's recipe for winning the White House.”20 “The heavy morning voting in Democratic areas in key states is also spooking the pollsters; traditionally the morning vote trends Republican because that is when white color workers vote; the late afternoon and evening voting has traditionally been more Democratic because that is when blue collar voters come out to vote.” Michael Frisby, Democratic political consultant, email to clients on election day.21
However, by the time the polls closed and the early actual results were being posted by state elections commissions, the voting had a totally different pattern than the exit polls had indicated. “"There were a couple of the initial tranches that were way out of line with the final results," noted U.S. News expert Michael Barone.23 Charles Gibson on ABC's "Good Morning America " was more blunt: “The exit polls got it flat wrong." The nationwide actual voting results that remained consistent all evening and into the next day had Bush up 51% to Kerry’s 48%. The initial exit poll data that was leaked had it exactly reversed, with Kerry nationally up by 3%."24 In most of the swing states or battleground states, the exit polls had early Kerry leads that invariably vanished once the actual early results were posted. “The exit polls missed the mark very badly last night (before they were reweighted to correspond to the actual results). The national exit poll consistently showed Kerry leading by 3 points--just the reverse of the actual vote. The Ohio exit poll had Kerry up by 4 and the Florida exit poll had it tied,” noted Alan Abramowitz, professor of political science at Emory University.25
“Republicans are fuming again over erroneous exit polls that showed President Bush losing re-election and over television networks quickly calling some states for Sen. John Kerry while withholding such predictions for solid Bush states. Although the Associated Press-led polling consortium was eventually proven wrong by actual hard tallies, the widely distributed exit polls prompted a number of TV pundits to talk on election night of how Mr. Bush likely had lost. The Associated Press and TV networks do not publicly release the spreads, but the numbers leaked out to numerous Internet sites.” 26
In perhaps one of the great understatements of the 2004 election, the Associated Press writer commented: ““Although no major problems in the new systems were reported, the early exit polls caused concern.” Bauder, supra. Of course, AP was a part of the NEP effort. Pool participants initially downplayed the problems: . "I think it would be premature to say that we had any substantial problems," explained said Bill Wheatley, NBC News vice president.26 The early “cover-up” statements are sharply contradicted by the internal NEP report on their exit polling which noted that “the surveys had the biggest partisan skew since at least 1988, the earliest election the report tracked.” New York Times, November 5, 2004.
What Really Happened
I will leave it to the thorough and dispassionate analysts to assess the damage done to the integrity of the election by the early exit poll data. “Certainly if a race is declared over based on exit polls while voters are still going to the polls that could discourage voting.” Andrew Kohut, president of the Pew Research Center.27 “These incredibly incorrect results seemed to have been used to spread rumors of a wild Kerry blowout, starting around noon yesterday, designed to influence voters who had yet to cast their ballots.”28 Rowan Scarborough of the Washington Times noted that “the widely distributed exit polls prompted a number of TV pundits to talk on election night of how Mr. Bush likely had lost.” Even NEP admits in its post-election analysis that “"[w]e made a mistake in not realizing the full impact of these leaked exit poll numbers on the political discourse of the day.'' New York Times, November 5, 2004.
I do not believe for a minute that it is fair to blame the Internet bloggers for posting the results, an allegation now made by NEP. It is inevitable that the initial exit polls will be leaked and that they will be used by commentators in the media in their running analysis on election night. Susan Estrich overtly referred to them in her election comments that night. Other commentators were clearly looking at them as they prognosticated on the election’s direction.
Explanations of the exit polling errors have covered the spectrum: “Some combination of bad precinct samples, response bias, or failing to accurately account for early and absentee votes must have been at work. Whatever it was, it was a major problem. In 2000, the national exit poll also overestimated Gore's vote, but not by nearly as big a margin,’ noted Alan Abramowitz, professor of political science at Emory University.29 NEP participators and managers argue that “…results based on the first few rounds of interviewing are usually only approximations of the final vote. Printouts warn that estimates of each candidate's support are unreliable and not for on-air use.”30 As I noted earlier, the normal margin of error in initial samples from exit polls in +/- 7%31 and here the error was 6%, but completely in Kerry’s direction and replicated in all the swing states. “The sample size was so large that it appeared beyond the usual margins of error in polling. Yet the poll was dead wrong. How can this be? You can be sure we'll be asking that question pointedly in the days ahead, as well as producing stories for readers about the issues.” Steve Coll, managing editor, Washington Post, November 3, 2004. He added that the initial exit polls “are out of whack with voting results in ways that are difficult to explain.”
The problem of the consistent direction of the errors in the initial exit polls lead Dick Morris to speculate publicly about the reasons:
"It takes a deliberate act of fraud and bias to get an exit poll wrong. Since the variables of whether or not a person will actually vote are eliminated in exit polling, it is like peeking at the answer before taking the test. The fact that they were so totally, disastrously wrong is a national scandal." 32
Michael Barone, one of the deans of American politics, has the same theory:
“My own suspicion is that some Democrats—at the command level, or somewhere below—had an election-day project of slamming the results. New Hampshire, Minnesota and Pennsylvania initial exit poll results had huge margins for Kerry—much larger percentages than he won in any pre-election poll. If somebody had slipped some Democratic operative the list of exit poll sites—40 to 50 sites in each critical state—he or she could have slipped several hundred operatives into the polling places to take the exit poll ballots and vote for Kerry. The results would have shown Kerry much farther ahead than he actually was and, broadcast through drugdereport.com and other sources, could have heartened Kerry supporters during the afternoon and disheartened Bush supporters.” 33
Several things jump out from the facts. Exit polls have had a sorry history in the last several elections. Given the role that they played in 2000 for Gore, it is clear that any campaign organization knows that they can be important in influencing the final vote.34
The fact that the exit polls consistently showed an erroneous Kerry margin is also very significant. “The random variance that occurs due to sampling would show itself by being off in one direction in some places and off in the other direction in others. The fact that they were all off in the same direction (much like Zogby’s polls) does not indicate that they just got a bum hand from the random variance Gods.”35 Even NEP’s internal report seems to admit that “the problems with the data seemed seriously exacerbated this year.” New York Times, November 5, 2004.
The major weakness in an exit poll is the sampling methodology. "This is the only parameter the polling organization can clearly control," according to Fritz Scheuren, professor of statistics at George Washington University.36 Another expert noted that “many pollsters complain that half the people they approach decline to be interviewed, which can skew the results.” Joan Konner, professor and dean emeritus of the Columbia University Graduate School of Journalism. Konner helped write a 2001 report commissioned by CNN and CBS that took the networks to task for their performance on election night 2000.37 “Even response rates for exit polls on election day have dropped to 50%.” Id. This means that self-selection can be a powerful bias in exit polls. If half or more of the voters decline to be interviewed, then those that volunteer to be interviewed can be manipulated to skew the sample. If Democratic operatives bussed in voters with instructions to find exit pollsters and be interviewed, they could theoretically skew the whole sample. This factor is suggested by NEP itself as one that could explain their problems:
“The [NEP] report also theorized that the poll results more frequently overstated support for Mr. Kerry than for President Bush because the Democratic nominee's supporters were more open to pollsters.” 38
The results could, of course, be explained by a more sinister plan, as Michael Barone noted: “If somebody had slipped some Democratic operative the list of exit poll sites—40 to 50 sites in each critical state—he or she could have slipped several hundred operatives into the polling places to take the exit poll ballots and vote for Kerry,” supra. It is also possible that Democratic activists signed up as exit poll takers, which is a part-time job and sought out Kerry voters. These scenarios seem unlikely until one considers the numerous thefts of GOTV voter rolls and other information from GOP campaign office this year, the slashing of tires on Republican GOTV vehicles on election day, and other criminal acts that can only be explained as election-oriented crime. “The Prowler" column in The American Spectator noted: “The early polling numbers are some of the most eagerly anticipated, if highly inaccurate, data on election day, and are widely distributed. Perhaps that was what the Kerry campaign was banking on.”39
Another factor that suggests tampering with the sample was mentioned in passing early here. The early exit poll sampling was heavily women and young people. This is inconsistent with normal anticipated demographics, as is the heavily Democratic content of the sample.40
The only explanation for errors in the exit polling in 2004 is “sample error.” The fact that the errors were unduly large and all in one direction belies attributing this to random statistical problems. I will be interested in seeing an eventual, professional statistical analysis of the probability of all of the errors running in Kerry’s favor. The other dirty tricks in the election and efforts by Democratic sympathizers seem to add to the likelihood that tampering with the polling was a deliberate act.
Whether Democratic voters bussed in early to the polls were simply told to look for exit pollsters to be interviewed or whether the organizations involved had actual polling locations and times or whether exit pol takers were "infiltrated" will undoubtedly be determined in the coming months. It does seem possible that exit polling will no longer be used for predicting elections if this phenomenon continues. "If they really wanted to do it right, they would just report the results," said Curtis Gans, vice president and director of the Committee for the Study of the American Electorate in Washington.41 He said that in 2003 and added : "They're refining their technologies so they can get it wrong again in 2004."
__________________________________________
Footnotes:
1. Mr. Mott was a practicing attorney in litigation in Washington, DC for 28 years, including some election law matters. He now lives and works in Warsaw, Poland.
2. "It takes a deliberate act of fraud and bias to get an exit poll wrong. Since the variables of whether or not a person will actually vote are eliminated in exit polling, it is like peeking at the answer before taking the test. The fact that they were so totally, disastrously wrong is a national scandal," Dick Morris, New York Post, November 4, 2004. http://www.nypost.com/postopinion/opedcolumnists/31590.htm:
3. “Why were the initial exit poll results more Democratic than the actual tabulated vote? No one is sure, though the national sample at midafternoon, which showed Kerry ahead 50 to 49 percent, was 58 percent women. My own suspicion is that some Democrats—at the command level, or somewhere below—had an election-day project of slamming the results. New Hampshire, Minnesota and Pennsylvania initial exit poll results had huge margins for Kerry—much larger percentages than he won in any pre-election poll. If somebody had slipped some Democratic operative the list of exit poll sites—40 to 50 sites in each critical state—he or she could have slipped several hundred operatives into the polling places to take the exit poll ballots and vote for Kerry. The results would have shown Kerry much farther ahead than he actually was and, broadcast through drugdereport.com and other sources, could have heartened Kerry supporters during the afternoon and disheartened Bush supporters. When I was active in Democratic politics, in 1964-80, it would have occurred to us to do no such thing. But Democrats these days are so filled with a sense of grievance and with a feeling of justification for employing any dirty tactics to win, that this is not unthinkable.” Michael Barone, http://www.usnews.com/usnews/opinion/baroneweb/mb_041103.htm.
4. http://www.freerepublic.com/forum/a3a096d852f70.htm
5. http://www.ajr.org/Article.asp?id=519.
6. Alicia C. Shepard , American Journalism Review online, Jan-Feb. 2001, http://www.ajr.org/Article.asp?id=519
7. Id.
8. Id.
9. “At VNS' temporary quarters on the 93rd floor of Manhattan's World Trade Center, two political scientists working for ABC, each with a strong statistical background, didn't think the Florida result was clear-cut. Nor did they completely trust the VNS model. When the decision desk telephoned the two analysts asking, "Can we make the call?" both men advised against awarding Florida to Gore.” Shepard, supra. Shepard’s article contains a treasure trove of contemporary expert opinions from that election and is worth reading for that alone. “TV's elections analysts with their sample precincts and illuminating exit polls had projected winners in thousands of races over the decades -- congressional, gubernatorial, presidential -- and been wrong in only a tiny fraction of them. Suddenly, on November 7, those systems melted down, leaving party activists, newspaper journalists, and the public demanding to know how the TV people had initiated such chaos.” Neil Hickey,”The Big Mistake,”Columbia Journalism Review, online, Jan-Feb 2001, http://www.cjr.org/archives.asp?url=/01/1/hickey.asp.
10. David Bauder, Associated Press, APOL News Service, http://aolsvc.news.aol.com/elections/article.adp?id=20041013122209990005&_ccc=2&cid=946
11. Bauder, supra.
12. Bauder, supra.
13. Bauder, supra.
14. “…the mid-day leaked numbers would have much greater error, perhaps +/- 7% or more.” Mark Blumenthal, Democratic pollster, http://www.mysterypollster.com/main/2004/11/exit_polls_what.html
15. One of the NEP architects (Lenski) quoted in “New Woes Surface in Use of Estimates,” Richard Morin, Washington Post Staff Writer, Thursday, November 4, 2004; Page A29
16. http://www.dailykos.com/story/2004/11/2/161046/605.
17. http://www.dailykos.com/story/2004/11/2/161046/605.
18. The Wonkette website, http://www.wonkette.com/archives/exit-polls-from-gop-pollster-luntz-very-kerry-024805.php.
19. Michael Barone :http://www.usnews.com/usnews/opinion/baroneweb/mb_041103.htm.
20. Seattle Times , “How did exit polls start avalanche of inaccuracy?,” Nov. 4, 2004, http://seattletimes.nwsource.com/html/politics/2002081763_exitpolls04.html.
21. Quoted in the American Spectator magazine online: http://www.spectator.org/dsp_article.asp?art_id=7346.
22. Quoted in the Washington Times, November 4, 2004.
23. Steve Coll, managing editor, Wash. Post, November 3, 2004, http://www.washingtonpost.com/wp-dyn/articles/A13590-2004Oct31.html
24. http://www.emergingdemocraticmajorityweblog.com/donkeyrising/abramowitz.php
25. Quoted by AP’s Bauder, surpa.
26. Rowan Scarborough, Washington times, November 4, 2004.
27. http://www.npr.org/chat/10202004.html.
28. Penracker web blog, http://www.penraker.com.
29. http://www.emergingdemocraticmajorityweblog.com/donkeyrising/abramowitz.php.
30. Richard Morin “New Woes Surface in Use of Estimates,” Richard Morin
Washington Post, Thursday, November 4, 2004; Page A29.
31. “…the mid-day leaked numbers would have much greater error, perhaps +/- 7% or more.” Mark Blumenthal, Democratic pollster, http://www.mysterypollster.com/main/2004/11/exit_polls_what.html
32. Dick Morris, New York Post column of November 3, 2004, http://www.nypost.com/postopinion/opedcolumnists/31590.htm.
33. http://www.usnews.com/usnews/opinion/baroneweb/mb_041103.htm
34. Kerry had the same problem in the Wisconsin primary against Edwards, when the early exit polling data showed him with a much smaller lead than anticipated. Feb. 21, 2004, “Blame talking heads, not the exit polls.” Cragg Hines, Houston Chronicle: http://www.chron.com/cs/CDA/story.hts/editorial/2413157
35. Daly Thoughts web blog, http://www.dalythoughts.com.
36. http://whyfiles.org/009poll/poll_principles2.html. See Merkle, Daniel M. (2000). and Murray Edelman. "Nonresponse in Exit Polls: A Comprehensive Analysis." In Election Polls, the News Media and Democracy, Paul J. Lavrakas and Michael Traugott (eds.), Chatham, N.J.: Chatham House Publishers.
37. http://www.electoral-vote.com
38. Rutenburg, New York Times, November 5, 2004.
39. The Prowler, American Spectator online, November 4, 2004. The Prowler goes on to mention several “leaks” of wildly inaccurate numbers, showing far greater disparities in the voting in Pennsylvania and elsewhere, suggesting that these were a “dirty trick” by Democrats. This could be true, but it would not explain the NEP numbers, which were also inaccurate but not comically so, as were some of the Internet posts that night. http://www.spectator.org/dsp_article.asp?art_id=7346.
40. Democratic analysts and consultants make this point themselves: "Exits come out in a several batches over the course of the day. Democrats, on average, tend to vote later in the day than Republicans. Not always, but that's the pattern, for fairly straightforward demographic reasons. And for that reason their exit poll numbers tend to get better over the course of the day." Josn Marshall, http://www.talkingpointsmemo.com/archives/week_ 2004_10_31.php #003906. See Blumenthal, supra, note 31.
41. http://www.electoral-vote.com
Randy M. Mott1
@All rights reserved.
To many voters, watching the coverage of the 2004 election on television or over the Internet, the early mood of the evening would abruptly change as actual results begin to be posted. Many of us saw leaked exit polls that portrayed a nationwide Kerry lead of 3 percent and surprisingly strong initial estimates from virtually all of the background states. When the Bush campaign commented on the results, they expressed their view about their inaccuracy and inconsistency with the Bush campaign’s internal polling and tracking. The early actual vote counts began to dispel the exit polling leaks, but some networks hung on to the exit polling data to delay announcing state projections later in the night.
How could scientifically conducted exit polls, presumably sharpened in their methodology since the 2000 debacle [miscalling Florida for Gore] and the 2002 implosion [crash of the computer] be consistently wrong in one direction? The short answer is that they cannot be. A statistical sample of over 12,000 voters randomly selected by interviewing every tenth or twentieth person leaving carefully selected precinct could be subject to error, but the error would be random. My view is that the exit polling sample in the morning of the election, and perhaps in the second round of exit polling as well, was skewed toward Kerry voters. This was not a result of errors by the firms doing the polling, one of their principals literally wrote the book on exit polling, but was likely due to a systematic campaign to pack the exit polls by Democratic organizations.
This view has been suggested by both Dick Morris2 and Michael Barone.3
This article will review the background and purpose of exit polling, discuss how it was done in 2004 and summarize the evidence of irregularities that point to “poll packing,” including some scenarios that could explain the results.
Background of Exit Polling
“During the day the exit pollers collect the data from bellweather precincts. A single bellweather precinct can represent many other similar precincts.”4 Morton Halperin, ABC News Director, explained the uses the networks make of exit polling:
“They have basically two purposes: First of all, they serve to allow us to understand who voted on Election Day, how they voted, and why. It's an incredible experience for America every two years, going to the polls; we're able to tell who the electorate was made up of. It's a wonderful way to learn about America and to look at America on Election Day. The second purpose they serve is to allow us to project races, who the winners are in individual races, and that is something that the country likes and it's something the media likes, but it's not a gimmick; it's an important function for the press, a free press, to be able to tell people as quickly as possible who has won national elections. And the exit polls, by surveying voters as they leave the polls around the country, allow us to do that.” Lehrer News Hour, PBS, http://www.pbs.org/newshour/bb/media/july-dec02/vns_11-07.html.
“Exit poll results, the least accurate of the three, come in three times during the day,” Alicia C. Shepard , American Journalism Review online, Jan-Feb. 2001.5
History of Exit Polling
“After 1980, each network conducted its own exit polls on a massive scale. But costs mounted and network news executives, far more focused on the bottom line than in the past, scrambled to save money. In 1990 they created Voter Research & Surveys to conduct exit polls, offer analysis and make projections for all networks. Mitofsky was named to head the new operation. By joining forces, each network would save $9 million over a four-year period, according to a report cited in David W. Moore's book, "The Superpollsters." It also made VRS research affordable for CNN. Philip Meyer, a pioneer in what is sometimes called "precision journalism," spoke out against a network consortium at a 1991 meeting of the American Association for Public Opinion Research, calling it "a bad idea." If VRS makes a mistake, Meyer said then, there could be "terrible consequences" because there would be no other exit polls to serve as a counterweight. If VRS was wrong, he reasoned, everyone would be wrong.” 6
Going into the 2000 Presidential election, the networks and subscribers had formed the Voter News Service to serve this collective function. The use of exit polls in the 2000 election in Florida proved to be problematic at say the least. “In Florida, VNS-trained interviewers conducted scientific exit polls in 45 precincts with 4,356 people after they voted and staffed 120 sample precincts and 67 county election offices.”7
“As votes arrived from sample precincts carefully chosen to represent voters across the state, the model predicted a 5.4 percent lead for Gore. It indicated Gore needed a "critical value"--a statistical degree of certainty--of 2.6 or higher before any network could comfortably hand the vice president Florida. At 7:50 p.m., the "critical value" showed 3.2 for Gore. The Voter News Service model was more than 99.5 percent sure Gore would carry the state.” 8
As a result, very early in the evening to those of us eagerly watching the returns, the Florida race was called for Gore by all of the major netwroks. This actually occurred while voters were still in line in the Florida panhandle, a heavily Republican area. Subsequent analysis indicated that this discouraged about 7,500 voters from continuing to wait in line in Florida. The results were also felt in other tight races in Western time zones where the polls would be open for hours. Close races in several other states went Democratic with Republicans blaming the Florida miscall.9
As a result of the 2000 election, the networks went to back to improve the VNS methodology. The Associated Press had its own exit poll data in 2000 and was a key part of a new methodology in the 2002 Congressional election. “The networks blamed Voter News Service, the company they had formed to count votes and conduct exit polls, for faulty data that led to the wrong calls in 2000. VNS tried to rebuild its system, but it broke down on election night 2002 and failed to provide usable exit polling information. VNS was then disbanded.”10
"Determined to avoid a repeat of high-profile failures in 2000 and 2002, television networks will rely on new systems on Nov. 2 to help project election winners and analyze why voters made their choices. And they have turned to The Associated Press to count the vote for them. The six news organizations that have formed the National Election Pool - ABC, CBS, NBC, CNN, Fox News Channel and the AP - say they're confident things will go better this time, based on test runs and the experience of people involved." 11
The 2004 Exit Polling Approach
“This time, the news organizations contracted with two veteran polling companies - Mitofsky International and Edison Media Research - to conduct exit polls. They agreed that the AP - which has been tallying votes in elections since 1848 - would be their sole source for vote counts, and the news cooperative has significantly beefed up its system in response. Each of the organizations will use data provided by NEP to make its own projections election night. The organizations also have promised, for the first time in a presidential election, not to call states that span two time zones until all of the polling places have closed. One flaw exposed in 2000 - the failure of VNS to account for the increased use of absentee ballots - has been corrected, said Linda Mason of CBS News, an NEP spokeswoman. Telephone surveys of people voting by absentee ballots will be conducted in 13 states this year, instead of just three.” 12
The NEP organization started with efforts to be more transparent in its methodology , even posting a FAQ on the Internet. "The things that clearly went wrong four years ago, it's hard to imagine them going wrong again with what they've done with this system," said David Bohrman, CNN's Washington bureau chief.13 That statement was made before the eventful day of November 2, 2004.
The Initial Exit Polling Errors
Experts, including the architects of the 2004 methodology, are quick to point out that exit polling has its proper uses and its limitations. Its margin of error is about +/- 3 percent with all of the results tabulated. An early exit poll will normally only be +/- 7 percent.14 The final exit polls used to tell us why voters made their decisions and other demographics are adjusted after a review of the actual voting results. These final results are normally used to analyze the voters’ reasons for voting and the performance of the candidates among various demographic groups. Obviously, the final exit polls have significantly more reliability for their intended use.15
Early in the day of the election a Democratic blogger reported that: “Democratic base precincts are performing 15% higher than GOP base precincts.” (Daily Cos website). In Florida, he reported: “Dem base precincts are performing 14% better than Bush base precincts. In precincts that went for Gore, they are doing 6% better than those that went for Bush. African American precincts at 109%, Hispanic precincts at 106%.”16 Full initial results ended up being posted on the Drudge Report and Slate magazine online. The Daily Kos web site expressed the view widely shared at that point in the election: “Before anyone worries that we may be complacent, consider this -- if these turnout numbers check out nationally, even to much smaller extent, we could be looking at serious gains in the House and Senate.”17 Other bloggers seized on comments by GOP pollster Frank Luntz interpreting the exit polls and predicting a big Kerry win. These comments were taken, allegedly, from an internal memorandum in Luntz’ firm.18
“Meanwhile, there were concerns among television executives about early exit polls that indicated John Kerry would do much better than he appeared to be faring as actual vote counts came in,” according to David Bauder of the Associated Press, supra. “At Fox News during the afternoon and early evening we assumed, based on the exit poll results, that Kerry was overwhelmingly likely to win.”19 Actually, “the national survey showed the senator with a statistically significant lead, one falling outside the survey's margin of error.” New York Times, November 5, 2004.
“The early polls seemed to verify the anticipated surge in the number of newly registered 18- to 29-year-old voters casting ballots and heavy turnout among women, key ingredients in Kerry's recipe for winning the White House.”20 “The heavy morning voting in Democratic areas in key states is also spooking the pollsters; traditionally the morning vote trends Republican because that is when white color workers vote; the late afternoon and evening voting has traditionally been more Democratic because that is when blue collar voters come out to vote.” Michael Frisby, Democratic political consultant, email to clients on election day.21
However, by the time the polls closed and the early actual results were being posted by state elections commissions, the voting had a totally different pattern than the exit polls had indicated. “"There were a couple of the initial tranches that were way out of line with the final results," noted U.S. News expert Michael Barone.23 Charles Gibson on ABC's "Good Morning America " was more blunt: “The exit polls got it flat wrong." The nationwide actual voting results that remained consistent all evening and into the next day had Bush up 51% to Kerry’s 48%. The initial exit poll data that was leaked had it exactly reversed, with Kerry nationally up by 3%."24 In most of the swing states or battleground states, the exit polls had early Kerry leads that invariably vanished once the actual early results were posted. “The exit polls missed the mark very badly last night (before they were reweighted to correspond to the actual results). The national exit poll consistently showed Kerry leading by 3 points--just the reverse of the actual vote. The Ohio exit poll had Kerry up by 4 and the Florida exit poll had it tied,” noted Alan Abramowitz, professor of political science at Emory University.25
“Republicans are fuming again over erroneous exit polls that showed President Bush losing re-election and over television networks quickly calling some states for Sen. John Kerry while withholding such predictions for solid Bush states. Although the Associated Press-led polling consortium was eventually proven wrong by actual hard tallies, the widely distributed exit polls prompted a number of TV pundits to talk on election night of how Mr. Bush likely had lost. The Associated Press and TV networks do not publicly release the spreads, but the numbers leaked out to numerous Internet sites.” 26
In perhaps one of the great understatements of the 2004 election, the Associated Press writer commented: ““Although no major problems in the new systems were reported, the early exit polls caused concern.” Bauder, supra. Of course, AP was a part of the NEP effort. Pool participants initially downplayed the problems: . "I think it would be premature to say that we had any substantial problems," explained said Bill Wheatley, NBC News vice president.26 The early “cover-up” statements are sharply contradicted by the internal NEP report on their exit polling which noted that “the surveys had the biggest partisan skew since at least 1988, the earliest election the report tracked.” New York Times, November 5, 2004.
What Really Happened
I will leave it to the thorough and dispassionate analysts to assess the damage done to the integrity of the election by the early exit poll data. “Certainly if a race is declared over based on exit polls while voters are still going to the polls that could discourage voting.” Andrew Kohut, president of the Pew Research Center.27 “These incredibly incorrect results seemed to have been used to spread rumors of a wild Kerry blowout, starting around noon yesterday, designed to influence voters who had yet to cast their ballots.”28 Rowan Scarborough of the Washington Times noted that “the widely distributed exit polls prompted a number of TV pundits to talk on election night of how Mr. Bush likely had lost.” Even NEP admits in its post-election analysis that “"[w]e made a mistake in not realizing the full impact of these leaked exit poll numbers on the political discourse of the day.'' New York Times, November 5, 2004.
I do not believe for a minute that it is fair to blame the Internet bloggers for posting the results, an allegation now made by NEP. It is inevitable that the initial exit polls will be leaked and that they will be used by commentators in the media in their running analysis on election night. Susan Estrich overtly referred to them in her election comments that night. Other commentators were clearly looking at them as they prognosticated on the election’s direction.
Explanations of the exit polling errors have covered the spectrum: “Some combination of bad precinct samples, response bias, or failing to accurately account for early and absentee votes must have been at work. Whatever it was, it was a major problem. In 2000, the national exit poll also overestimated Gore's vote, but not by nearly as big a margin,’ noted Alan Abramowitz, professor of political science at Emory University.29 NEP participators and managers argue that “…results based on the first few rounds of interviewing are usually only approximations of the final vote. Printouts warn that estimates of each candidate's support are unreliable and not for on-air use.”30 As I noted earlier, the normal margin of error in initial samples from exit polls in +/- 7%31 and here the error was 6%, but completely in Kerry’s direction and replicated in all the swing states. “The sample size was so large that it appeared beyond the usual margins of error in polling. Yet the poll was dead wrong. How can this be? You can be sure we'll be asking that question pointedly in the days ahead, as well as producing stories for readers about the issues.” Steve Coll, managing editor, Washington Post, November 3, 2004. He added that the initial exit polls “are out of whack with voting results in ways that are difficult to explain.”
The problem of the consistent direction of the errors in the initial exit polls lead Dick Morris to speculate publicly about the reasons:
"It takes a deliberate act of fraud and bias to get an exit poll wrong. Since the variables of whether or not a person will actually vote are eliminated in exit polling, it is like peeking at the answer before taking the test. The fact that they were so totally, disastrously wrong is a national scandal." 32
Michael Barone, one of the deans of American politics, has the same theory:
“My own suspicion is that some Democrats—at the command level, or somewhere below—had an election-day project of slamming the results. New Hampshire, Minnesota and Pennsylvania initial exit poll results had huge margins for Kerry—much larger percentages than he won in any pre-election poll. If somebody had slipped some Democratic operative the list of exit poll sites—40 to 50 sites in each critical state—he or she could have slipped several hundred operatives into the polling places to take the exit poll ballots and vote for Kerry. The results would have shown Kerry much farther ahead than he actually was and, broadcast through drugdereport.com and other sources, could have heartened Kerry supporters during the afternoon and disheartened Bush supporters.” 33
Several things jump out from the facts. Exit polls have had a sorry history in the last several elections. Given the role that they played in 2000 for Gore, it is clear that any campaign organization knows that they can be important in influencing the final vote.34
The fact that the exit polls consistently showed an erroneous Kerry margin is also very significant. “The random variance that occurs due to sampling would show itself by being off in one direction in some places and off in the other direction in others. The fact that they were all off in the same direction (much like Zogby’s polls) does not indicate that they just got a bum hand from the random variance Gods.”35 Even NEP’s internal report seems to admit that “the problems with the data seemed seriously exacerbated this year.” New York Times, November 5, 2004.
The major weakness in an exit poll is the sampling methodology. "This is the only parameter the polling organization can clearly control," according to Fritz Scheuren, professor of statistics at George Washington University.36 Another expert noted that “many pollsters complain that half the people they approach decline to be interviewed, which can skew the results.” Joan Konner, professor and dean emeritus of the Columbia University Graduate School of Journalism. Konner helped write a 2001 report commissioned by CNN and CBS that took the networks to task for their performance on election night 2000.37 “Even response rates for exit polls on election day have dropped to 50%.” Id. This means that self-selection can be a powerful bias in exit polls. If half or more of the voters decline to be interviewed, then those that volunteer to be interviewed can be manipulated to skew the sample. If Democratic operatives bussed in voters with instructions to find exit pollsters and be interviewed, they could theoretically skew the whole sample. This factor is suggested by NEP itself as one that could explain their problems:
“The [NEP] report also theorized that the poll results more frequently overstated support for Mr. Kerry than for President Bush because the Democratic nominee's supporters were more open to pollsters.” 38
The results could, of course, be explained by a more sinister plan, as Michael Barone noted: “If somebody had slipped some Democratic operative the list of exit poll sites—40 to 50 sites in each critical state—he or she could have slipped several hundred operatives into the polling places to take the exit poll ballots and vote for Kerry,” supra. It is also possible that Democratic activists signed up as exit poll takers, which is a part-time job and sought out Kerry voters. These scenarios seem unlikely until one considers the numerous thefts of GOTV voter rolls and other information from GOP campaign office this year, the slashing of tires on Republican GOTV vehicles on election day, and other criminal acts that can only be explained as election-oriented crime. “The Prowler" column in The American Spectator noted: “The early polling numbers are some of the most eagerly anticipated, if highly inaccurate, data on election day, and are widely distributed. Perhaps that was what the Kerry campaign was banking on.”39
Another factor that suggests tampering with the sample was mentioned in passing early here. The early exit poll sampling was heavily women and young people. This is inconsistent with normal anticipated demographics, as is the heavily Democratic content of the sample.40
The only explanation for errors in the exit polling in 2004 is “sample error.” The fact that the errors were unduly large and all in one direction belies attributing this to random statistical problems. I will be interested in seeing an eventual, professional statistical analysis of the probability of all of the errors running in Kerry’s favor. The other dirty tricks in the election and efforts by Democratic sympathizers seem to add to the likelihood that tampering with the polling was a deliberate act.
Whether Democratic voters bussed in early to the polls were simply told to look for exit pollsters to be interviewed or whether the organizations involved had actual polling locations and times or whether exit pol takers were "infiltrated" will undoubtedly be determined in the coming months. It does seem possible that exit polling will no longer be used for predicting elections if this phenomenon continues. "If they really wanted to do it right, they would just report the results," said Curtis Gans, vice president and director of the Committee for the Study of the American Electorate in Washington.41 He said that in 2003 and added : "They're refining their technologies so they can get it wrong again in 2004."
__________________________________________
Footnotes:
1. Mr. Mott was a practicing attorney in litigation in Washington, DC for 28 years, including some election law matters. He now lives and works in Warsaw, Poland.
2. "It takes a deliberate act of fraud and bias to get an exit poll wrong. Since the variables of whether or not a person will actually vote are eliminated in exit polling, it is like peeking at the answer before taking the test. The fact that they were so totally, disastrously wrong is a national scandal," Dick Morris, New York Post, November 4, 2004. http://www.nypost.com/postopinion/opedcolumnists/31590.htm:
3. “Why were the initial exit poll results more Democratic than the actual tabulated vote? No one is sure, though the national sample at midafternoon, which showed Kerry ahead 50 to 49 percent, was 58 percent women. My own suspicion is that some Democrats—at the command level, or somewhere below—had an election-day project of slamming the results. New Hampshire, Minnesota and Pennsylvania initial exit poll results had huge margins for Kerry—much larger percentages than he won in any pre-election poll. If somebody had slipped some Democratic operative the list of exit poll sites—40 to 50 sites in each critical state—he or she could have slipped several hundred operatives into the polling places to take the exit poll ballots and vote for Kerry. The results would have shown Kerry much farther ahead than he actually was and, broadcast through drugdereport.com and other sources, could have heartened Kerry supporters during the afternoon and disheartened Bush supporters. When I was active in Democratic politics, in 1964-80, it would have occurred to us to do no such thing. But Democrats these days are so filled with a sense of grievance and with a feeling of justification for employing any dirty tactics to win, that this is not unthinkable.” Michael Barone, http://www.usnews.com/usnews/opinion/baroneweb/mb_041103.htm.
4. http://www.freerepublic.com/forum/a3a096d852f70.htm
5. http://www.ajr.org/Article.asp?id=519.
6. Alicia C. Shepard , American Journalism Review online, Jan-Feb. 2001, http://www.ajr.org/Article.asp?id=519
7. Id.
8. Id.
9. “At VNS' temporary quarters on the 93rd floor of Manhattan's World Trade Center, two political scientists working for ABC, each with a strong statistical background, didn't think the Florida result was clear-cut. Nor did they completely trust the VNS model. When the decision desk telephoned the two analysts asking, "Can we make the call?" both men advised against awarding Florida to Gore.” Shepard, supra. Shepard’s article contains a treasure trove of contemporary expert opinions from that election and is worth reading for that alone. “TV's elections analysts with their sample precincts and illuminating exit polls had projected winners in thousands of races over the decades -- congressional, gubernatorial, presidential -- and been wrong in only a tiny fraction of them. Suddenly, on November 7, those systems melted down, leaving party activists, newspaper journalists, and the public demanding to know how the TV people had initiated such chaos.” Neil Hickey,”The Big Mistake,”Columbia Journalism Review, online, Jan-Feb 2001, http://www.cjr.org/archives.asp?url=/01/1/hickey.asp.
10. David Bauder, Associated Press, APOL News Service, http://aolsvc.news.aol.com/elections/article.adp?id=20041013122209990005&_ccc=2&cid=946
11. Bauder, supra.
12. Bauder, supra.
13. Bauder, supra.
14. “…the mid-day leaked numbers would have much greater error, perhaps +/- 7% or more.” Mark Blumenthal, Democratic pollster, http://www.mysterypollster.com/main/2004/11/exit_polls_what.html
15. One of the NEP architects (Lenski) quoted in “New Woes Surface in Use of Estimates,” Richard Morin, Washington Post Staff Writer, Thursday, November 4, 2004; Page A29
16. http://www.dailykos.com/story/2004/11/2/161046/605.
17. http://www.dailykos.com/story/2004/11/2/161046/605.
18. The Wonkette website, http://www.wonkette.com/archives/exit-polls-from-gop-pollster-luntz-very-kerry-024805.php.
19. Michael Barone :http://www.usnews.com/usnews/opinion/baroneweb/mb_041103.htm.
20. Seattle Times , “How did exit polls start avalanche of inaccuracy?,” Nov. 4, 2004, http://seattletimes.nwsource.com/html/politics/2002081763_exitpolls04.html.
21. Quoted in the American Spectator magazine online: http://www.spectator.org/dsp_article.asp?art_id=7346.
22. Quoted in the Washington Times, November 4, 2004.
23. Steve Coll, managing editor, Wash. Post, November 3, 2004, http://www.washingtonpost.com/wp-dyn/articles/A13590-2004Oct31.html
24. http://www.emergingdemocraticmajorityweblog.com/donkeyrising/abramowitz.php
25. Quoted by AP’s Bauder, surpa.
26. Rowan Scarborough, Washington times, November 4, 2004.
27. http://www.npr.org/chat/10202004.html.
28. Penracker web blog, http://www.penraker.com.
29. http://www.emergingdemocraticmajorityweblog.com/donkeyrising/abramowitz.php.
30. Richard Morin “New Woes Surface in Use of Estimates,” Richard Morin
Washington Post, Thursday, November 4, 2004; Page A29.
31. “…the mid-day leaked numbers would have much greater error, perhaps +/- 7% or more.” Mark Blumenthal, Democratic pollster, http://www.mysterypollster.com/main/2004/11/exit_polls_what.html
32. Dick Morris, New York Post column of November 3, 2004, http://www.nypost.com/postopinion/opedcolumnists/31590.htm.
33. http://www.usnews.com/usnews/opinion/baroneweb/mb_041103.htm
34. Kerry had the same problem in the Wisconsin primary against Edwards, when the early exit polling data showed him with a much smaller lead than anticipated. Feb. 21, 2004, “Blame talking heads, not the exit polls.” Cragg Hines, Houston Chronicle: http://www.chron.com/cs/CDA/story.hts/editorial/2413157
35. Daly Thoughts web blog, http://www.dalythoughts.com.
36. http://whyfiles.org/009poll/poll_principles2.html. See Merkle, Daniel M. (2000). and Murray Edelman. "Nonresponse in Exit Polls: A Comprehensive Analysis." In Election Polls, the News Media and Democracy, Paul J. Lavrakas and Michael Traugott (eds.), Chatham, N.J.: Chatham House Publishers.
37. http://www.electoral-vote.com
38. Rutenburg, New York Times, November 5, 2004.
39. The Prowler, American Spectator online, November 4, 2004. The Prowler goes on to mention several “leaks” of wildly inaccurate numbers, showing far greater disparities in the voting in Pennsylvania and elsewhere, suggesting that these were a “dirty trick” by Democrats. This could be true, but it would not explain the NEP numbers, which were also inaccurate but not comically so, as were some of the Internet posts that night. http://www.spectator.org/dsp_article.asp?art_id=7346.
40. Democratic analysts and consultants make this point themselves: "Exits come out in a several batches over the course of the day. Democrats, on average, tend to vote later in the day than Republicans. Not always, but that's the pattern, for fairly straightforward demographic reasons. And for that reason their exit poll numbers tend to get better over the course of the day." Josn Marshall, http://www.talkingpointsmemo.com/archives/week_ 2004_10_31.php #003906. See Blumenthal, supra, note 31.
41. http://www.electoral-vote.com
Comments