My Response to the Mystery Pollster’s Critique of RFK Jr.


Richard Charnin


March 20, 2010


In 2006, RFK, Jr. wrote Was the 2004 Election Stolen? in Rolling Stone Magazine. The article was immediately thrashed by Salon’s Farhad Manjoo. Farhad's piece was also immediately debunked by a score of election researchers. But there were other attempts to rebut RFK’s work. In this post, I comment on Mark Blumenthal’s critique of the RFK piece – nearly four years after the fact.  


Blogging as the Mystery Pollster (MP), Mark has attempted to denigrate the evidence provided by the 2004 exit polls which indicated a solid Kerry victory. He had some initial success and was the most popular exit poll naysayer on the Internet. Mark claimed to be a Democratic pollster and appeared on Nightline shortly after the election; the mainstream media had an effective spokesman (although unknown) for raising doubt about the exit polls. Only Keith Olbermann on Countdown dared to report on the election anomalies in Florida and Ohio, but the media immediately locked down on any further discussion of election fraud. The lockdown continues to this day.


So why rebut MP now? It’s worthwhile to take a look back at the early attempts by the mainstream media to shut down the evidence provided by the exit polls. The 2006 midterm exit poll discrepancies and impossible 2008 National Exit Poll indicate that both landslides were much bigger than reported. Not only that, they also confirm that the 2004 state and national exit polls were very close to the True Vote. John Kerry was an even bigger winner than Al Gore in 2000. RFK, Jr. and the election fraud researchers have been vindicated.


Mystery Pollster: Is RFK, Jr. Right About Exit Polls? - Part I



Late last week, Rolling Stone published an article by Robert Kennedy, Jr. that asks provocatively, "Was the 2004 Election Stolen?"  While it covers many topics involving alleged suppression and fraud in Ohio, the article disappoints in its discussion of the exit poll controversy, because on that aspect of the controversy Kennedy manages to dredge up nearly every long-ago discredited distortion or half-truth on this subject without any acknowledgement of contrary arguments or the weaknesses in his argument.  It is as if the exit poll debate of the last eighteen months never happened. With this two-part post, I want to review the article's discussion of the exit poll controversy in-depth, for it provides a good opportunity to learn something about what exit polls can tell us -- and mostly what they cannot -- about whether fraud was committed in the 2004 elections.



RFK dredges up half-truths? Really? You say “alleged suppression and fraud”? No. It is a documented fact.



But before getting to exit polls I want to make two things clear.  First, despite its weaknesses, the Kennedy article raises some important and troubling questions about real problems inOhio in 2004.  As Ohio State University Law Professor Dan Tokaji puts it, the article is "useful in exposing how shoddy election administration practices can result in lost votes, and how some recently enacted laws will make things worse rather than better."  The summary of problems deserving attention includes long lines in minority precincts, efforts of the Republican Party to selectively challenge (or "cage") new registrants and the many examples of pure incompetence by local election officials.  And then there is partisanship of Republican Secretary of State Ken Blackwell, now his party's nominee for governor.  Blackwell will need to answer to Ohio voters for, as's Farhad Manjoo writes, having "used his powers for partisan gain," issuing "a series of arbitrary and capricious voting and registration rules that could well have disenfranchised many people in the state" (but interests disclosed: I am a Democratic pollster with clients in Ohio).



You have just refuted your "alleged" comment.



Second, while I have devoted 68 posts and tens of thousands of words to the exit poll controversy since Election Day 2004, I have never argued that the exit polls can be used to rule out or disprove the possibility that vote fraud may have occurred in Ohio or anywhere else during in 2004.  The question has always been whether the exit polls provide affirmative evidence that fraud did in fact occur. This involves a very basic concept of statistical inquiry:  We assume no effect until one can be proven, or more technically, we assume a "null hypothesis" until we can prove some alternative.  The same principle exists in law as the presumption of innocence.  We do not assume a crime has been committed and work backwards to try to disprove it.  We presume innocence until enough evidence has been established to prove guilt.



In civil court, a preponderance of the evidence (50% probability) is required to convict. In criminal court, evidence is required beyond a reasonable doubt (over 99%). Once again, incidents of vote suppression and miscounts have been documented in virtually every state– and the statistical evidence based on the exit polls proves it to a near-certainty.  And let's be clear to call it  election fraud  to indicate that it was the work of election officials (of which there is much evidence) and not voter fraud, a term favored by those who would blame the voters (of which there is near-zero evidence). The odds that Bush’s recorded vote share would exceed his unadjusted exit poll share by more than the margin of error in 29 states is less than 1 in 100 trillion. It is also a fact that in the five presidential elections from 1988 to 2004, a (conservative) 3% margin of error was exceeded in 66 states, of which 65 favored the GOP candidate. The WPD data is provided by the Edison-Mitofsky 2004 Election Report. Do the math.



Everyone agrees that the 2004 exit poll results gathered by the news media consortium known National Election Pool (NEP) showed a small but statistically significant difference that favored John Kerry when compared to the official count.  But is that discrepancy evidence of fraud?  It might be, if we could rule out the possibility that other problems or potential sources of error in the exit polls that can also explain the discrepancy.   What I have argued for the last year and a half is that the exit polls have many such weaknesses that have long been in evidence.


At the center of the exit poll debate is a basic concept about polls that deserves a lot more attention:  Statistical sampling error -- the random variation that comes from drawing a sample of voters rather than interviewing the whole population -- is just one source of potential error in a survey.  There are others including bias from selected respondents who decline to participate (response error), from voters missed altogether (coverage error), from questions that do not accurately measure the attitude of interest (measurement error) or from a failure to choose exiting voters at random using the correct sampling interval.



Exit polls are not biased in favor of the Democrats.  On the contrary, millions of mostly Democratic votes are uncounted in every election, therefore recorded vote counts are biased in favor of the Republicans. In 2004, average exit poll response was higher in Republican than Democratic states.



The first indication that something was gravely amiss on November 2nd, 2004, was the inexplicable discrepancies between exit polls and actual vote counts. Polls in thirty states weren't just off the mark -- they deviated to an extent that cannot be accounted for by their margin of error.



It is certainly true that the 2004 exit poll estimates produced by the National Election Pool (NEP) generally overstated John Kerry's share of the vote compared to the vote count.  That overstatement in statewide exit poll estimates averaged five (5) percentage points on the Bush-Kerry margin, according to the report that the exit pollsters, Edison Media Research and Mitofsky International, released in January 2005.  


The overstatement was slightly larger (5.5 percentage points) for the estimate of the national popular vote.  The national exit poll sample showed Kerry with 51% and Bush with 48%, but the final count showed a 2.5% margin (50.73% for Bush and 48.27% for Kerry).  It was larger still (6.5 percentage points) in terms of the average error within individual precincts -- something the report termed "within precinct error" (WPE).



No. The aggregate unadjusted state exit polls gave Kerry a 52-47% margin. That means the discrepancy was closer to 7.5%.



The key point:  Everyone -- including the exit pollsters -- agrees that the average discrepancy was statistically significant. In all but four states, the discrepancy favored President Bush.



No.  While the discrepancy was certainly widespread, this sentence misstates the statistics provided in the citation, the Edison-Mitofsky report.  Even if we ignore statistical significance and simply count up the number of states where the exit poll running showed Kerry doing better than the count even by some small fraction of a percent, then the discrepancies favored Bush in all but nine states, not four (see pp. 22-23).   The reference to four states appears to come from the number of states where exit polls overstated Kerry's vote by more than one standard error. But the equivalent number where the discrepancy favored President Bush by more than one standard error was 26 states, not "all but four."



No. The unadjusted exit polls exceeded a 3% margin of error in 28 states for Bush at the most commonly used 95% level of confidence (2 standard deviations).  The number of states exceeding the MoE in each partisanship grouping (Democratic, Republican, Battleground) is given below.



And to try to translate that into something approximating English, a difference of one standard error or more means that we can be roughly 68% confident that the difference is meaningful.  "Statistical significance" is a subjective judgment -- in the eye of the beholder -- but in attitude surveys that term usually implies a confidence level of 95% or greater.


Aside from the distortion of the statistics, however, this point is not particularly relevant.  Again, everyone agrees that the overall exit poll discrepancy was widespread and statistically significant.



Not true. Let’s take a closer look at the 2004 election. We assume a very conservative 3% state exit poll margin of error and consider the unadjusted exit polls, not theadjusted polls which were in the process of being forced to match the recorded vote count. At the 95% confidence level, the probability that the  MoE would be exceeded in a given state in favor of  Bush or Kerry is 1 in 20 (5%). The probability that it would be exceeded in favor of Bush alone is 1 in 40 (2.5%). Now let’s look at the average WPE/MoE by state partisanship.


- The average Democratic state WPE was a whopping 8.9. The MoE was exceeded in 11 of 15 states (73%) - all for Bush.

- The average Battleground state WPE was 6.9. The MoE was exceeded in 10 of 15 states (67%) – all for Bush.

- The average Republican state WPE was 3.8. The MoE was exceeded in 7 of 21 states (33%) - all for Bush.


The 3% margin of error was exceeded in a total of 28 states - all in favor of Bush. The probability is 1 in 19 trillion that the MoE would be exceeded in at least 16 states. Imagine what it is for 28 states. Assuming a 2% average MoE, the probability is even lower since it was exceeded in 36 states: 34 in favor of Bush, 2 in favor of Kerry.



Over the past decades, exit polling has evolved into an exact science. Indeed, among pollsters and statisticians, such surveys are thought to be the most reliable. Unlike pre-election polls, in which voters are asked to predict their own behavior at some point in the future, exit polls ask voters leaving the voting booth to report an action they just executed.



It is certainly true that exit polls benefit from having ready access to actual voters who have just made their choices.  Exit pollsters need not jump through hoops to identify "likely voters" nor find ways to allocate those who say they are "undecided."   And yes, if you look back at my first post on exit polls on Election Day 2004, I too described exit polls as "among the most sophisticated and reliable political surveys available."



That is true. So what has made you change your opinion?



However, I have certainly learned a great deal about exit polls since then, and calling them the "most reliable" of surveys ignores a host of other practical challenges.  Exit polls generally sample a larger number of voters than telephone polls, but they do so because the "cluster sample" technique used on exit polls-- which first selects sample precincts and then voters at those precincts -- has more sampling error than comparabbly sized telephone poll samples.  Exit polls also miss the growing number that vote by mail or cast absentee ballots.



Even with the average 30% cluster factor, the 12:22am National Exit Poll (13047 respondents) had a mere 1.12% MoE. And considering that over 100,000 were polled in the states, the aggregate MoE is even lower.



Clarification: the exit pollsters used telephone surveys to reach absentee voters in 2004 in 13 states that had high proportions of absentee or vote-by-mail voters. However, these telephone polls face all the usual challenges of pre-election surveys in identifying actual voters.



But the WPE for paper ballots was 2%, according to the exit pollsters, compared to 7% for optical scanners, 7% for DREs and 11% for levers. And calling up to askhow they voted is a lot different than asking how they will vote. Oregon voted 100% by mail-in paper ballots and had an equivalent 1.8% discrepancy based on a telephone survey. The other 14 battleground states all voted by machine and had an average 7.5% WPE. Explain that.



Most important, exit polls rely on their interviewers to randomly select voters at each polling place.  Interviewers are instructed to keep a running tally of voters as they exit the polling place and attempt to interview only those voters at a specific "interval," such as every third voter or every fifth that passes by.  A host of real world conditions, such as  -- the number of precincts voting at any given polling places, how far the interviewer is required to stand from the exit, the number of exits, inclement weather or simply the interviewer's level of experience -- can interfere with their ability to intercept and interview voters at random.



So what does that prove?  These conditions should affect Kerry and Bush equally, no?



Exit poll interviewers must also cope with a phenomenon impossible on telephone polls:  Curious voters who offer to volunteer to participate, even if they would not have been selected according to the random interval procedure.



Are you implying that these curious voters are Democrats? If so, where is the evidence?



Finally, the NEP exit pollsters face an immense logistical challenge:  Once every four years, they conduct exit polls both nationally and in every state.  Thus, they must recruit and deploy enough interviewers to cover nearly 1500 precincts scattered randomly throughout 50 states and the District of Columbia.



That is irrelevant. What is your point? The pollsters get paid millions for their efforts.



The results are exquisitely accurate: Exit polls in Germany, for example, have never missed the mark by more than three-tenths of one percent.



Not true.  That 0.3% statistic comes from averages calculated by Steven Freeman on the exit polls conducted by one German exit pollster (Forschungsgruppe Wahlen) for the ZDF television network in elections held in 2002, 1998 and 1994.  But even Freeman's paper concedes that other German exit polls have been off by slightly more and in one case by as much as 1.5% for individual candidates.



Freeman's average calculation is correct. He concedes that other polls have been off by as much as 1.5%. But even that is a very low number. You are nitpicking now.  



The results were also not quite so accurate for FG Wahlen in the 2005 parliamentary elections (results available here).  They showed a slightly higher error averaged across the five main parties (0.9%).  However, if we group the parties into coalitions as Freeman did in his paper "to make the numbers more comparable to the U.S. Presidential election" (p. 8, see table 1.3) the most recent F.G. Wahlen exit poll showed an error on the margin of 3.8% (my calculation).


However, while the more recent German exit polls may not be quite as "exquisitely accurate" as Kennedy implies, he and Freeman are right that the German exit polls have typically been more accurate than in the U.S.  And as I explained back in December 2004, that greater accuracy occurs for sound fundamental reasons having to do with measures that appear to reduce sampling, coverage and non-response error: The German exit polls feature larger sample sizes and benefit from significantly better cooperation from election officials.  FGWahlen assigns two "experienced" interviewers per precinct and they are allowed to stand at the door of the polling place for the entire day.  The NEP assigned one interviewer to a polling place in 2004, three quarters had never worked as an exit poll interviewer before, all had to leave the their polling place uncovered several times during the day and only about half were allowed to stand inside or just outside the door of the polling place.  The German exit pollsters typically obtain an 80% response rate, the US exit polls in 2004 had a 53% completion rate (p. 31). All of this means that the German exit polls are less prone to coverage and response error.



Are you saying that German exit pollsters are superior to Mitofsky? Isn’t he known as the father of exit polling? And let’s not forget: the 2004 exit poll completion rates were higher in GOP partisan precincts.



''Exit polls are almost never wrong,'' Dick Morris, a political consultant who has worked for both Republicans and Democrats, noted after the 2004 vote. Such surveys are ''so reliable,'' he added, ''that they are used as guides to the relative honesty of elections in Third World countries.”



Dick Morris is entitled to his opinion, but many others with more relevant exit poll experience disagree.  As noted here eighteen months ago (and reported this weekend by Salon'sFarhad Manjoo, the ACE Project (an acronym for Administration and Cost of Elections, a joint project funded by the UN and the US Agency for International Development) concluded:


Exit poll reliability can be questionable. One might think that there is no reason why voters in stable democracies should conceal or lie about how they have voted, especially because nobody is under any obligation to answer in an exit poll. But in practice they often do. The majority of exit polls carried out in European countries over the past years have been failures.



The majority have been failures? Prove it.  To follow your logic, you must believe that in 2004 many more Bush voters lied and told the exit pollsters they voted for Kerry than vice-versa? Why would they do that? Do you have proof?



Also, as Bard College political scientist Mark Lindeman reports, senior election observers from the Carter Center have repeatedly advised against the use of exit polls for election monitoring in Central American Countries, calling them "risky," "unreliable" and "misleading."



Yes, that was certainly true in the Venezuela presidential election where the U.S. funded exit pollsters rigged the results in order to undermine the duly elected Chavez. The U.S. had already attempted a coup which failed due to a popular uprising. In any case, they have voting transparency - and hand-count 55% of the ballots as a check. How come we don’t do the same thing?



In 2003, vote tampering revealed by exit polling in the Republic of Georgia forced Eduard Shevardnadze to step down. And in November 2004, exit polling in the Ukraine -- paid for by the Bush administration --- exposed election fraud that denied VViktor Yushchenko the presidency.)



And thus we come to an oft-repeated legend: Exit polls "exposed" fraud in Ukraine and elsewhere, so why not here?  The biggest problem with that story is that the election monitors in those counties did not depend on exit polls to provide evidence of fraud.   In Ukraine, at least, the solid evidence came from eye-witnesses, taped phone conversations, and physical evidence of vote tampering.  Review the reports of the most authoritative monitor on the elections in Georgia and Ukraine -- the Office of Security and Cooperation in Europe (OSCE) --, and you will find plenty of evidence cited but not a single mention of the phrase "exit poll."


The report of MIT Political Scientist Charles Stewart (as aptly summarized by Salon's Farhad Manjoo) also provides a series of reasons worth reviewing as to why the Ukraineexample provides a poor parallel to the 2004 U.S. election.



Are you saying that Secretary of State Powell and others in the Bush administration did not refer to the Ukraine exit polls as proof of election fraud?



But that same month, when exit polls revealed disturbing disparities in the U.S. election, the six media organizations that had commissioned the survey treated its very existence as an embarrassment.



There is reason for a sense of embarrassment and it involves one of the most blatant omissions from the Kennedy article:  U.S. exit polls have been wrong before.  In fact, according to the Edison-Mitofsky report, they have shown a consistent discrepancy favoring the Democrats in every presidential election since 1988.  And while the 2004 discrepancy was the highest ever, they were almost as far off in 1992.  More specifically, the "within precinct error" (WPE) reported by Edison-Mitofsky showed differences favoring the Democrat of 2.2 points on the margin in 1988,  5.0 in 1992, 2.2 in 1996, 1.8 in 2000 and 6.5 in 2004 (see p. 34).



The exit polls have been wrong before? How do you know that? That is the biggest myth of all. In fact, the unadjusted exit polls have been essentially correct.  The proof is simple: There are millions of uncounted votes in every election. The U.S. Census (0.30% MoE) indicates that there were nearly 11 million in 1988, 9 million in 1992, 9 million in 1996, 6 million in 2000 and 4 million in 2004. The Democrats had from 70-80% of these uncounted votes. Greg Palast reported that over 50% of uncounted votes were from minority districts which are 90% Democratic. When one considers total votes cast, rather than total votes recorded, the resulting vote shares closely approximate the exit polls. How come you never talk about that?



Go back and watch the classic political documentary, The War Room -- or easier, go back and read my post from January 2005 -- and you will see that that leaked exit polls on Election Day 1992 provided as distorted a view as those leaked in 2004.  The difference was that the leaked exit polls in 1992 were known mostly to insiders and served to exaggerate the size of Bill Clinton's eventual victory.  Clinton won by less than those early exit polls suggested, but he still won the election, so there was little lingering outrage.



No.  Once again for emphasis: the "leaked" 1992 exit polls reflected total votes cast which exceeded the recorded vote by 9 million. And the vast majority of these uncounted votes were Democrats. Did you know that? If you did, why did you fail to mention it? You should have done your homework, rather than make the same mistake Farhad Manjoo did in relying on the chief exit poll naysayer, Mark Lindeman. 


Is RFK, Jr. Right About Exit Polls? - Part II



Instead of treating the discrepancies as a story meriting investigation, the networks scrubbed the offending results from their Web sites and substituted them with ''corrected'' numbers that had been weighted, retroactively, to match the official vote count.



That sounds like quite a cover-up, doesn't it?  Unfortunately, like so much of the discussion of exit polls in the RFK, Jr. article, that first sentence is wildly misleading.  The practice of gradually adjusting the network exit poll tabulations to reflect the actual count has been a standard practice for decades.  As this procedure seems to confuse almost everyone - including as of just yesterday, Robert Kennedy himself -- a bit more explanation is in order. 



Yes, it is standard operating procedure to force the exit polls to match the count.  But what if the vote count is corrupted? What if the election is fraudulent? Does it make sense to match the exit polls to a fraudulent recorded count? Do you think it is statistically valid? Do you believe it is in accordance with the scientific method? Where is the peer-review? How come the corporate media and AAPOR never question it?


In matching to the recorded vote count, the Final 2004 National Exit Poll proved the fraud.  It required 6-7 million more returning Bush 2000 voters than were living in 2004.  The proof is simple. In forcing a match to the recorded vote, The Final indicated that returning Bush voters comprised 43% (52.6 million) of the 2004 electorate (122.3 million).


But Bush only had 50.5 million votes in 2000. Approximately 2.5 million of them died prior to the 2004 election. Of the 48 million living, 46-47 million voted (assuming 96-98% living 2000 voter turnout).  Therefore, the Final implied that there were 6-7 million more returning Bush voters than actually voted – in other words, phantom voters.


What would you call them? We should just stop right here. Despite this evidence, the media never questioned the official count. That sure is evidence of a cover up to any rational observer. 



I have attempted to explain the workings of this process in the past (especially here, here and here), but my understanding has grown considerably since Election Day 2004 and some of my initial explanations were, in retrospect, a bit over-simplified.  So let me try again.  Let's start with two important terms in the exit pollster lexicon:  "projections" and "tabulations."



Oh, so you didn’t know about exit polls? As an experienced Democratic pollster, you should have. So who chose you to appear on Nightline? Why were you chosen and not Freeman? Why has not a single election fraud researcher appeared on ABC, FOX, CNN or MSNBC? Maybe it’s because the researcher would talk about the millions of phantom Bush voters and millions of uncounted votes.



The exit pollsters use the term "projections" to refer to various estimates of the overall vote preference in a given state that appear on the computer screens of the "decision desk" analysts responsible for calling winners at the television networks and at the Associated Press.  While the details of these proprietary "decision screens" are closely held, I am told that they include a large number of different estimates that change during the course of the day.  Some of these were defined in the glossary of the Edison-Mitofsky (E/M) post-election report released in January 2005, and have names like Best Survey Estimate, Best Geo Estimate, Composite Estimate, County Model, etc. (see pp. 7-10).


The most relevant projection to this controversy has the charming name of "Best Geo Estimator," which the E/M report describes as "the best estimate" displayed on the "Decision Screens" of network analysts (p. 19).  Early on Election Day, before the polls close, these estimates derive entirely from exit poll tallies.  These estimates are not based on "raw" data. As part of the survey design, they are statistically adjusted (or weighted) so that the relative sizes of geographic regions within each state match their sizes in recent presidential elections (hence the "Geo" in Best Geo Estimator). 


In some states, these adjustments also weight intentional "over-samples" of heavily minority precincts to their actual contribution in past elections.  The selection of additional precincts allows for adequate sub-samples of Latino or African-American voters in cross-tabulations.  Finally, in the 13 states where the exit pollsters conducted telephone interviews to determine the preferences of early and absentee voters, these results must be also combined and weighted to the appropriate proportions.


The main point here is that every exit poll estimate, from those done in the middle of the day, is typically "weighted" to some degree to reflect the design of the survey.  Exit pollsters never look at "raw" or unweighted results simply because the survey is not designed to work that way. 



The polls are “weighted” to bogus official vote counts. To compound the felony, pristine raw precinct data is not made available to election researchers and the general public. Do the exit pollsters consider the millions of uncounted votes in matching to the recorded vote? No, they are paid to project the “official” recorded vote. Hell, anyone can do that. Why bother to poll? Just recite the official returns. They are obviously not interested in the True Vote. Does that not bother you?  Does it not bother you that the Final National Exit Poll was “adjusted” and “forced to match” the recorded vote by requiring 6 million more returning Bush 2000 voters than were living in 2004? It should.



In the middle to end of the day, the weights used for the Best Geo Estimator (and the other estimates) may be altered to reflect any available precinct level hard-counts of actual turnout.  In other words, interviewers attempt to obtain counts of the number of voters that have actually cast ballots at the selected precinct from polling place officials.  When available, these counts are usually based on the number that signed into the precinct registration book (something I described in more detail here).  Although this process is far from perfect, the goal is to determine if turnout patterns appear to be different than the assumptions used to design the survey sample.  At some point during the day, the exit pollsters begin to make use of interviewer tallies of the gender, race and approximate age of the voters that refused to be interviewed.  They may weight their estimates to correct any apparent bias in gender, race or age suggested by those tallies. 


Just before the polls close, the exit pollsters do a final tabulation (referred to as the "Call 3" data) based on the complete exit poll sample.  Again, the networks analysts make use of a number of different estimates for use in projecting winners. However the final "Call 3 Best Geo Estimator" is probably the most relevant to this discussion because it is the projection based on the exit poll only, without any adjustments to match the official vote count. 


Once the polls close, the pollsters work to obtain the official vote count for each precinct in the exit poll sample (as well as for a larger random sample of precincts in each state). They gradually swap out exit poll tallies within each precinct and substitute the actual vote count, on the theory that the net result will provide a gradually clearer picture of the eventual winner.  At some point, their models will also incorporate the county vote data being obtained by the Associated Press.



They gradually swap out exit poll tallies within each precinct and substitute the actual vote count on the theory that the net result will provide a gradually clearer picture of the eventual winner. That is total farce. What if the “winner” of the recorded vote is not the winner of the True Vote?  



The term "tabulations" refers to the cross-tabulations that show the complete exit poll results by demographic subgroups (gender, age, race, party, etc).  These tabulations appear on the network decision screens, but also are run to PDF files (like this one) and sent to the newspapers and other news organization that pay for access to the exit poll data.   On election night, the weighted tabulations appeared on lacked an overall "vote estimate."  For months, extrapolations of the overall vote based on the sex-by-vote tabulations posted to CNN on Election Night were the closest data only available to the "Call 3 Best Geo Estimator" projections.


Although note, according to the E/M report the tabulations are weighted more often to the "Composite Estimate," than to the Best Geo Estimator, especially early in the day.  The Composite Estimate combines the exit poll tallies with a summary of public pre-election polls.



The 12:40am state exit polls (the “Composite Estimate”) were already “contaminated”.  Kerry’s 52% unadjusted share was reduced to the 51% Best GEO to the 50.3% Composite.  But that was not enough. The exit pollsters had to radically reduce the Composite in matching Kerry’s 48.3% official recorded share.



Here is the key point:  At any given time on Election Day or Night, the exit poll "tabulations" are weighted (or "forced," to use the exit pollster lingo) to match whatever estimator the analysts consider "best" for that purpose at any given time.  Since the estimators gradually change to include more and more data based on the count, the exit poll tabulations -- including those posted to network web sites -- will ultimately get "retroactively" weighted to match the vote count.  That procedure has always been part of the exit poll system. 



Forced to match the estimate and retroactively matched to the vote count?  So what’s the point of an exit poll if it’s only going to be matched to the vote count in the end? That seems like a waste of time and money - unless the objective was to convince the public that Bush won the election.



So the weighting procedure is not evidence of a cover-up. It is a feature intended to allow the exit poll cross-tab tabulations to provide as accurate a read on the actual electorate as possible. 



An accurate read on the “actual” electorate? Or an accurate read on the miscounted votes? What nonsense.



Rather than finding fault with the election results, the mainstream media preferred to dismiss the polls as flawed. ''The people who ran the exit polling, and all those of us who were their clients, recognized that it was deeply flawed,'' says Tom Brokaw, who served as anchor for NBC News during the 2004 election. ''They were really screwed up -- the old models just don't work anymore. I would not go on the air with them again.''



Tom Brokaw was just promoting the propaganda that the exit polls “screwed up” without considering that the votes may have been miscounted.



The mainstream media had good reason to be cynical about exit poll results, as they had memories of many such past snafus.  As summarized at the end of Part I of this series, the national election polls had shown a consistent discrepancy favoring the Democrats in every presidential election since 1988.  The discrepancy in 1992 was almost as great as the one in 2004.  There were also similar problems resulting in overstatements of the votes cast for Pat Buchanan in the Republican primaries in New Hampshire in 1992 and Arizona in 1996. 



Exit poll snafus? Or miscounted votes? In 1988, Bob Dole was way ahead in the NH primary pre-election polls, but Bush pulled an upset.  The governor, John Sununu, was a former engineer who designed the voting machines. He was hired by Bush to serve in his cabinet.  In 1992, there were over nine million uncounted votes (70-80% for Clinton). No wonder the exit polls were far off – all those uncounted (and miscounted) votes.  The common denominator in 1988, 1992, 2000 and 2004 was that…a Bush was running.



Richard Morin, polling director for the Washington Post, recounted using a 1988 national exit poll sampling that showed the presidential race "to be a dead heat, even though Democrat Michael Dukakis lost the popular vote by seven percentage points to Dubya's father."



There were 10.6 million net uncounted votes in 1988 – about 8 million were for Dukakis. Why don’t you talk about that?



In fact, the exit poll created for the 2004 election was designed to be the most reliable voter survey in history. The six news organizations -- running the ideological gamut from CBS to Fox News -- retained Edison Media Research and Mitofsky International, whose principal, Warren Mitofsky, pioneered the exit poll for CBS in 1967 and is widely credited with assuring the credibility of Mexico's elections in 1994. For its nationwide poll, Edison/Mitofsky selected a random sub-sample of 12,219 voters -- approximately six times larger than those normally used in national polls) -- driving the margin of error down to approximately plus or minus one percent.



Here we have yet another technically true but highly misleading paragraph.  It is true that in 2004, the news organizations hired Warren Mitofsky and Edison Research to replace its forerunner, Voters News Services (VNS).  However, Mitofsky had been the founder and director of VNS (originally known as Voter Research and Services) in 1990, and his principal deputies continued to run it after he left in 1993.  Not everyone was optimistic about the change.  Writing in the Columbia Journalism Review, former NBC News president Larry Grossman voiced the concerns of those worried about "the very short time frame, new election day complications such as the growing trend toward mail balloting and people's increasing tendency to mislead pollsters or refuse to be polled, and the recent history of vote-tallying failures."



Vote tallying failures or vote-recording failures? What would you expect from the president of NBC News?



The National Election Pool (NEP) certainly touted improvements made in the 2004 exit polling and projection system.  It is true, for example, that the 2004 exit polls expanded to thirteen the number of states where they supplemented polling place interviews with telephone polling of those who voted absentee or by mail.  It is true that Mitofsky and his partner Joe Lenski updated the computer models used by the NEP to require more statistical confidence before making a projection.  


But this paragraph creates the misleading impression that Mitofsky worked to "drive down" the margin of error on the national sample.   While they may have juggled statewide allocations to boost samples in some battleground states (such as Florida), according to data archived with at the University of Michigan the sample size of the so-called national sample used to estimate views of voters nationally was actually smaller in 2004 (n=12,219) than in 2000 (n=13,225) or 1996 (n=16,637).  Some may also get the impression that this national sample plays a significant role in projecting the outcome of the election.  It does not.  Only the state exit poll estimates are used for that purpose.  The national exit poll tabulation are done only to provide subgroup results for national news coverage.  Note that the "six times larger than normally used" comparison refers to the typical size of a national telephone poll, not an exit poll. 



Kerry led by a steady 51-48% right up to 12:22am. But Mitofsky forced the final exit polls to match the vote count. The fact that it is a standard procedure does not make it legitimate.  Mitofsky could have just said that the exit polls did not match the count and left it at that. Instead, he chose to thrash his own polls without questioning the vote count. But he needed an explanation, so he concocted the ridiculous canard that the discrepancy was due to Kerry voters being polled at a 56/50 ratio to Bush voters. It was the famous reluctant Bush responder theory – which was totally debunked by the US Count Votes simulation and my Exit Poll Response Optimizer.  Mitofsky’s Final National Exit Poll indicated that returning Bush voters outnumbered Gore voters by an impossible 7.3 million! Once again, until it seeks in: the Final indicated that there were 52.6 million returning Bush 2000 voters. But Bush only had 50.5 million recorded votes in 2000. Approximately  2.5 million died prior to the 2004 election, therefore only 48 million were alive who could vote in 2004. How come you don’t talk about that?



June 09, 2006

Is RFK, Jr. Right About Exit Polls? - Part III

Posted by Mark Blumenthal on June 8, 2006 at 10:48 PM



This post resumes my paragraph-by-paragraph review of the discussion of the exit polls in the article by Robert F. Kennedy, Jr. in Rolling Stone, "Was the 2004 Election Stolen?"  Part I looked at Kennedy's claims that exit polls have been "exquisitely accurate" in the past.  Part II examined the claim that networks "scrubbed the offending results."



On the evening of the vote, reporters at each of the major networks were briefed by pollsters at 7:54 p.m. Kerry, they were informed, had an insurmountable lead and would win by a rout: at least 309 electoral votes to Bush's 174, with fifty-five too close to call. (28) In London, Prime Minister Tony Blair went to bed contemplating his relationship with President-elect Kerry.



Of the many distortions in this article, this one is the most bizarre.  Perhaps it depends on the meaning of "briefed." I have checked with three individuals who worked within the NEP consortium on Election Night, and none remembers any such "briefing" by the exit pollsters anywhere near that hour.  One source told me flatly that Edison/Mitofsky conducted no conference call briefings in the early evening.  The data streamed constantly to "decision screens" of network analysts and, if necessary, they were able to communicate with the pollsters via instant message.



You say “many distortions” and “bizarre”?  There are no distortions in RFK’s piece. You resort to smears. The networks were in fact "briefed".



The eight o'clock hours was an important time, of course, because polls were about to close in fifteen states plus the District of Columbia. The networks would be projecting winners in many of those states - the ones where the outcome had never been in doubt - at the top of the hour on the basis of exit poll results.  However, the idea that the exit pollsters "informed" their NEP clients at that hour that Kerry "had an insurmountable lead and would win by a rout" defies all logic and available evidence.



Is it not plausible that the exit pollsters would inform their paymasters of a Kerry landslide – and ask for guidance on what to do next?



First, if the networks believed Kerry had a lock on 309 electoral votes at 7:54 p.m., it certainly had no effect on the projections they made on the air.  In fact, none of the networks ever projected an overall winner.  The margins in Iowa, Wisconsin and New Mexico remained so close that the networks were not able to project a winner until after Kerry had conceded the next day.  And the networks split on Ohio and Nevada Only FOX and NBC called Ohio for Bush, but not until just before 1:00 a.m. and neither network projected Nevada CBS, ABC and CNN called Nevada for Bush at about 3:00 a.m., but never called a winner in Ohio.   Thus it almost goes without saying that back at 7:54 p.m., the states that would decide the Election remained too close to call -- well within the margin of error --on all of the network's exit poll based estimates and projections.



Too close to call? Kerry won the unadjusted Ohio exit poll by 8.8% and Florida by 2.6%.



And if that's not enough, the Edison/Mitofsky pollsters conducted a well-documented briefing at 4:30 p.m. very different from the one Kennedy claims occurred three hours later. Warren Mitofsky discussed it a few days later on the News Hour, and the E/M report described it as follows:


On Election Day, at 4:30 PM ET, we convened a conference call with the Decision Teams of the NEP members and cautioned them that we expected sizeable errors in the exit polls in nine states; in seven states (Connecticut, Delaware, Minnesota, New Hampshire, Pennsylvania, Vermont and Virginia) we suspected that the exit poll estimates were overstatement [sic] of the vote for Kerry; in two states (South Dakota and West Virginia) we suspected an overstatement of the Bush vote. We made these warnings based upon the discrepancies between the exit polls and our prior estimates in these nine states. [p. 17 - Note:  the term "prior estimates" refers to a summary of pre-election polls results for each state collected by NEP].



So they were briefed, after all! Were they also briefed at 7:30pm when Kerry was leading by 3%? Were they briefed at midnight? Who is to say they weren’t? Kerry led the National Exit Poll timeline by a constant 51-48% from 4:30pm to 12:22am.  He also led the state aggregate GEO by 51-48%.  Of course, the polls were forced to match the recorded vote around 1:00am.



And here is the back-story that provides some added confirmation:  The NEP "subscribers" -- mostly newspapers including the New York Times and the Washington Post -- were not included on that call and were quite livid as a result.  Edison-Mitofsky had to essentially apologize in their report (p. 17) for "not sharing with the subscribers our concerns about the accuracy of the exit poll estimates." 


There was however, one person -- a blogger -- who communicated the substance of that call to a much wider audience.  Her name was Ana Marie Cox, and at 5:40 p.m. on Election Day she posted the following message to Wonkette Disclaims a birdie: "*** There appear be problems with exits in the following states that could be tipping numbers toward Kerry: MN, NH, VT, PA, VA, CT, DE. described only as 'serious' issues we're looking at. So I would not put too much faith in those results."


So there you have it.  For all the disparagement of bloggers that followed, those who read Wonkette on Election Day got better real-time information on the exit poll's shortcomings than the paid subscribers at the New York Times and the Washington Post. 


No, they were only problems and issues for the National Election Pool (NYT, AP, WP, CNN, FOX and ABC) who were the ultimate exit poll decision makers. Wonkettewas apparently unaware of Kerry’s steady lead in the National Exit Poll timeline from 4pm to 12:22 am. She was surely unaware of his 52-47% margin in the unadjusted state exit poll aggregate (the WPE’s were in the Jan. 19 report). She may have been unaware of his 51-48% lead in the Best GEO estimate.



As the last polling stations closed on the West Coast, exit polls showed Kerry ahead in ten of eleven battleground states -- including commanding leads in Ohio and Florida -- and winning by a million and a half vvotes nationally. The exit polls even showed Kerry breathing down Bush's neck in supposed GOP strongholds Virginia and North Carolina.(30) Against these numbers, the statistical likelihood of Bush winning was less than one in 450,000.



That reference to West Coast poll closings does add a dramatic touch to this story.  While no one denies that the Virginia and North Carolina exit polls showed discrepancies favoring Kerry, by the time "the last polling stations closed on the West Coast," George Bush had been the declared winner of those two states for more than two hours. 



So that means the exit polls were wrong, because Bush was declared the “winner”?



And while everyone concedes that the discrepancy in Kerry's favor on the national exit poll sample was statistically significant, Kennedy's contention that the exit polls gave Kerry "commanding leads" in Ohio and Florida has generated a confusing bit of controversy over last few days. Writing on Salon,com, Farhad Manjoo linked to one of my blog posts from December 2004 and concluded:


Of the ten battleground states that the exit poll showed Kerry winning, he ultimately lost four -- states that, you could say, cost him the election. These were Ohio, Iowa, Nevada andNew Mexico. But in none of those states was Kerry's lead outside the poll's margin of error.  In other words, the poll results showed a race that was too close to call . . . As Mitofsky told me, television news networks, looking at the exit poll data, seemed to understand that Kerry did not top the margin of error, and so did not call these states for him [emphasis in original].



No.  Kerry was won the unadjusted Ohio exit poll by 8.8% and the Florida poll by 2.6%. The average battleground state WPE was 7.5%. But Oregon, a battleground state with 100% mail-in ballots had a 1.8% vote discrepancy from the exit pollster telephone survey.  Farhad was immediately debunked by a score of well-known, math-degreed election researchers. Like you, he referred to Mark Lindeman as an advisor in writing the piece.



In a reply posted Wednesday on, Kennedy answered back: 

With regard to his claim that the exit poll numbers were within the margin of error, Manjoo is referring to the "corrected" numbers that Mitofsky retroactively weighted to more closely resemble the certified vote tallies. Freeman and other statisticians, mathematicians and social scientists disturbed by the poll numbers rely instead on Mitofsky's raw data. Those numbers, as I report, indicated that Kerry had an advantage outside the margin of error in Ohio, Florida, Nevada and New Mexico. He also had advantages within the margin of error in two other states: Iowa and Colorado, which also went to Bush in the final results. It's hard to engage in an honest debate with someone who won't debate the same set of numbers.


One thing that truly mystified me was Kennedy's assertion that the final exit polls showed Kerry "ahead" by any margin, commanding or not in Florida or Colorado Steven Freeman, whose forthcoming book Kennedy cites in Rolling Stone to support the "commanding lead" claim, had originally authored a paper touting numbers extrapolated from tabulations he believed to be "uncalibrated" and "based solely on subjects surveyed leaving the polling place" that had appeared on November 3, 2004 at 12:21 a.m. Rather than showing a "commanding lead," Freeman's numbers showed Bush very narrowly ahead of Kerry in Florida (49.8% to 49.7%) and Colorado (49.9% to 48.1% -- see the Simon/Freeman data on p. 15).  



That is a misleading statement. It ignores Freeman’s reference to the unadjusted state exit poll WPEs. You chose to refer to the “composite" Florida exit poll which preceded the final match to the recorded vote. In his  book, Freeman uses the E-M provided Florida WPE  of 7.6% to show that Kerry led the unadjusted exit poll by 50.9-48.3%.  



Far more important, the Edison/Mitofsky report included the "Call 3 Best Geo Estimator" data for each state.  Again, these are the estimates used to make election "calls" as the polls closed based solely on the exit poll results.  They were absolutely not "'corrected' more closely resemble the certified vote tallies," because no such tallies existed at the time.  And these estimates showed Bush ahead of Kerry by 1.1 points in Florida and 5.5 percentage points in Colorado (p. 21).



The exit polls were in the process of being “forced to match” the bogus “certified” vote tallies. The unadjusted exit polls had Kerry leading by 52-47%. Do the math. The Best GEO adjusted estimate reduced his margin to 51-48%.



As long as we are on the subject, the Edison/Mitofsky report also includes the statistics necessary to do a far more precise calculation of the statistical significance of the projected Kerry leads in the states that Kerry ultimately lost.  The "t-scores" I calculate (0.4 for Iowa, 0.4 for Nevada, 1.4 for New Mexico and 1.7 for Ohio) indicate that Kerry's leads in these states were not significant even at a 95% confidence level.  Note that Edison/Mitofsky required a much more demanding 99.5% confidence level, among other criteria, before recommending that the networks project a winner (see the discussion of "critical value" on p. 7). 


So what on earth was Kennedy talking about?  It took some digging to figure out, but Freeman has apparently discarded numbers from his original paper and now believes he has better data.  This handout, from an October presentation, is what he now claims represents "how random samples of 114,559 voters nationwide said they voted for as they walked out of the voting booth."   And it purports to show Kerry ahead "beyond the polling margin of error" in Florida, Ohio, New Mexico and Nevada and ahead within the margin in Coloradoand Iowa



You resort once again to thrashing Dr.  Freeman.  You say he “purports” to show; no, he showed that you are wrong There were 114,000 sampled nationwide Mark, you can take your 99.5% confidence level right to the disinformation heap. The unadjusted exit poll discrepancies were extremely significant at the polling standard 95% confidence level. You are once again misdirecting by referring to the contaminated, adjusted 12:40 am composite numbers. Is Mark Lindeman telling you that Freeman was backtracking?  Lindeman used the same tactic when Ron Baiman of U.S. Count Votes refined his analysis. The process is called the scientific method. Lindeman never uses it.



And where did these numbers come from?  It is not obvious, but the handout indicates that Freeman took the average "within precinct error" (WPE) from the Edison/Mitofsky report for each state (which he trusts, apparently, but renamed as "Precinct Level Disparity"), divides it in half, adds half of the average precinct error to Kerry's official statewide vote, subtracts half the average precinct error from Bush's official statewide total, and describes the result as indicating what the "exit polls projected" about each race.   



Calculating the state unadjusted exit poll vote shares using the WPEs supplied by Mitofsky was a perfectly reasonable thing to do. So why didn't Mitofsky do it? It’s a simple calculation. Was it because it would show that the weighted average WPE was 7.4% not 6.5%? In other words, the discrepancies were greater than he reported them to be.



Now I can think of a whole host of reasons why that is a bad idea. The exit polls in 13 states -- including Florida, Colorado, Iowa, Neevada and New Mexico -- included telephone interviews of absentee voters for which WPE does not apply.  The exit pollsters weighted these absentee samples to match their estimate of what absentee voters would contribute to the total vote: 27% in Florida, 30% in Iowa and 50% in both Nevada and Colorado Many states, such as Florida, included over-samples of Latino or African America precincts that could have a disproportionate effect on WPE.  Finally, WPE is calculated by comparing the official count for each precinct to the exit poll result, but 29% of the sampled precincts nationwide were at polling places where two or more precincts voted. 


Steven Freeman apparently believes that he has found a better way to use exit polls to determine which candidate won the election.   Note, however, that his novel extrapolation requires access to the official precinct level count -- something that was obviously impossible to obtain before the polls closed.  Some observers might accept that Freeman's approach represents an improvement on the projections developed and applied by all the statisticians and analysts at all of the networks.  But one thing is absolutely clear:  The numbers in Freeman's handout are not the estimates that those analysts used on Election Night.



That begs the issue.  The exit polls the analysts used were already “contaminated”.  Had the unadjusted numbers been made available, Kerry's margin would have been even greater than the composites indicated. Steve Freeman used the state WPEs to calculate the actual aggregate exit poll shares. So where's the beef, Mark? Why do you knock Freeman for doing calculations that Mitofsky and you avoided?  Do you have a better way of calculating the unadjusted vote shares? You are blowing smoke by just repeating what the other Mark is telling you.



And there is also the issue of sampling error.  Even if we accept Freeman's "projections," he provides absolutely no documentation or explanation in the handout for how he calculated statistical significance for these estimates. The statisticians I talked were absolutely stumped as to how Freeman concluded that the 2.6 percent "lead" he gives Kerry in Florida could be "beyond the polling margin of error" given his assumption of a 97.5% confidence level.



It's beyond the margin of error. Bush "won" FL by 52.1-47.1, but he only had 48.3% in the preliminary exit poll. That's a 3.8% deviation, way beyond the exit poll margin of error.  The final Florida exit poll was skewed as usual to match the recorded vote count. For example, Bush’s FL approval was set to 53% when it was actually closer to his 48% national rating. That is just another example of how bogus demographic weightings were used to force a match to the recorded vote share. Furthermore, pre-election Florida polls (see the table below) indicate that Kerry’s lead was increasing right up to the election.



And one more thing: After telling us about Kerry's supposedly "commanding lead" in Florida, Kennedy reverts in the very next paragraph to the numbers from Freeman's original paper (that showed discrepancies between the exit poll and the vote in Ohio, Pennsylvania and Florida of 6.7, 6.5 and 4.9 respectively - I will take up that paragraph in Part IV).  These were the same numbers that I used to calculate margins of error back in 2004, and thus the same numbers that Kennedy condemned on Wednesday as "corrected" by Mitofsky "to more closely resemble the certified vote tallies".



Wrong. The WPD's were 10.9 in Ohio, 8.8 in PA and 7.6 in FL.



So perhaps before lecturing Farhad Manjoo and the readers of Salon about how "hard" it is "to engage in an honest debate with someone who won't debate the same set of numbers," Robert Kennedy might want to decide which set of numbers he wants to debate. 


Nonsense. You appear frustrated here. Is that you speaking or is it the other Mark? Sounds like the other Mark.



''Either the exit polls, by and large, are completely wrong,'' a Fox News analyst declared, ''or George Bush loses.''



But wait, did some anonymous network "analyst" betray a belief among the pollsters that their numbers pointed to a Kerry victory?  Hardly. The "Fox News analyst" in question was Susan Estrich, a Democrat and Michael Dukakis's campaign manager in 1988 (the Fox clip was rebroadcast on CNN and quoted in the Washington Post).   Moreover, according to the Fox transcript (via Nexis), a few moments later Estrich added, "there are years in which exit polls have been wrong."



You quote Estrich as your polling expert from Fox News? It’s funny how you always quote a “Democrat” as if that makes them credible. Have you ever heard of Blue Dogs? Or DINOs?



Yes, the exit polls did significantly overstate Kerry's vote in Ohio and nationwide and, yes, they did show Kerry with a statistically significant lead in the national popular vote. And, as the Washington Post's Richard Morin put it in right after the election, those discrepancies were "just enough to create an entirely wrong impression about the direction of the race in a number of key states and nationally."  Many other cable news talking heads were fooled, as were many reporters, editors, campaign staffers and millions of ordinary Americans.  That result is certainly a problem for the exit polls and the journalists that depend on them (a topic I considered in this post).



Wrong impression? No, it was the bogus vote count that created the wrong impression. Once again,  the same mantra...the exit polls were wrong, the vote count was right. How do you know that, Mark? Where is your proof?



But the main point here is that the network analysts and exit pollsters were not fooled when they looked at their estimates on Election Night.  They missed no projections.  Robert Kennedy tries very hard to create the opposite impression, perhaps because the fact that the exit polls lacked the statistical power to project a winner makes it tough to show that the election was stolen.



They missed no projections? Are you forgetting that as late as 12:22am, E-M was projecting that Kerry would win by 51-48%? In thrashing RFK, Jr., Farhad Manjoowas exposed for getting almost everything wrong.  And you have relied on the same “expert” he did.  



Continues with Part IV, which discusses the larger issue - whether the statistically significant discrepancies between the exit polls and the vote count present any evidence of fraud.  I am grateful to Mark Lindeman and Elizabeth Liddle for their helpful comments on drafts of this post. 



In 2006, when you wrote this, you knew that they consulted or worked for Mitofsky. Are you still grateful today?


The following were comments by readers of the MP blog:


I'll just chime in that, based on what I've seen of her statements (see ).  Susan Ostrich is definitely a Fox shill today--so Dukakis paid her in 1988, so what--Fox is paying her now.

Posted by: | Jun 12, 2006 2:02:40 AM


MP, no it's...ML

I don't think it really matters much what Susan Estrich is. It doesn't even matter much that Kennedy cherry-picked the Estrich quotation. The quotation simply doesn't provide any guidance on how to interpret the exit poll results.



ML? Where is MP? It sure does provide guidance.  Estrich is another DINO, just like Dick Morris and Joe Lieberman.



That said, even before Estrich said this on Fox (between 8 and 9 Eastern, much closer to 9), Bill Kristol had said (among many other things), "...there's a gap at least in a couple of states between what the exit polls are suggesting... and what the actual vote coming [sic from transcript] is suggesting.... You know, Virginia -- one reason we were so hesitant, I think, to call Virginia is what we do is we take the exit polls and we match it [sic] up with the precincts. If the precincts confirm the exit polls, we figure, OK, the exit poll is right. In this case, the precincts didn't confirm the exit polls."


This quotation obviously doesn't tell us what Really Happened in Virginia. But -- yes, even coming from the mouth of Bill Kristol -- it does give some sense of the gap beetween how Kennedy apparently wants people to think about exit polls, and how the exit polls are actually used.


Posted by: Mark Lindeman | Jun 14, 2006 8:52:47 AM



Quoting Bill Kristol? Come on, Mark. Do you really want to do this? You place credence in what Bill Kristol has to say? And you call yourself a Democrat? And you believe that RFK is misleading people about the exit polls? What are we to conclude about your bias, ML?


Question to MP:

Do you know if any comparison was made using the "complete" exit poll data in the pollster's prediction model (appropriately scaling the data for geography, sex, age, etc) with the "actual" vote tallys?

Posted by: Bill R | Jun 9, 2006 11:11:15 AM



Bill, I for one am not sure what you are asking. Are you asking whether the state-level model predictions (prior to the inclusion of vote counts) have been compared with the official returns? The answer to that would be yes, for two Call 3 models (Best Geo and Composite); the results appear on pp. 21-22 of the E/M evaluation report. The average model error is smaller than the average Within Precinct Error (but not small), for whatever that is worth.


Posted by: Mark Lindeman | Jun 9, 2006 11:26:19 AM



2000 Exit Poll


State Aggregate

National   Gore        Bush        Margin     Error

Recorded 48.38%    47.87%     0.51%     -

WPE        49.38       46.87       2.51         2.01


New York

Recorded 60.2         35.2         25.0         -

WPE        61.9         33.5         28.3         3.3



Evaluation of Edison/Mitofsky Election System - January 19, 2005                   


2004 State Exit Poll Aggregate


National   Kerry       Bush        Margin     Error

Recorded 48.27%    50.73%     -2.46%    -    


 IMS        52.0         47.0         5.0           7.4  

 DSS        52.2         46.8         5.3           7.8

 VNS        51.8         47.2         4.7           7.1  


Best GEO    51.0     48.5          1.5           5.0 

Composite   50.3     49.1         1.2           3.8 



2004 Final National Exit Poll

(forced to match recorded vote)


National   Vote (mil)                                                Pct           Share (%)                                Vote (mil)                                

2000        Cast         Recorded Alive        Turnout   Mix          Kerry       Bush        Other       Kerry       Bush        Other       Total

DNV        -              -               -              20.8         17.0         54.0         44.0         2.0           11.2         9.1           0.4           20.8

Gore        55.4         51.0         48.5         45.2         37.0         90.0         10.0         0.0           40.7         4.5           0.0           45.2

Bush        51.4         50.5         47.9         52.6         43.0         9.0           91.0         0.0           4.7           47.9         0.0           52.6

Other       4.0           4.0           3.8           3.7           3.0           64.0         14.0         22.0         2.3           0.5           0.8           3.7


Final       110.8        105.4       100.1       122.3                       48.3         50.7         1.0           59.0         62.0         1.2           122.3


2000        Bush        Gore        Other       Recorded                48.3          50.7         1.0           59.0         62.0         1.2           122.3

Recorded 47.9         48.4         3.8           Diff                           0.0          0.0           0.0           0.0           0.0           0.0           0.0

ExitP        46.9         49.4         3.8           Exit Poll                   51.9         47.1         1.0           63.5         57.6         1.2           122.3

Cast         46.4         50.0         3.6           Diff                          -3.6          3.6           0.0           -4.5          4.5           0.0           0.0




2004 True Vote


2000                                                                                                                        Turnout in 2004          Unctd/stuffed      

Cast         Official Unctd          Alive        Cast         Official   Unctd        Mortality Gore         Bush        Gore        Bush        Other

110.8       105.4       5.4           105.3       125.7       122.3       3.4           6.1           98%         98%         75%         24%         1%

-               95.1%      4.9%        95.0%      -               97.3%      2.7%        5.0%        -               -               0%           100%       -


National   Vote (mil)                                                 Pct          Share (%)                                 Vote (mil)                                               

2000        Cast         Recorded Alive        Turnout   Mix          Kerry       Bush        Other       Kerry       Bush        Other       Total

DNV        -              -               -              22.6         17.9         57.0         41.0         2.0           12.9         9.3           0.4           22.6

Gore        55.1         51.0         52.3         51.3         40.8         91.0         8.0           1.0           46.6         4.1           0.5           51.3

Bush        51.6         50.5         49.0         48.0         38.2         10.0         90.0         0.0           4.8           43.2         0.0           48.0

Other       4.0           4.0           3.9           3.9           3.1           64.0         17.1         18.9         2.5           0.7           0.7           3.9


True        110.8       105.4       105.3       125.7                       53.1         45.5         1.3           66.8         57.3         1.7           125.7


2000        Bush        Gore        Other       Recorded                48.3          50.7         1.0           59.0         62.0         1.2           122.3

Recorded 47.9         48.4         3.8           Diff                           4.8          -5.2          0.3           7.8           -4.8          0.5           3.4

ExitP        46.9         49.4         3.8           Exit Poll                  51.9          47.1         1.0           63.5         57.6         1.2           122.3

Cast         46.4         50.0         3.6           Diff                          1.2           -1.5          0.3           3.3           -0.3          0.5           3.4



Sensitivity Analysis


Bush                        Gore voter turnout                                                    Share of                  Kerry Share of DNV                                               

turnout     90%         92%         94%         96%         98%                         Bush        55%         56%         57%         58%         59%


                                Kerry Share (%)                                                                                       Kerry Share (%)                                      

90%         53.4         53.7         54.0         54.3         54.6                         12%         53.5         53.7         53.9         54.1         54.2

92%         53.1         53.4         53.6         53.9         54.2                         11%         53.1         53.3         53.5         53.7         53.9

94%         52.7         53.0         53.3         53.6         53.8                         10%         52.8         52.9         53.1         53.3         53.5

96%         52.3         52.6         52.9         53.2         53.5                         9%           52.4         52.6         52.7         52.9         53.1

98%         52.0         52.3         52.6         52.8         53.1                         8%           52.0         52.2         52.4         52.5         52.7


                                Kerry Margin (mil.)                                                                                  Kerry Margin (mil.)

90%         10.40       11.10       11.80       12.50       13.20                       12%         10.55       11.00       11.46       11.91       12.36

92%         9.48         10.18       10.88       11.58       12.28                       11%         9.59         10.04       10.49       10.95       11.40

94%         8.56         9.26         9.96         10.67       11.37                       10%         8.63         9.08         9.53         9.98         10.44

96%         7.65         8.35         9.05         9.75         10.45                       9%           7.67         8.12         8.57         9.02         9.47

98%         6.73         7.43         8.13         8.83         9.53                         8%           6.71         7.16         7.61         8.06         8.51



                                Kerry Electoral Vote                                                                                 Kerry Electoral Vote              

90%         390          390          390          398          398                          12%         390          390          390          390          398

92%         379          390          390          390          398                          11%         390          390          390          390          390

94%         351          379          390          390          390                          10%         360          373          379          390          390

96%         351          351          370          390          390                          9%           351          351          351          357          379

98%         346          351          351          351          379                          8%           346          351          351          351          351




New York State Exit Polls


Recorded 58.4         40.1         18.3         -    


 IMS        64.5         34.0         30.5         12.2  

 DSS        64.3         34.1         30.2         11.9  

 VNS        64.1         34.4         29.3         11.4  


Best GEO    65.1      33.8        31.3         13.0 

Composite   63.1     35.5         27.6        9.3 




New York State

Initial (lever) votes vs. Late (paper ballots)

17,553 Precincts                                                                                


2000        Total        Gore        Bush        Other       Gore        Bush        Other


Lever   6,270           3,747       2,222       300          59.8%      35.4%      4.8%


Paper     552              361          181        11           65.4%      32.7%      1.9%


Final        6,822       4,108       2,403       311          60.2%     35.2%     4.6%



2004    Total            Kerry       Bush        Other       Kerry       Bush        Other


Lever   6,892           3,993       2,796       104          57.9%      40.6%      1.5%


Paper     499              321          167        10           64.3%      33.6%      2.1%


Final   7,391            4,314       2,963       114          58.4%     40.1%     1.5%


Top 15 NY counties

Election Day (Lever) votes                                             



Total        Kerry       Bush        Nader/other

5267        3253        1943        72

Share       61.8%      36.9%      1.4%



Total        Gore        Bush        Nader/other

4742        3045        1482        215

Share       64.2%      31.3%      4.5%


Net Change             

525          208          461          -143

Share       -2.4%       +5.6%      -3.1%



NY 2004 pre-election polls

Undecided voter allocations: Kerry 75%; Bush 25%


                Kerry       Bush        Kproj       BProj       Kma         Bma

2-Jul        55            36            61.00       38.00       61.00       38.00

13-Jul      58            30            66.25       32.75       63.63       35.38

20-Jul      51            29            65.25       33.75       64.17       34.83

14-Aug    53            35            61.25       37.75       64.25       34.75

2-Sep       56            37            60.50       38.50       62.33       36.67


17-Sep     48            40            56.25       42.75       59.33       39.67

22-Sep     55            39            58.75       40.25       58.50       40.50

27-Sep     51            31            63.75       35.25       59.58       39.42

3-Oct       53            41            56.75       42.25       59.75       39.25

11-Oct     53            41            56.75       42.25       59.08       39.92


16-Oct     58            35            62.50       36.50       58.67       40.33

24-Oct     57            36            61.50       37.50       60.25       38.75

27-Oct     54            38            59.25       39.75       61.08       37.92

28-Oct     52            37            59.50       39.50       60.08       38.92

29-Oct     57            39            59.25       39.75       59.33       39.67


Avg         54.07       36.27       60.57       38.43       60.74       38.26





Florida Recorded Vote (in thousands)

2000        Vote         Pct           2004        Vote         Pct

Gore        2912        48.8%      Kerry      3584        47.1%

Bush        2913        48.8%      Bush       3965        52.1%

Other       139          2.4%        Other      62            0.8%


Recorded Vote   by County Type

County     Vote   Kerry   Bush    Other   Kerry   Bush Other

DRE         3.90   51.3%   47.8%   0.9%    2.00    1.86 .04

OS           3.71   42.3%   57.0%   0.7%    1.57    2.11 .03


Total       7.61   47.1%   52.1%   0.8%    3.57    3.98 .06


County     Mix          Votes      Kerry       Bush        Other       Kerry       Bush        Other


Dem         41.57%    1.62        84%         15%         1%           1.36         0.24         0.02

Rep          36.13%    1.41        6%           93%         1%           0.08         1.31         0.01

Ind           22.30%    0.87        60%         38%         2%           0.52         0.33         0.02


Vote        3.903       3.90        50.5%     48.3%     1.2%       1.97         1.89         0.05



Dem         41.15%    1.53        71%         28%         1%           1.08         0.43         0.02

Rep          39.52%    1.47        5%           94%         1%           0.07         1.38         0.01

Ind           19.33%    0.72        60%         38%         2%           0.43         0.27         0.01


Vote        3.707       3.71        42.8%     56.0%     1.2%       1.59         2.08         0.04



Dem         41.37%    3.15        77.7%      21.3%      1.0%        2.45         0.67         0.03

Rep          37.79%    2.88        5.5%        93.5%      1.0%        0.16         2.69         0.03

Ind           20.85%    1.59        60.0%      38.0%      2.0%        0.95         0.60         0.03


Vote        7.610       7.61        46.7%     52.1%     1.2%       3.56         3.96         0.09



Florida Pre-election Polls

The final moving average projection: Kerry 51.1-48.8%


                                                Poll                                          Projection                                Moving Avg

Date         Pollster                    Kerry       Bush        Nader       Kerry       Bush        Nader       Kerry       Bush

23-May    Zogby                       49           48            1              50.4         48.6         1.0           50.4         48.6

31-May    Rasmussen              39            51            1              45.3         53.7         1.0           47.9         51.2

06-Jun     Zogby                      50            48            1              50.7         48.3         1.0           48.8         50.2

14-Jun     Survey USA             43            50            1              47.2         51.8         1.0           48.4         50.6

17-Jun     Rasmussen              48            44            1              52.9         46.1         1.0           49.3         49.7


20-Jun     Zogby                      46            50            1              48.1         50.9         1.0           49.1         49.9

22-Jun     Rasmussen              48            42            1              54.3         44.7         1.0           49.8         49.2

23-Jun     ARG                        47            46            1              51.2         47.8         1.0           50.0         49.0

27-Jun     Quinnipiac               43            43            5              49.3         45.7         5.0           49.9         48.6

30-Jun     Rasmussen              48            43            0              54.3         45.7         0.0           50.4         48.3


11-Jul      Survey USA             47            44            0              53.3         46.7         0.0           50.7         48.1

15-Jul      ARG                        47            44            3              51.2         45.8         3.0           51.3         47.4

21-Jul      LA Times                44            45            2              50.3         47.7         2.0           51.2         47.3

22-Jul      Gallup                      46            50            1              48.1         50.9         1.0           51.3         47.2

23-Jul      Zogby                      48            49            1              49.4         49.6         1.0           51.0         47.6


30-Jul      Zogby                      50            47            2              50.7         47.3         2.0           51.2         47.2

05-Aug    ARG                        50            43            2              53.5         44.5         2.0           51.1         47.2

10-Aug    Quinnipiac               47            41            4              52.6         43.4         4.0           51.3         46.7

21-Aug    Zogby                      50            49            0              50.7         49.3         0.0           51.4         47.1

22-Aug    Gallup                      46            48            2              48.8         49.2         2.0           51.2         47.3


24-Aug    Rasmussen              47            49            2              48.4         49.6         2.0           50.9         47.5

25-Aug    Research2k              46            46            2              50.2         47.8         2.0           50.6         47.7

11-Sep     Rasmussen              47            48            1              49.8         49.2         1.0           50.3         47.9

14-Sep     Survey USA             45            51            0              47.8         52.2         0.0           50.0         48.4

16-Sep     Rasmussen              47            48            0              50.5         49.5         0.0           50.0         48.5


17-Sep     Zogby                      48            48            1              50.1         48.9         1.0           50.2         48.4

20-Sep     ARG                        46            45            2              50.9         47.1         2.0           50.3         48.2

22-Sep     Gallup                      45            47            2              49.2         48.8         2.0           50.2         48.3

26-Sep     Rasmussen              49            48            0              51.1         48.9         0.0           50.0         48.7

27-Sep     Gallup                      44            49            2              47.5         50.5         2.0           49.6         49.3


29-Sep     Rasmussen              47            50            0              49.1         50.9         0.0           49.5         49.4

03-Oct     Survey USA             46            51            0              48.1         51.9         0.0           49.4         49.6

04-Oct     Rasmussen              46            52            0              47.4         52.6         0.0           49.3         49.9

05-Oct     Mason-Dixon          44            48            0              49.6         50.4         0.0           49.3         50.1

05-Oct     ARG                        47            45            2              51.2         46.8         2.0           49.4         49.9


05-Oct     Zogby                      50            49            1              50.0         49.0         1.0           49.6         49.6

05-Oct     Rasmussen              45            52            0              47.1         52.9         0.0           49.3         49.9

10-Oct     Rasmussen              45            49            0              49.2         50.8         0.0           49.2         50.1

10-Oct     Wash Post                47            47            1              50.5         48.5         1.0           49.2         50.2

14-Oct     Rasmussen              46            48            0              50.2         49.8         0.0           49.3         50.3


16-Oct     Mason-Dixon          45            48            0              49.9         50.1         0.0           49.2         50.4

17-Oct     Survey USA             50            49            0              50.7         49.3         0.0           49.4         50.3

18-Oct     Zogby                      49            50            0              49.7         50.3         0.0           49.5         50.2

18-Oct     Rasmussen              47            47            0              51.2         48.8         0.0           49.7         49.9

21-Oct     Research 2000         48            47            2              50.1         47.9         2.0           50.0         49.6


23-Oct     Rasmussen              48            48            0              50.8         49.2         0.0           50.1         49.5

24-Oct     Survey USA             50            48            0              51.4         48.6         0.0           50.1         49.6

25-Oct     ARG                        49            46            0              52.5         47.5         0.0           50.3         49.5

26-Oct     Quinnipiac               44            44            1              51.7         47.3         1.0           50.7         49.0

26-Oct     Rasmussen              48            48            0              50.8         49.2         0.0           50.8         48.9


27-Oct     Zogby                      46            48            0              50.2         49.8         0.0           50.8         49.0

27-Oct     NY Times                48            47            2              50.1         47.9         2.0           50.8         48.8

28-Oct     Rasmussen              46            49            0              49.5         50.5         0.0           50.7         48.9

29-Oct     Mason-Dixon          45            49            0              49.2         50.8         0.0           50.6         49.0

29-Oct     Zogby                      47            45            0              52.6         47.4         0.0           50.8         48.7


29-Oct     Rasmussen              47            48            0              50.5         49.5         0.0           50.8         48.8

30-Oct     Gallup                      49            45            0              53.2         46.8         0.0           51.0         48.7

30-Oct     Zogby                      49            47            0              51.8         48.2         0.0           51.1         48.6

30-Oct     Rasmussen              47            49            0              49.8         50.2         0.0           51.0         48.8

31-Oct     Opinion Dyn            49            44            1              53.2         45.8         1.0           51.1         48.6


31-Oct     Survey USA             48            49            0              50.1         49.9         0.0           50.9         48.8

31-Oct     Zogby                      48            47            0              51.5         48.5         0.0           51.0         48.8

31-Oct     Rasmussen              47            50            0              49.1         50.9         0.0           50.9         48.9

01-Nov    ARG                        50            48            0              51.4         48.6         0.0           51.0         48.9


01-Nov    Zogby                      48            48            0              50.8         49.2         0.0           51.1         48.8




2004 Florida Exit Poll Timeline


Poll                          Kerry      Bush

WPE (7.8%)           51.0%     48.2%   <unadjusted

Best GEO                 49.2         50.3   

Composite               49.3         50.1

Best SPM 48.6         51.0   

Recorded 47.1         52.1


Florida Recorded Vote by County Voting Method


County                     Mix          Vote         Kerry       Bush        Other       Kerry       Bush        Other

DRE                         51.3%      3.90         51.3%      47.8%      0.9%        2.00         1.86         0.04

OS                           48.7%      3.71         42.3%      57.0%      0.7%        1.57         2.11         0.03


Total                       100%      7.61         47.1%     52.1%     0.8%       3.57         3.98         0.06


Kerry won using exit poll shares and actual voter registration weights


Party-ID               Mix          Votes       Kerry       Bush        Other       Kerry       Bush        Other

Dem                         41.4%      3.15         86%         13%         1%           2.71         0.41         0.03

Rep                          37.8%      2.88         7%           92%         1%           0.20         2.65         0.03

Ind                           20.9%      1.59         60%         38%         2%           0.95         0.60         0.03


Vote                        7.61         7.61         50.7%     48.1%     1.2%       3.86         3.66         0.09


Party-ID               Mix          Votes       Kerry       Bush        Other       Kerry       Bush        Other

Dem                         38%         2.89         86%         13%         1%           2.49         0.38         0.03

Rep                          39%         2.97         7%           92%         1%           0.21         2.73         0.03

Ind                           23%         1.75         60%         38%         2%           1.05         0.67         0.04


Total                       7.61         7.61         49.2%     49.6%     1.2%       3.74         3.77         0.09



Bush needed 21% of the statewide Democratic vote


Total                       Mix          Votes       Kerry       Bush        Other       Kerry       Bush        Other

Dem                         41.4%      3.15         77.7%      21.3%      1.0%        2.45         0.67         0.03

Rep                          37.8%      2.88         5.5%        93.5%      1.0%        0.16         2.69         0.03

Ind                           20.9%      1.59         60.0%      38.0%      2.0%        0.95         0.60         0.03


Vote                        100%      7.61         46.7%     52.1%     1.2%       3.56         3.96         0.09


Bush needed 15% of the Democratic vote in DRE counties


DRE                        Mix          Votes       Kerry       Bush        Other       Kerry       Bush        Other

Dem                         41.6%      1.62         84%         15%         1%           1.36         0.24         0.02

Rep                          36.1%      1.41         6%           93%         1%           0.08         1.31         0.01

Ind                           22.3%      0.87         60%         38%         2%           0.52         0.33         0.02

Vote                        100%      3.90         50.5%     48.3%     1.2%       1.97         1.89         0.05


Bush needed 28% of the Democratic vote in Op Scan counties


Op Scan

Dem                         41.2%      1.53         71%         28%         1%           1.08         0.43         0.02

Rep                          39.5%      1.47         5%           94%         1%           0.07         1.38         0.01

Ind                           19.3%      0.72         60%         38%         2%           0.43         0.27         0.01

Vote                        100%      3.71         42.8%     56.0%     1.2%       1.59         2.08         0.04



Florida Preliminary and Final Exit Poll


Bush Approval: 53%


Approval                 Mix          Kerry       Bush        Other

Strong                      35            4              96            0

Approve  1              18            17            82            1

Disapprove              12            84            13            3

Strong                      35            98            1              1


Total                       100          48.8         50.3         0.9


Bush Approval: 53%


Strong                      33            5              94            1

Approve                  20            15            83            2

Disapprove              12            80            18            2

Strong                      35            97            2              1


Total                       100          48.2         50.5         1.3


Bush Approval: 48.5%


Approval Mix          Kerry       Bush        Other

Strong                      30.5         4              96            0

Approve                  18            17            82            1

Disapprove              14            84            13            3

Strong                      37.5         98            1              1


Total                       100          52.8         46.2         1.0


Bush Approval: 48.5%


Strong                      30.5         5              94            1             

Approve                  18            15            83            2             

Disapprove              14            80            18            2             

Strong                      37.5         97            2              1             


Total                       100          51.8         46.9         1.3          


When Decided

Decided                 Mix          Kerry       Bush        Other

3 days                      8%           53%         45%         2%

Week                      3              70            27            3

Month                      12            61            38            1

Before                     77            46            54            0 does not match pre-election poll


Total                       100          49.1         50.6         0.4

                                7610        3735        3847        28


3 days                      8%           53%         45%         2%

Week                      3              70            27            3

Month                      12            61            38            1

Before                     77            50            50            0 pre-election poll match


Total                       100          52.2         47.5         0.4

                                7610        3969        3612        28