home

Daily Kos Sues Research 2000

Markos Moulitsas and Daily Kos have filed their federal lawsuit against Research 2000 alleging the weekly polls Daily Kos commissioned from them included phony and falsified data. You can read the lawsuit here. Attorney Adam Bonin (pro hac vice) and Nathan Dooley of COZEN O'CONNOR in LA are representing Markos and Kos Media, LLC in the suit which has been filed in the federal district of Northern California.

A few days ago, I read some blogposts on the suit and got the general gist of what happened, but not being a statistician, I got lost by reading their findings.

Adam and Nathan do a great job with the civil complaint. They understand judges aren't statisticians, and write in simple language that lawyers and lay people can understand. (Theoretically, after all, since these cases are usually settled, juries don't hear most of them. But you can never be sure, so you better be able to break down the tech talk to something they can relate to. Adams succeeds very well at this.

Here's how. [More...]

The text of the lawsuit, thanks to Greg Sargeant of the Plum Line at the Washington Post is here. On June 6, 2010, noted statistician and political analyst Nate Silver of FiveThirtyEight.com and the New York Times published a comprehensive article titled “Pollster Ratings v4.0: Results,” ranking all political polling firms based on detailed statistical analysis of the accuracy of their polling. Of the sixty-four polling firms which had conducted ten or more polls which Silver ranked, Research 2000 was among the five worst in terms of accuracy. http://www.fivethirtyeight.com/2010/06/pollster-ratings-v40-results.html">available on online.

Paragraph 22 begins with how the problems came to light:

On June 6, 2010, noted statistician and political analyst Nate Silver of FiveThirtyEight.com and the New York Times published a comprehensive article titled “Pollster Ratings v4.0: Results,” ranking all political polling firms based on detailed statistical analysis of the accuracy of their polling. Of the sixty-four polling firms which had conducted ten or more polls which Silver ranked, Research 2000 was among the five worst in terms of accuracy.

As part of his correspondence to Ali terminating the relationship, Moulitsas noted: “Currently, you owe us some polls. We won't pursue any refund. We'll just call it even.” Ali responded by writing, “Best of luck to you” and attached a lengthy self-defense of his polling methods, lashing out at the NYT’s Silver as “nothing more than a fringe blogger.” At neither that point nor at subsequent point did Ali communicate to Moulitsas any different understanding as to the parties’ financial obligations to each other.

On June 14, 2010, Moulitsas was approached by a number of independent statistical analysts with regards to Research 2000’s polling for DailyKos.Their analysis of the published data regarding the results which revealed that Research 2000 had almost certainly falsified the results in whole or in part.

As part of his correspondence to Ali terminating the relationship, Moulitsas noted: “Currently, you owe us some polls. We won't pursue any refund. We'll just call it even.” Ali responded by writing, “Best of luck to you” and attached a lengthy self-defense of his polling methods, lashing out at the NYT’s Silver as “nothing more than a fringe blogger.” At neither that point nor at subsequent point did Ali communicate to Moulitsas any different understanding as to the parties’ financial obligations to each other.

On June 14, 2010, Moulitsas was approached by a number of independent statistical analysts with regards to Research 2000’s polling for DailyKos. Their analysis of the published data revealed a number of statistical anomalies regarding the results which revealed that Research 2000 had almost certainly falsified the results in whole or in part.

Now, here's the rub:

The anomalies detected included that among 778 weekly polling questions regarding a politician’s “favorables” allegedly performed by Research 2000, the purported male and female sub-samples either came out both with even numbers or both with odd numbers 776 of 778 times; the “unfavorables” were both even or both odd 777 of 778 times. Since the odds of getting a match each time randomly is 50%, the odds of obtaining 776/778 matches is the odds of obtaining 776 heads on The only logical conclusion is that the numbers for there groups were not generated by independent polling but by falsification of data in whole or in part.

Even I can understand that. Then there's Ali's stall:

On Monday, June 14, 2010, Moulitsas approached Ali with his concerns. In response, Ali promised via email on that date to provide Moulitsas with the raw data from which the polling results were constructed to verify their validity.

On Wednesday, June 16, 2010, Ali emailed Moulitsas with regards to the raw data, stating that “you can expect it either Friday, if not, no later than Monday.”

On Monday, June 21, Ali again stalled, emailing Moulitsas to state: “I am at a Kinkos computer because we cannot read any mail from our PC,s [sic] and cannot attached [sic] any documents or files to send. Reading email from my cell. Just
got to a Kinkos. Computer Geeks cannot do anything until tomorrow morning 778 tosses of a fair coin, an event which should occur one in every 10 228 (ten followed by 228 zeroes) times. The likelihood of the unfavorable results having occurred by chance is one in every 10 231. The only logical conclusion is that the numbers for there groups were not generated by independent polling but by falsification of data in whole or in part.

As of this date, Wednesday June 30, 2010, despite numerous requests from Moulitsas and purported commitments from Ali to provide the raw data which could verify Research 2000’s polling’s validity, Ali has refused to provide this data.

Those are the facts that Dkos presents. Even I can understand them. I'm not sure what Research 2000 will say in response, but I'd be worried if I were them.

Granted, Markos has been a friend of mine since 2002, when we both began our blogs, and I'd take his side in just about anything. But reading through the lawsuit, with my lawyer hat on, I see difficulties ahead for Research 2000. I also think it was very well laid out by Adam (and Nathan, if he participated.)

This lawsuit could have tremendous repercussions for Research 2000. As Greg Sargant and indeed the company's owner points out, they do polling for a lot of MSM publications and their client base may start departing in droves if Markoos and Adam make their case -- even in the media. Knowing how smart and strategic the Markos and Adam are, I'm expecting them to do just that.

Good luck, Markos and Adam. TalkKeft is rooting for you.

< Friday Morning Open Thread | White House Fantasy: "Signs Of Gradual Labor Market Recovery" >
  • The Online Magazine with Liberal coverage of crime-related political and injustice news

  • Contribute To TalkLeft


  • Display: Sort:
    I suspect there is (5.00 / 1) (#1)
    by JamesTX on Fri Jul 02, 2010 at 05:26:55 AM EST
    more to it than the complaint suggests, which seems to contain the same statements Kos has posted on his site. The problem here is that professionals in that field are unlikely to make that kind of simplistic error if they are fudging data. If they were in fact "fabricating" results they would more than likely do it in a way that was not so obvious. They certainly would know better than to try to make up the numbers from scratch in the way it has been suggested throughout this scandal. If they have the least bit of understanding of statistics (which they must to have gotten as far as they did in the business), they would generate the data in ways that would not involve human bias -- such as the implied scenario of their "accidentally forgetting" that all the numbers can't be odd or even. That would be like suggesting Richard Nixon tried to get in and out of the Watergate by wearing a baseball cap as a disguise. First, he wouldn't have ever been there to begin with, and furthermore he would know a cap wouldn't work, or at least his advisors would know! I seriously doubt pollsters of this caliber would be that incompetent at cheating. If they were cheating, they would know how to cover their tracks a little better. There is likely and explanation coming for the odd and even sample sizes being what they are, and it is something the people who have been evaluating the reports have simply not considered. Citing the huge numbers (gazillions of zeros) for the probabilities of the results occurring by chance serves to hype the claim, and gives it that sort of awe inspiring Carl Sagan rhetorical quality, but we haven't heard from the other side yet. The people who came to the conclusions that the data were falsified are not experts in political or social science polling and measurement. IIRC, one is a physicist and the other a biological technician. Granted, those people elicit unconditional respect and awe as they are seen by the public at large as being more knowledgeable about math (and generally smarter to begin with) than social science people, but the fact is that their fields are very different and they use statistics in a very different way. They also frequently misunderstand some statistical concepts because those concepts are not relevant to their fields. They are no more qualified to critique polling results than a sociologist would be qualified to critique statistical results in a physics experiment.

    Know about paragraphs? (3.50 / 2) (#7)
    by Yes2Truth on Fri Jul 02, 2010 at 08:43:34 AM EST

    Your comments might gain in readership numbers
    if you would take the time to learn how to use
    short paragraphs the next time you desire to give
    TL readers the benefits of your "pearls".


    Parent
    Possibly, but this way James' comments (5.00 / 2) (#8)
    by observed on Fri Jul 02, 2010 at 08:47:13 AM EST
    are easy to spot, for those of us who want to read them.

    Parent
    ALL CAPS (none / 0) (#19)
    by Yes2Truth on Fri Jul 02, 2010 at 09:47:03 AM EST
     
    ARE EVEN EASIER TO SPOT.

    Parent
    Readership is not (5.00 / 1) (#66)
    by JamesTX on Fri Jul 02, 2010 at 02:20:53 PM EST
    something I want for, and really, don't shoot the messenger. I am largely playing the Devil's advocate here.

    I think there may be more to it than the complaint suggests, which seems to contain the same statements Kos has posted on his site. The problem here is that professionals in that field are unlikely to make that kind of simplistic error if they are fudging data. If they were in fact "fabricating" results they would more than likely do it in a way that was not so obvious. They certainly would know better than to try to make up the numbers from scratch in the way it has been suggested throughout this scandal.

    If they have the least bit of understanding of statistics (which they must to have gotten as far as they did in the business), they would generate the data in ways that would not involve human bias -- such as the implied scenario of their "accidentally forgetting" that all the numbers can't be odd or even. That would be like suggesting Richard Nixon tried to get in and out of the Watergate by wearing a baseball cap as a disguise. First, he wouldn't have ever been there to begin with, and furthermore he would know a cap wouldn't work, or at least his advisors would know! I seriously doubt pollsters of this caliber would be that incompetent at cheating. If they were cheating, they would know how to cover their tracks a little better.

    There is likely and explanation coming for the odd and even sample sizes being what they are, and it is something the people who have been evaluating the reports have simply not considered. Citing the huge numbers (gazillions of zeros) for the probabilities of the results occurring by chance serves to hype the claim, and gives it that sort of awe inspiring Carl Sagan rhetorical quality, but we haven't heard from the other side yet.

    The people who came to the conclusions that the data were falsified are not experts in political or social science polling and measurement. IIRC, one is a physicist and the other a biological technician. Granted, those people elicit unconditional respect and awe as they are seen by the public at large as being more knowledgeable about math (and generally smarter to begin with) than social science people, but the fact is that their fields are very different and they use statistics in a very different way. They also frequently misunderstand some statistical concepts because those concepts are not relevant to their fields. They are no more qualified to critique polling results than a sociologist would be qualified to critique statistical results in a physics experiment.

    Parent

    Nice Paragraphs (5.00 / 1) (#67)
    by squeaky on Fri Jul 02, 2010 at 02:23:15 PM EST
    Thanks, makes it much more enjoyable to read.

    Parent
    Maybe a little less tone (none / 0) (#10)
    by Big Tent Democrat on Fri Jul 02, 2010 at 09:08:15 AM EST
    with the advice next time.

    Parent
    This seems like a lawsuit within your (none / 0) (#11)
    by observed on Fri Jul 02, 2010 at 09:09:49 AM EST
    areas of expertise. Any opinion?

    Parent
    No opinion (none / 0) (#12)
    by Big Tent Democrat on Fri Jul 02, 2010 at 09:26:49 AM EST
    Markos is our friend.

    Parent
    Ok. Do you know of other lawsuits (none / 0) (#13)
    by observed on Fri Jul 02, 2010 at 09:32:37 AM EST
    against pollsters or firms providing some similar services.

    Parent
    No (none / 0) (#14)
    by Big Tent Democrat on Fri Jul 02, 2010 at 09:34:37 AM EST
    You sure you don't want a cease and desist.... (none / 0) (#26)
    by magster on Fri Jul 02, 2010 at 09:59:48 AM EST
    ... letter too?

    Parent
    Heh (none / 0) (#27)
    by Big Tent Democrat on Fri Jul 02, 2010 at 10:02:29 AM EST
    Nah. That does not worry me.

    I just don't think giving my opinion on this issue will be particularly enlightening.

    My opinion on the polling industry generally is a matter of record.

    Parent

    Well, yeah, but: (none / 0) (#28)
    by andgarden on Fri Jul 02, 2010 at 10:08:31 AM EST
    a client walks into his lawyer's office and says "I want to sue a fly-by-night company that sold me trash for lots of cash."

    I'm sure you must have an opinion on that--unrelated to the quality of the polling.

    Parent

    No opinion (none / 0) (#42)
    by Big Tent Democrat on Fri Jul 02, 2010 at 11:01:06 AM EST
    will be expressed by me.

    Parent
    Understandable (none / 0) (#45)
    by andgarden on Fri Jul 02, 2010 at 11:04:48 AM EST
    i don't know (none / 0) (#9)
    by pitachips on Fri Jul 02, 2010 at 08:49:45 AM EST
    People make up stuff all the time - especially if they believe (and in the polling industry it's not an unreasonable belief since it's totally unregulated) that they will never be called out on it.  

     

    Parent

    Statistical analysis (none / 0) (#72)
    by JamesTX on Fri Jul 02, 2010 at 03:03:01 PM EST
    is, at its core, all about "making things up". It is the way the things are made up that matters. It remains to be seen if R2k followed the rules.

    Parent
    The other side sent a kinda unhinged (none / 0) (#15)
    by magster on Fri Jul 02, 2010 at 09:34:57 AM EST
    e-mail to TPMMuckraker last night, which 538 has already analyzed.

    Essentially the defense is Ali believes his social science degree allows him to fudge the topline numbers within the margin of error according to his expertise, and then fit the crosstab numbers to fit the new topline number. All justified by his expertise in the field.

    Pretty lame defense IMO.

    Parent

    links (none / 0) (#17)
    by magster on Fri Jul 02, 2010 at 09:45:03 AM EST
    What's the quantitative measure (none / 0) (#18)
    by observed on Fri Jul 02, 2010 at 09:45:15 AM EST
    of the error?
    Some of the numbers could be wildly improbable, but still accurate.
    Suppose every number ends with .*2. This is virtually impossible, if the numbers are random.
    Suppose this is a glitch in a number smoothing scheme that results in no error, on average.
    Where's the harm or the fraud?

    To me, fraud would mean not even conducting the polls, or taking the raw results and  weighting them according to highly unrealistic assumptions.
    Furthermore, the result ought to be measurably wrong.

    Parent

    So the results are fake but accurate? (5.00 / 1) (#60)
    by sarcastic unnamed one on Fri Jul 02, 2010 at 12:09:00 PM EST
    No (none / 0) (#61)
    by squeaky on Fri Jul 02, 2010 at 12:13:46 PM EST
    The poll results were out of wack several times, which is why the statisticians started looking into it.

    Parent
    I don't know. I'm just saying that the (none / 0) (#62)
    by observed on Fri Jul 02, 2010 at 12:13:58 PM EST
    non-randomness doesn't show per se that the polls are way off.


    Parent
    Fair enough. (none / 0) (#65)
    by sarcastic unnamed one on Fri Jul 02, 2010 at 12:47:59 PM EST
    In the 538 post (none / 0) (#22)
    by magster on Fri Jul 02, 2010 at 09:56:33 AM EST
    Silver said that if the raw data showed a dem winning 46-44 with a +/- 4 percent, Ali could say to himself "gee that doesn't feel right because of my expertise" and produce a result of the dem winning 50-42 because it is a result within the margin of error of his own poll.  Then he'd rework all the crosstabs to fit the new 50-42 result.

    I suppose that could be a defense to fraud if he actually conducted the poll, but if there's an accepted standard of quality within the field of polling that's implied in the contract, I'd imagine R2k's method is "just a bit outside"

    Parent

    Nate Silver saying that Ali "could (none / 0) (#25)
    by observed on Fri Jul 02, 2010 at 09:58:52 AM EST
    say to himself" something like that is not remotely convincing.


    Parent
    ?? I don't follow... (none / 0) (#30)
    by magster on Fri Jul 02, 2010 at 10:25:11 AM EST
    For any data set (none / 0) (#70)
    by JamesTX on Fri Jul 02, 2010 at 02:42:42 PM EST
    there are a very, very large number of the kinds of pattern criteria which can be applied, such as the .*2 you mention. A full five percent of those will be outside the 95% confidence interval and some will have a probability of one over gazillions and gazillions. All you have to do is find whatever is unique about the particular dataset, and then claim it couldn't have happened that way. You can find it in lottery numbers.

    Parent
    I'm not sure that (none / 0) (#69)
    by JamesTX on Fri Jul 02, 2010 at 02:39:08 PM EST
    is the defense. Ali doesn't write well (although he uses paragraphs). He says there is more to come. I haven't studied the data so I don't know exactly what he is saying, but he mentioned weighting. I am sure weighting is common in the polling field by the very nature of what they have to do and the time lines they work on.

    Parent
    Paragraphs are overrated (none / 0) (#89)
    by Militarytracy on Sat Jul 03, 2010 at 11:30:58 PM EST
    I am an expert in rating the public and private need for paragraphs :)  I'm board certified by the National Association of Paragraph Need Raters :)  I'm getting bored with it though.  Next year I think I want to be an ophthalmologist.

    Parent
    WOW! (none / 0) (#88)
    by Militarytracy on Sat Jul 03, 2010 at 11:26:20 PM EST
    I'm no lawyer.  But did the dumbarse just turn the burner on high and sit on it and fry his own arse or what?  He's a diety, who knew?  If I've read correctly he is no degreed expert in this field.  So he's an expert human ouija?

    Parent
    I know this sounds (5.00 / 2) (#90)
    by JamesTX on Sun Jul 04, 2010 at 02:09:26 AM EST
    naive, but "expertise" and "discretion" are not always synonyms for arbitrary and capricious. Sometimes those terms are used as placeholders for complex methods that would take too long to explain and would distract or lose the attention of the current audience.

         Asking a surgeon why they made a specific decision in a given case may well result in an answer like "expertise" or "discretion", when the decision was based on objective defensible facts. If the audience isn't capable of understanding the technical knowledge on which the decision is based, you lose the argument in the public forum. This is what I was talking about with the favoring of simplicity.

         The "expertise" and "discretion" terms may mean more than just picking the numbers he liked, and Ali seems to leave that as a distinct possibility. For instance, if the estimates were based on some method, such as some kind of resampling algorithm, some people would say the estimates were better than the raw data. Given that, if they are within the raw data confidence interval, he may have an argument.

         If someone maliciously attacked his honesty and integrity, why wouldn't he bait them with that kind of talk while organizing his plan to slaughter them when it mattered. He's not going to get anywhere trying to describe advanced statistical methods to blog audiences. Even if he could, why give up his trade secrets? He's dead in the progressive press anyway, because Kos, not Ali, is the "diety" in that world. He may be choosing his battles. If Kos said Barack Obama was a woman, thousands of followers would troll rate any comments which referred to the president with the pronoun "he".

         Ali doesn't have a chance in the progressive blog forum, and he likely knows it. So why give up his defense before discovery? If he successfully made the argument and convinced them to drop the suit, he is still ruined, especially if the public can't comprehend the statistics on which his reports were based. They will assume what they have been taught to assume in such cases. They will say he "got off on technicalities", or Kos just cut his losses because Ali's pockets weren't deep enough. His only choice is to go to trial and win, and stand on the courthouse steps yelling he's ready for trial if they try to drop the suit. Of course, IANAL, and I am sure there are other options.

         I've been thinking of presenting my new paragraph markup technique to the association. Do you think it would be accepted? Is this coming out as garbage on anyone's browser? We aim to please.



    Parent
    I can respect where you are coming (5.00 / 1) (#91)
    by Militarytracy on Sun Jul 04, 2010 at 08:53:24 AM EST
    from, and I don't doubt that this is how it could be.  I'm stuck though with an expert who couldn't come up with the raw data for the people who paid for it when this was "sprung" on him.  Then we are told this by him.  If there some factually based reason, considering that he stands the chance of being literally cruicified professionally here, based on the flipping coin in my head if he has a scinlitta of the intellect he has been said to have this could not have been his first choice on how to defend the questioning :)  But I realize my flipping coin in my head is not the same flipping coin in his head :)

    Parent
    Good luck to Ali. (5.00 / 1) (#2)
    by observed on Fri Jul 02, 2010 at 06:06:51 AM EST
    I'm skeptical of the lawsuit for reasons similar to JamesTX. And Kos may be a nice fellow, but AC Bonin  most certainly is not , judging from what I saw on DK when I was ther(For instance, he repeatedly brought up Juanita Broaddrick as evidence of something. For a lawyer, I thought that was disgraceful. His post "My great Law teacher" also set a new bar in simpering sycophancy) , so I'd love to see him get wiped out.


    R2K is doomed (5.00 / 1) (#20)
    by me only on Fri Jul 02, 2010 at 09:49:01 AM EST
    Markos is going to try this in the court of public opinion.  The lawsuit is a vehicle to keep the story going.  R2K's clients will probably bail long before this sees a courtroom.

    That almost all of Markos' posts over the last couple of years are based on lousy polling should be something he is embarrassed about.

    But the lawsuit only refers to the (none / 0) (#21)
    by observed on Fri Jul 02, 2010 at 09:54:34 AM EST
    weekly State of the Nation polls.
    It doesn't include the battleground staate polling, for example.


    Parent
    Nate Silver's ranking of (none / 0) (#31)
    by me only on Fri Jul 02, 2010 at 10:26:19 AM EST
    pollsters would seem to indicate that R2K was one of the "not so good ones."

    Parent
    Yes, which I suspect is (none / 0) (#68)
    by JamesTX on Fri Jul 02, 2010 at 02:24:59 PM EST
    really what this is all about. As R2K points out, the same kinds of anomalies can be found in other polls, but they predicted better so nobody cares. Hindsight bias is what it is called (the "I knew it all along" phenomenon).

    Parent
    Really (none / 0) (#71)
    by squeaky on Fri Jul 02, 2010 at 02:47:29 PM EST
    I thought that there were allegations of using a randomizing program to save on the expense of doing actual polling.

    At least that is how I understood the basis of the claim. Fraud.

    Parent

    Again, I don't (none / 0) (#73)
    by JamesTX on Fri Jul 02, 2010 at 03:16:02 PM EST
    know anything about the data yet, but if they generated it from a random number generator then odd and even thing seems unlikely. I will have to look at exactly what is being critiqued, but it sounds like something related to weighting to me. That is not always wrong. It depends on what they had.

    Parent
    Your comments take me back (none / 0) (#74)
    by Cream City on Fri Jul 02, 2010 at 03:34:10 PM EST
    to stats class.  But I forgive you. :-)

    And what you are saying here, in comment after comment, does refresh my memory sufficiently to suggest that I think you are correct in raising all sorts of possibilities . . . as a defense could do, and probably pretty effectively.

    Above all, what I learned, is that numbers can be made -- much like pols -- to say anything.  And especially when weighting and other methods are used (and as you say, I also have read that it is a common and even requisite methodology in polls).

    In sum, it does seem that Markos may not have much of a case -- although, of course, that means when in court.  If his intent was to ruin the firm's reputation, he already has accomplished that and can drop the case without more legal costs.

    But then, as we discuss further down in this thread, the cost to Markos' and DKos' reputations does not make this seem a wise move in the end.

    Parent

    I think its a strong case just reading ... (none / 0) (#77)
    by magster on Fri Jul 02, 2010 at 04:21:36 PM EST
    ... Ali's rant on TPM.  He admits to "heavy weighting" of the top line number and then working backwards to make the cross tabs fit the top-line result based on nothing more than his claimed expertise in political science.  Why even commission a poll from this guy instead of just giving him a Kos user name to write a diary on why a PPP poll is wrong.

    Kos always insisted on posting the cross-tabs to give legitimacy and transparency to his polls without having been told by his client that the cross-tabs were manipulated to fit an artificial top-line number that could be anywhere within an 8% margin of error.  Sounds like fraud to me, or at least breach of contract.

    I wonder what other media organizations have been "KOS'd!!" and whether everyone will be demanding more data from their pollsters.

    Parent

    Randomizer (none / 0) (#76)
    by squeaky on Fri Jul 02, 2010 at 03:51:55 PM EST
    Oh, I just reread the dkos post, and randomizer is not the right word, but it appears that instead of doing the usual footwork, they used may have used a computer program that repeated various poll results that are virtually impossible to repeat even with rounding, and adjusting.

    None of the links seem to be working above, so here is where I originally saw the story reflecting the quotes above:
    dkos

    Parent

    There are all kinds of (none / 0) (#79)
    by JamesTX on Fri Jul 02, 2010 at 05:51:47 PM EST
    things they could be doing, as pollsters no doubt use the latest in available techniques to get their work done. I'm hesitant to take Ali's initial comments as the whole explanation, as he made it clear that it wasn't. The explanation is likely to be complex, because the reason people farm this kind of work out is because it is complex. People imagine it in three simple steps from what they learned in some introductory class, but when they try to go do it, they run into a thousand questions at every turn. More than likely, their agreement is not specific enough to easily interpret what was offered or to say for sure what was offered was not delivered.

    Part of doing this type of work for pay in the market is that it is inherently a gamble. If your results are right, then nobody questions your work. If your results are not so good, then everyone thinks there is a simple explanation. The fact is poll results can be wrong. There is no guarantee. It is game of chance, literally.

    I think it is fair to say it is time for Ali to step up and explain what he did, and every professional in this line of work keeps the possibility of having to do that in mind at all times. He doesn't have to do it in the media, though. He has as much right as everyone else to plan and strategize regarding what he tells the media. Also, he probably doesn't even know all the facts about the poll because he probably hired someone to do it.

    The court of public opinion favors simplicity. If the facts are too complicated for the typical person to understand, Ali's task is to come up with a way to even the odds (pun) such that his explanation can match the force of the combination of Kos's popularity and his titillating Carl Sagan rhetoric, and can be just as convincing in the public forum which notoriously favors simplicity over truth. It is easier to understand Kos's yelling "fraud" as a lawyer and saying gazillions of zeros than it is to understand the more complex theory underlying frequency analysis, sampling, and statistical estimates. Ali is likely not empty handed, though, whether or not things will ultimately go his way. It's likely not as simple as the public would like to think.

    Parent

    Poll results really are not "wrong" (none / 0) (#82)
    by Cream City on Fri Jul 02, 2010 at 10:47:12 PM EST
    If your results are right, then nobody questions your work. If your results are not so good, then everyone thinks there is a simple explanation. The fact is poll results can be wrong. There is no guarantee. It is game of chance, literally.

    As you well know, you're talking about election-day results -- long after most polls are done.

    So polls that are reported as "wrong" often are unfairly blasted for being wrong on election day but were right on the day when the poll was done.

    Parent

    Point well made... (5.00 / 1) (#83)
    by JamesTX on Sat Jul 03, 2010 at 12:09:44 AM EST
    and if properly done they are actually never "wrong" even when they are not good predictors. They are always subject to that (typically) 5% outlier category, and the results mean exactly that -- 95% confidence. You can't be wrong if you never guaranteed you're right to begin with!

    Your point about the lag time and other things is one thing that bothered me on my initial scan of the Nate Silver statistics. That is complicated and I will have to take some time to understand it. In fact, I may give it to my class next term as an example of an ongoing controversy based on regression. The equation appears to be a very complicated thing that is full of dummy variables. I'm sure Silver is an expert, but interpreting results for dummy coding can be complicated.

    One thing I noticed right off is that the equation appears to put all those who are members of the organization (or whatever) at the lowest predicted values. I also noticed there is a big discrepancy between R2k's PIE and rawscore. The raw score doesn't look so bad. So, I'm not sure I understand how bad their predictions were to begin with.

    The point is that the fact that they were fingered by this very complex free lance evaluation and that report led to scrutiny of their results is sort of cheesy to begin with. How did they look for discrepancies and anomalies? How many things did they consider when looking for the "culprit"? How many things did they evaluate before they found the odd/even issue (which another commenter has already pointed out one good explanation for)? In fact, how many things would have "caught their attention"? Each one counts and reduces power.

    Looking for something post hoc in the results which would have a low probability of occurrence by chance alone amounts to multiple comparisons, and it starts cutting Kos's one-in-a-gazillion probability to pieces. In fact, how many things would have even been a candidate for being "suspicious"? Each one costs you power. That amounts to fishing if they were trolling for things that looked out of place, and fishing costs you statistical power.

    If there are 100 things that each have a 1 in 20 chance of being outliers, the probability that you will find one of them to be an outlier is almost 1. That is, some unpredictable one of the 100 things that are individually almost certain not to happen will almost certainly happen. The key for the probability to be valid is that you predict the thing that will be an outlier a priori, not just "locate" it among many things post hoc. You cannot then take that one event and quote its original 1:20 probability as the probability that the sample would have been obtained.

    If there are 100 things that might go wrong, and each has just a 1% chance of happening ( p = .01 ), then the probability that at least one of those 100 things will occur in any given sample is




     p_overall = 1 - ( 1 - p_individually )^100

      = 1 - ( 1 - .01)^100
      = 1 - ( .99 )^100
      = .63, or more than twice for every 3 samples



    The point here is that if many sorts of anomalies are candidates for being evidence of fraud, and each has a very low probability of occurrence, the joint probability that at least one of them will occur is not so low at all. It then becomes very important to ask how many things would have "looked suspicious" enough to cry foul? Unless those things were specified in advance, it isn't very convincing!

    Parent

    ...obviously .63 < 2/3, not greater n/t (none / 0) (#84)
    by JamesTX on Sat Jul 03, 2010 at 12:11:43 AM EST
    Isn't the potential downside for Kos (5.00 / 1) (#24)
    by observed on Fri Jul 02, 2010 at 09:57:39 AM EST
    enormous?
    If the suit fails, he'll be liable for damage to R2K's reputation, which is probably driving the company out of business now.

    And a loss would open a gaping (5.00 / 1) (#29)
    by BTAL on Fri Jul 02, 2010 at 10:11:36 AM EST
    credibility hole below the waterline at Kos.

    Parent
    Which is why (5.00 / 1) (#32)
    by me only on Fri Jul 02, 2010 at 10:28:44 AM EST
    most companies in Kos' position just sever ties and move on.  Airing the dirty laundry rarely is beneficial.  I mean does it really add to your credibility to admit that you were defrauded and would never have known it if your readership hadn't pointed it out?

    Parent
    Supposedly he is looking for different pollster (5.00 / 1) (#35)
    by magster on Fri Jul 02, 2010 at 10:38:37 AM EST
    Kos might have more trouble finding one since R2k is ruined.

    Parent
    You would think that, (5.00 / 2) (#38)
    by me only on Fri Jul 02, 2010 at 10:54:14 AM EST
    but I suspect in this economy someone will jump at the revenue.  I mean don't people keep thinking that Larry King would make a good husband...

    Parent
    Exactly. I would expect that Markos (5.00 / 3) (#37)
    by Cream City on Fri Jul 02, 2010 at 10:51:25 AM EST
    is a less credible teevee pundit now, if it is possible for teevee pundits to be even less credible.  But the damage to Markos' reputation, too, makes me wonder why he is pursuing this -- just pure pique?

    Of course, it just affirms me in my opinion of DKos.

    Parent

    Isn't Glen Beck a tevee pundit? (5.00 / 2) (#43)
    by me only on Fri Jul 02, 2010 at 11:02:54 AM EST
    I think Markos is taking this personally.

    Armando leaving DKos set in motion so many changes.  Just as DKos was starting to become popular his best blogger left.  However, the effect was masked by the growing popularity.  Today the FP's are lousy.  There is no debate.  It is all throwing gas on the fire.

    How people can be outraged all the time is beyond me.  Politics is too much theater to be taken personally on a daily basis.

    Parent

    If there was a profanity filter applied (5.00 / 3) (#48)
    by BTAL on Fri Jul 02, 2010 at 11:05:37 AM EST
    Today the FP's are lousy.  There is no debate.  It is all throwing gas on the fire.

    That site would see 3/4 of its content disappear.

    Parent

    moderation is the problem there. (none / 0) (#51)
    by observed on Fri Jul 02, 2010 at 11:07:33 AM EST
    That, and abuse of the ratings system.
    There are many excellent posters and commenters there, but it's too much trouble wading through shiit to find them.

    Parent
    Someone has fed him Baloney (none / 0) (#57)
    by Militarytracy on Fri Jul 02, 2010 at 11:33:56 AM EST
    that he paid good money for, and it infected his site.  He doesn't strike me as the sort of person who would just let that slide if after taking a look at his case he felt he could make the case.  If he were that risk adverse I doubt his site would have become what it is or even gotten off the ground.

    Parent
    simple explanation? (5.00 / 1) (#75)
    by ls on Fri Jul 02, 2010 at 03:35:10 PM EST
    If you sample an even number of people and have 2 answers the you'll always get even numbers for both answers or odd numbers for both answers. (If you got an even and an odd then the total would be odd, contradicting our stipulation that there be an even total.) If it's not the case that you ALWAYS have an even total, but just that you have a bias towards having an even total (say, because you have a list with 100 people on it, and while occasionally someone doesn't answer a particular question at all usually everyone answers all of them) then you will have a corresponding bias toward even+even or odd+odd. I haven't read the source materials for this story (and have only quickly read the comments here), but is there a known reason why this isn't the explanation.

     -Lee

    Isn't it the burden of the plaintiffs to (5.00 / 1) (#86)
    by observed on Sat Jul 03, 2010 at 08:27:44 AM EST
    show that R2K's methods were worse than the industry standard? That's certainly a point I would bring up if I were defending.
    There are questions about the practices of a LOT of polling firms. Is it even true that R2K is egregiously bad? And no, saying their polls have a bad track record doesn't prove anything.
    You have to show their methodology is flawed, or that they fabricated poll results entirely.
    The last seems to be what Bonin is saying, and seems far stronger than what the evidence supports at this point.

    Is there any precedent for this sort of (none / 0) (#3)
    by observed on Fri Jul 02, 2010 at 06:14:32 AM EST
    lawsuit? I don't know exactly why Kos is suing.
    According to the lawsuit, the other polls by R2K did not have the same flaws.
    Weren't the tracking polls a small minority of the total work done by R2K?


    Maybe not in the context of polling (5.00 / 1) (#4)
    by scribe on Fri Jul 02, 2010 at 07:15:52 AM EST
    but this sort of commercial dispute ("you promised to sell me a high-quality 'X' and the thing you sold me fell apart after a week because it was junk") get filed every day of the week in courts all across the country.  It's a garden-variety commercial dispute which popped up on our radar only because (a) it's Kos who's involved and (b) such a large part of the content he (as opposed to diarists) post on his site is the polling and horserace data.  

    The case is pleaded more elaborately than it needs be, with the exception of the fraud counts.  As to the fraud counts, they have to be pleaded, in the language of the relevant rules, "with specificity".  And they are.  

    The problem for Kos is precisely because his site is so saturated with horserace polling data, and was so out front both with the crosstabs and commissioned polls for more-indepth tea-leaf reading, he and the site have to move to a new provider and develop their own confidence and their audience's confidence in those results.

    Parent

    Specifically I mean precedents for (none / 0) (#5)
    by observed on Fri Jul 02, 2010 at 07:48:05 AM EST
    suits over polling or similar statistical services.


    Parent
    I wouldn't be surprised if there were, (none / 0) (#54)
    by scribe on Fri Jul 02, 2010 at 11:15:53 AM EST
    though I'm not personally aware of any.  

    At some point in history, any goods or services which can be sold, have also been the subject of a lawsuit over breach of contract and, likely, over fraud.

    Or, in other words, "there is nothing new under the sun."  [It might feel new, but that's just because it's new to you.]

    Parent

    The statistics are above my pay grade (none / 0) (#6)
    by andgarden on Fri Jul 02, 2010 at 08:07:44 AM EST
    but in all honesty, I don't see how Markos expects to see any money out of this. I assume whatever he sent to R2K is long since spent.

    I think there's more to it than money (5.00 / 2) (#16)
    by magster on Fri Jul 02, 2010 at 09:41:22 AM EST
    Reputation for one.

    Also, pollsters have a ton of influence in the political discussion, and exposing disreputable pollsters while enlightening media outlets on how pollsters can scam clients would be a benefit overall.  

    PPP had a funny post, "proving" they do their own polling by posting e-mail complaints from people who didn't like being called.

    Parent

    "More to it than money" (none / 0) (#23)
    by andgarden on Fri Jul 02, 2010 at 09:56:44 AM EST
    sounds good--until your legal bills start to arrive. I assume Adam isn't working for free

    Parent
    Contingency? (none / 0) (#44)
    by oculus on Fri Jul 02, 2010 at 11:03:57 AM EST
    Knowing what you know about R2K, (none / 0) (#47)
    by andgarden on Fri Jul 02, 2010 at 11:05:20 AM EST
    would you take that bet?

    Parent
    No. But I wouldn't have filed the complaint (none / 0) (#50)
    by oculus on Fri Jul 02, 2010 at 11:07:23 AM EST
    to begin with.

    Parent
    QED (none / 0) (#53)
    by andgarden on Fri Jul 02, 2010 at 11:08:48 AM EST
    Wouldn't that be unusual for (none / 0) (#49)
    by observed on Fri Jul 02, 2010 at 11:05:53 AM EST
    a lawsuit like this?
    It's not as if they can get much money if they win.

    Parent
    Maybe it is worth it if............ (none / 0) (#64)
    by BarnBabe on Fri Jul 02, 2010 at 12:31:53 PM EST
    If 'You' are the one who choose and relied upon data of a pollster who was rated in the 5 worse, maybe you would feel duped. At best, you might feel as if all your opinions based upon the R2k results made your opinions nullified and tarnish your reputation. AND, you are personally publishing these 'worse 5' results on your Blog and passing on the possible tainted information.

    Also, I read some 'Best blogs' results the other day, don't remember where, but DKos was in the Overrated category which surprised me. So maybe it really is just personal.  

    Parent

    Less on lawyers (none / 0) (#33)
    by waldenpond on Fri Jul 02, 2010 at 10:31:48 AM EST
    Ali accuses Markos of doing this because Kos owes him in the 6 figures.  So maybe Ali's theory is Kos' bottom line will benefit by spending less on lawyers.

    It's odd because in other statements there was request for payment from Kos to assist Ali with a cash flow issue...... but then Ali's response was a little rambling, accusing Kos of unmentionable 'sinister' motives.

    Parent

    Shrug (none / 0) (#36)
    by andgarden on Fri Jul 02, 2010 at 10:46:22 AM EST
    I'm content to watch the papers. If there's ever discovery, it ought to be interesting.

    Parent
    Most boring trial ever (none / 0) (#39)
    by waldenpond on Fri Jul 02, 2010 at 10:54:31 AM EST
    I like numbers but it would have to be the most boring trial to sit through.  

    Seems it would be hard to win.  The defense could ask for detail that would put everyone to sleep if not frustrate them.

    Battling statisticians.... ugh.

    Parent

    Actually, if you get a good explainer (5.00 / 2) (#55)
    by scribe on Fri Jul 02, 2010 at 11:23:02 AM EST
    who's also a good statistician, and you prepare carefully, it could be a good, interesting trial.

    A while back, I tried a case where the core of it involved reading and explaining to the jury the balance sheet of a bank.  (Cascading tables of numbers and only numbers) I had a good accountant who'd done this kind of work before and who could do math at the blackboard and explain what he was doing and why.  Despite the defendant's best efforts to cloud the issues, the jury understood it all.  They gave me every penny I asked for and they were only out about 3 hours.

    Cozen's a good, tough, litigating law firm;  they do mostly commercial and insurance litigation.  I've gone against them in the past and they are tough.  And they would not have taken it without a substantial likelihood of there being a recovery at the end of the lawsuit.

    Parent

    Where's the recovery coming from though? (none / 0) (#59)
    by andgarden on Fri Jul 02, 2010 at 11:57:37 AM EST
    You think Del Ali has a secret stash?

    Parent
    Out of his hide? (none / 0) (#63)
    by Militarytracy on Fri Jul 02, 2010 at 12:24:05 PM EST
    It isn't as sexy as firing a General, but this is pretty profound in some resepects isn't it?  Del Ali is supposed to be this GIANT of political analysis, and along comes a Rolling Stone :)

    I was floored with the whole can't email thing, can't attach, I'm at Kinkos and they can't help me either. WTF man?  Perhaps I should send my husband over to help him, I've had such problems and I give my husband half an hour and it is all better. Give me a break :)  Things are jacked up and his only resource is Kinkos?  I ain't buyin, the excuses are worse than pretty pathetic.

    Parent

    He might have insurance (none / 0) (#80)
    by scribe on Fri Jul 02, 2010 at 06:22:44 PM EST
    This might be covered under his business policy, or a CGL (depending on what coverage he purchased and which exclusions apply or don't apply).  You'd be surprised what might be covered.  

    I don't have access to a D&B report, but I'd bet R2K was doing in the $5 Mil/year range, so it's likely he has assets somewhere.

    And, as the lawer once said when asked why he kept pushing a case against an impecunious litigant:  "he might hit the lottery".  Judgments are good for 10 or 20 years or so (Depending on the state) and a lot can happen in that time.

    Parent

    Snort. Yeh, and for real fun (none / 0) (#40)
    by Cream City on Fri Jul 02, 2010 at 10:57:16 AM EST
    for anyone who would enjoy this trial, I've got some dissertations for really fascinating reads.

    Parent
    Honestly, I expect Del Ali to default (none / 0) (#41)
    by andgarden on Fri Jul 02, 2010 at 11:00:28 AM EST
    I highly doubt he has the cash to hire a lawyer and see this through to trial.

    Parent
    If I were a rich right winger, I'd (none / 0) (#46)
    by observed on Fri Jul 02, 2010 at 11:05:03 AM EST
    want to look into footing some of the bill.

    Parent
    Heh (5.00 / 1) (#52)
    by andgarden on Fri Jul 02, 2010 at 11:08:33 AM EST
    I would want to see the call center records first.

    Parent
    Ali should have (none / 0) (#85)
    by standingup on Sat Jul 03, 2010 at 01:39:19 AM EST
    a Professional Liability (E&O) Insurance policy that will come into play here. This is a more specialized type of insurance which will provide more coverage than just a general business liability coverage. Legal defense is typically included as part of the policy so the insurance company would bear the cost of defense.

    Parent
    Blog "hits"? (none / 0) (#34)
    by oculus on Fri Jul 02, 2010 at 10:35:06 AM EST
    Or is that too cynical?

    Parent
    Really not too cynical (none / 0) (#56)
    by lilburro on Fri Jul 02, 2010 at 11:29:12 AM EST
    as I work in the SEO biz...however that would be as far as I know the first time that someone would go to trial as linkbait though.

    Parent
    Research 2000 (none / 0) (#58)
    by Militarytracy on Fri Jul 02, 2010 at 11:46:47 AM EST
    Is probably finished in my opinion.  Settle now and hope everyone forgets soon.  I'm not big on polls, I lack the time and inclination to deconstruct them like you and andgarden do and get to some red meat, but I'm always glad to read and understand your hard work.

    The various pollsters only have their credibility that they conducted an actual poll to stand on.  Considering how often polling is "off", sometimes drastically off when we get to the actual hardcopy outcome we will all live with, how can anyone ever trust what comes out of Research 2000 ever again?  And anytime anyone ever attempts to use their results again when making a point those countering will always point to the time Research 2000 just wrote down some numbers and sent them off.

    Settle now Research 2000.  And you are probably still finished with Dem hiring you because DKos will always point out that your numbers caveat and when it comes election time every Dem wants DKos focusing on them.........not rip off polsters.

    Pay now, and hope to work for poor Republicans.

    Case Number? (none / 0) (#78)
    by kaleidescope on Fri Jul 02, 2010 at 04:38:45 PM EST
    Anybody know the three letters at the end of the case number?

    the initials of hte judge (none / 0) (#81)
    by scribe on Fri Jul 02, 2010 at 06:23:50 PM EST
    to which the case has been assigned.

    Parent
    Meh-a pox on both their houses. (none / 0) (#87)
    by Sweet Sue on Sat Jul 03, 2010 at 10:51:15 AM EST
    This suit is a fool's errand from a fool.

    is anyone else (none / 0) (#92)
    by desmoinesdem on Sun Jul 04, 2010 at 02:47:31 PM EST
    shocked that Markos didn't have a written contract with R2K?

    Mark Blumenthal's recap of what we know now is worth reading.

    I went back and looked at the cross-tabs for Research 2000's three Iowa polls this year (Feb, May and June for KCCI-TV in Des Moines). One of the anomalies the statisticians noticed was present: for every question on every poll, the answers for male and female respondents were either both odd numbers or both even numbers.