Pollster Rating Wars!

With both sides surviving the 2020 Model Wars, the election forecasting blood feud continues through another election.

In April 2023, news leaked out that Nate Silver and several FiveThirtyEight staffers would be victims of Disney’s mass layoffs.  Silver’s last day working for “The Mouse” was June 30^th and per his contract, ABC/Disney kept rights to the domain and the name “FiveThirtyEight” while Silver retained rights to his models.  He took his IP to Substack, ordinarily a graveyard for journalists, and started the “Silver Bulletin.” 

Seemingly to spite Silver, The Mouse hired his arch-nemesis Elliot Morris to take over the rebranded “538.”

The first shots of the “Pollster Ratings Wars” were fired at the end of June 2023, after Morris started at ABC but a few days before Silver officially walked out the door.

Morris emailed Rasmussen Reports the following questions/demands on June 29^th:

Nate Silver immediately clapped back on Twitter:

Very bad to do a Spanish Inquisition with pollsters based on their political orientation. I love my ex-colleagues (this is coming from a new guy they hired) but if this is their practice, hope ABC will stop use of 538 brand so it isn't associated with me.https://t.co/PCRrFhRKMf pic.twitter.com/1v0ygL590N

— Nate Silver (@NateSilver538) June 29, 2023

And Silver went into a lengthy critique of the political litmus test on his first day of “free agency” on July 1^st:

First, I strongly oppose subjecting pollsters to an ideological or political litmus test. Look, there might be good reasons to exclude Rasmussen based on their methodology, although I’d note that their track record of polling accuracy is average, not poor.

…

Second, even if you’re going to remove Rasmussen from the averages going forward, it’s inappropriate to write them out of the past, as Morris has threatened to do.

…

Third, I think it’s clear that the letter is an ad hoc exercise to exclude Rasmussen, not an effort to develop a consistent set of standards.

…

The thing about running a polling average is that you need a consistent and legible set of rules that be applied to hundreds of pollsters you’ll encounter over the course of an election campaign. Going on a case-by-case basis is a) extremely time-consuming (don’t neglect how busy you’ll be in the middle of an election campaign) and b) highly likely to result in introducing your own biases, whether it’s the political outcome you’re rooting for or whatever you think will make your model look smart. That’s why, after 15 years of doing this, I’ve been a stickler for consistency, even if that means including some pollsters whom I subjectively don’t like, politically or methodologically.

Perhaps Morris’s questions were getting at some larger theme or more acute problem. But if so, he have should stated it more explicitly in his letter. Journalists, in most circumstances, shouldn’t act like Vincent D’Onofrio in Law & Order trying to sniff around for clues or throw a suspect off-kilter. Ask clear, concise questions that make your intentions clear.

Instead, this looks like a fishing expedition, with Morris hoping to catch Rasmussen in some sort of venial methodological sin that is probably fairly common within the industry. Or, because the questions are onerous, the tone of his email is hostile, and Carroll was only given a day-and-a-half to respond just before a four-day summer weekend, he was hoping that thy wouldn’t be answered at all — so he could say “See! They refused to answer my questions!”. Either way, this is the letter you get only once someone has already made their mind up.

[Emphasis in original]

Whatever else may have transpired or not between Morris and Rasmussen Reports, the polling firm is no longer included in polling averages today.  However, it’s still in the pollster ratings and ranked #69 out of 277 with 2.1 stars in the combined (accuracy plus transparency) ratings.  Morris has said that between 1.9 and 2.8 stars puts a pollster in “America’s core block of good pollsters.”  This is even more confusing since much lower ranked pollsters are included in the averages.  (And Silver Bulletin’s grade for Rasmussen is a “B”).

Below were going to look at the 2024 pollster ratings for 538 (Morris) and Silver Bulletin (Silver.)  We’re not trying to find the “best” system, even if that is possible, but are merely interested in differences in how each platform is rating pollsters.  Because Silver merely took the pre-Morris FiveThirtyEight pollster rating methodology and updated the data, we’re basically comparing FiveThirtyEight in 2023 (Silver) versus 538 in 2024 (Morris). 

Silver Bulletin (old FiveThirtyEight) pollster ratings

For the 2024 election cycle, Silver went back to FiveThirtyEight’s 2023 methodology:

The 2023 FiveThirtyEight methodology is below.  (Please note: The original passage has been rearranged for brevity and clarity while preserving the original words and intent.)

Again, this was edited for clarity and brevity from FiveThirtyEight’s 2023 Pollster Ratings article.  Please read the original article for more details and to find older rating methodologies.

538 (new) pollster ratings

Elliot Morris put his own mark on 538 by adding transparency scores for pollsters, and adding a 3-star rating system on top off the combined (accuracy + transparency) scores.  [Block quotes below are edited for clarity and brevity.]

POLLSCORE

The accuracy portion is now called “POLLSCORE” (Predictive Optimization of Latent skill Level in Surveys, Considering Overall Record, Empirically) includes error and bias, but not transparency.  Lower is better, indicating less error and bias.

Here is Morris describing POLLSCORE:

We quantify error by calculating how close a pollster’s surveys land to actual election results, adjusting for how difficult each contest is to poll. Bias is just error that accounts for whether a pollster systematically overestimates Republicans or Democrats. We average our final error and bias values together into one measure of overall accuracy…POLLSCORE tells us whether a pollster is more accurate than a theoretical replacement-level pollster that polled all the same contests. Negative POLLSCOREs are better and mean that a pollster has less error and bias than this theoretical alternative.

For example, the ABC News/The Washington Post poll had a POLLSCORE of -1.1 and Rasmussen Reports -0.5.

The first step is to calculate a Raw Error and Raw Bias metric for each poll. Raw Bias is the difference between the margin between the top two candidates in the poll and the margin between those candidates in the actual election result. Directionality matters here: A positive Raw Bias means the poll overestimated support for Democrats, and a negative Raw Bias means the poll overestimated Republicans. Raw Error is the absolute value of Raw Bias (in other words, it’s the same thing, except directionality doesn’t matter). For example, if a poll showed the Democratic candidate leading by 2 percentage points but she actually won by 5 points, that poll’s Raw Bias is -3 points (overestimating Republicans), and its Raw Error is 3 points.

We calculate these values only for polls that identified the correct first- and second-place candidates. Further, we calculate Raw Bias only for polls with both Democratic and Republican candidates (we cannot detect the partisan bias of a survey if there are only two Democrats in the race).

Of course, we should expect that some surveys will have higher errors and biases than others. A poll with a sample size of 500 people, for example, has a larger margin of error than a poll of 5,000 people and should usually be less accurate. So the next thing we have to do is to calculate Excess Error and Bias.

…

But for Excess Error, we run a multilevel regression model on every nonpartisan poll in our dataset to calculate how much error we would expect it to have based on the implied standard deviation from its sample size, the square root of the number of days between the median date of the poll and the election, plus variables for the cycle the poll was conducted and the type of election it sampled (e.g., presidential primary, presidential general election, gubernatorial general election, Senate general election, House general election or House generic ballot). Each poll’s Excess Error, then, is simply its Raw Error minus that expected error. Our regression weights polls by the square roots of their sample size (which we capped at 5,000 to avoid giving any one poll too much weight), the number of polls in each race and the number of pollsters surveying each race.

Finally, we calculate an Excess Error and Excess Bias score for each pollster by taking a weighted average of the Excess Error and Bias of each of its polls, with older polls given less weight. The precise amount of decay changes every time we add more polls to the database, but it’s currently about 14 percent a year. For example, the Transparency Score of a 1-year-old poll would be weighted 86 percent as much as that of a fresh poll; the Transparency Score of a 2-year-old poll would be weighted 74 percent (86 percent times 86 percent) as much; etc.

…

So the next step in our process is to adjust each poll’s Excess Error and Bias based on how difficult of a race it polled. First, for every poll in our database, we calculate the weighted average Excess Error and Bias for all other polls in the race, with polls with larger sample sizes given higher weight. Then, we subtract those values from the Excess Error and Bias of the poll. That gives us the Relative Excess Error and Bias of every poll.

Finally, we calculate statistics called Adjusted Error and Adjusted Bias, which are a weighted combination of each poll’s Excess Error/Bias and Relative Excess Error/Bias, where the Relative Excess statistics get more weight when more pollsters release more surveys in a given race. We make this adjustment to reflect the fact that we are more confident in a pollster’s relative performance in a race when the benchmark we’re comparing them against (all the other pollsters in a race) is based on a larger sample of data.

ABC News/The Washington Post had a Predictive Error of -1.0 and a Predictive Bias of -1.2 (average -1.1).  Rasmussen Reports received -0.4 and -0.6 (average -0.5), respectively.  Note that the averages makeup each POLLSCORE.

Transparency

So we now also score firms based on their methodological transparency. To do this, we have quantified how much information each pollster released about every poll in our archive since 2016…Each poll gets 1 point for each of 10 criteria it meets, ranging from whether it published the actual question wording of its poll or listed sample sizes for key subgroups. We give each pollster a Transparency Score based on the weighted average of the scores of its individual polls and whether it shares data with the Roper Center for Public Opinion Research at Cornell University or is a member of the American Association for Public Opinion Research’s Transparency Initiative.

1. Did the pollster publish the exact trial-heat question wording used in this poll?
2. Did the pollster publish the exact question wording and response options for every question mentioned in the poll release?
3. Did the pollster release both weighted and unweighted sample sizes for any demographic groups, or acknowledge the existence of a design effect in their data?
4. Did the pollster publish crosstabs for every subgroup mentioned in the poll release?
5. Did the pollster disclose the sponsor of the poll (if there was a sponsor)?
6. Did the poll specify how the sample was selected (e.g., via a probability-based or non-probability method)? If the sample was probability, was the sampling frame disclosed? If non-probability, did the pollster disclose what marketplace or online panels were used to recruit responses or its model for respondent selection?
7. Did the pollster list at least three of the variables the poll is weighted on?
8. Did the pollster disclose the source of its weighting targets (e.g., “the 2022 American Community Survey”)?
9. Did the poll report a margin of error or sample size for a “critical mass” of subgroups? We do not mandate this be a complete count, but if it looks like groups are intentionally missing (e.g., they are referenced in the press release but are missing in the crosstab documents), we withhold the point.
10. Did the poll methodology or release include a general statement acknowledging a source of non-sampling error, such as question wording bias, coverage error, etc., in addition to the normal margin of sampling error inherent to surveying?

We award each poll a 0, 0.5 or 1 on each question; a perfect score is 10, while the worst is 0…We then calculate a Transparency Score for each pollster by taking a weighted average of the Transparency Scores of all its polls, with the Transparency Scores of older polls getting less weight.

The final Transparency Score for a given pollster is a weighted average of all of the above, with 70 percent of the weight going to its directly measured transparency and 30 percent of the weight on its implied transparency — a value of 10 if it is a member of the AAPOR/Roper group, and 0 if it is not, with one exception: If a pollster’s directly measured transparency is 8 or higher and it has released more than 15 polls, we treat it as an honorary member of the AAPOR/Roper group.

We developed this metric in collaboration with Mark Blumenthal, a pollster, past 538 contributor and co-founder of the (now sadly defunct) poll-aggregation website Pollster.com. Blumenthal has found that pollsters that released more information about their work tended to be more accurate during the 2022 election cycle.

ABC News/The Washington Post received a transparency score of 9.2 and Rasmussen received 3.3.

Combined 3-star score

This ranking procedure yields a single, combined score that summarizes each pollster’s performance along both dimensions. For interpretability, we convert this value to a star rating, between 0.5 and 3.

Only the best of the best will get 3.0 stars; these are pollsters who score in the 99th percentile or better for both accuracy and transparency. Pollsters scoring between 2.8 and 3.0 are still very good — just not the best of the best. Most pollsters score between a 1.9 and 2.8, representing what we see as America’s core block of good pollsters. Pollsters between 1.5 and 1.9 stars are decent, but they typically score poorly on either accuracy or transparency. Generally, we are very skeptical of pollsters that get less than 1 star, as they both have poor empirical records and share comparatively little about their methodology. A 0.5-star rating — the bare minimum — is reserved for pollsters that have records of severe error or bias or are disclosing only the bare minimum about their polls.

We use something called Pareto Optimality [or “Pareto Efficiency”]. In layman’s terms, the “best” pollster in America would theoretically be one with the lowest POLLSCORE value and a Transparency Score of 10 out of 10. But in practice, the pollster with the lowest POLLSCORE and the pollster with the best Transparency Score are different pollsters. So our algorithm determines the “best” pollster to be the one that is closest to ideal on both metrics, even if it is not the best on any one metric.

What we really want from our ratings are not single point predictions of each pollster’s POLLSCORE, Transparency Score and final rank, but distributions of them. We need a way to calculate how much each pollster’s scores change if you ignore certain good (or bad) polls it has released. The method we turn to for this is called “bootstrapping.” To bootstrap our model essentially means re-running all the steps we’ve described so far 1,000 times. Each time, we grade pollsters based on a random sample of their polls in our database. As is standard in bootstrapping, we sample the polls with replacement, meaning individual polls can be included multiple times in the same simulation. We do this to keep the number of polls we have for each pollster constant across simulations. In the end this procedure yields 1,000 different plausible pollster scores for each organization.

Finally, we calculate the median of these simulated POLLSCORE and Transparency Scores. Pollsters are re-ranked according to the algorithm described [above]. Compared to our point predictions, the bootstrapped results change rather little for most pollsters, but they punish those who score well because they got lucky once or twice and reward pollsters that more reliably have lower bias and error than other firms. As a final hedge against modeling error here, we average the bootstrapped results and point predictions together.  These numbers are the final ratings you see on our dashboard.

ABC News/The Washington Post received a final rating of 3 stars (ranked #1) and Rasmussen received 2.1 stars (ranked #69).

Please check out What are the best pollsters in America? and How 538’s pollster ratings work for 538’s entire description of their 2024 ratings and methodology.

Comparison

Let’s look at the top 20 pollsters for 538 (combined star ratings) and Silver Bulletin.  The top pollsters mostly overlapped, with 13/20 of the top 20 pollsters shared between Silver and 538, albeit in different orders.  There are 12 colleges in 538’s top 20, as opposed to 8 in the Silver Bulletin’s.

538 Top 20	Rank	Silver Bulletin Top 20
The New York Times/Siena College	1	Selzer
ABC News/The Washington Post	2	The New York Times/Siena College
Marquette University Law School	3	ABC News/The Washington Post
YouGov	4	Research & Polling Inc.
Monmouth University Polling Institute	5	SurveyUSA
Marist College	6	Siena College
Suffolk University	7	Marquette University Law School
Data Orbital	8	AtlasIntel
Emerson College	9	Beacon Research/Shaw & Co. Research
University of Massachusetts Lowell Center for Public Opinion	10	Marist College
Muhlenberg College Institute of Public Opinion	11	Cygnal
Selzer & Co.	12	Monmouth University
University of North Florida Public Opinion Research Lab	13	Landmark Communications
SurveyUSA	14	Emerson College
Beacon Research/Shaw & Co. Research	15	MassINC Polling Group
Christopher Newport University Wason Center for Civic Leadership	16	University of Massachusetts Lowell
Ipsos	17	University of North Florida
MassINC Polling Group	18	TIPP Insights
Quinnipiac University	19	Public Policy Institute of California
Siena College	20	CBS News/The New York Times

Spearman’s rank correlation coefficient is similar to Pearson correlation but applied to ranked variables.  The coefficient goes from -1 to 1, with 1 being perfectly similar and -1 being perfectly dissimilar.  The coefficient here is 0.573 (p=2.7×10⁻²¹), indicating moderate correlation between Silver Bulletin and 538.  It should be noted that 538 now only ranks 277 pollsters while Silver Bulletin ranks 516; we only looked at the 228 that overlapped based on overlapping ‘Pollster Rating ID’.  The missing data shouldn’t be an issue because the coefficient is based on the ranking rather than absolute position.

The pollsters with the greatest disparity in rankings are:

Pollster_538	Rank_538	Rank_Silver
Roanoke College Institute for Policy and Opinion Research	31	478
Mitchell Research & Communications	77	497
SurveyMonkey	94	464
Targoz Market Research	89	451
Rutgers University Eagleton Center for Public Interest Polling	58	418
Baldwin Wallace University Community Research Institute	134	482
American Research Group	59	406
SoonerPoll.com	118	456
Marketing Resource Group (MRG)	124	460
Meredith College Department of History, Politics, and International Studies	146	481

This is a little misleading because of the difference between the two in the number of pollsters ranked, so let’s look at quartiles.

Pollsters in the Upper Quartile for 538 but in the Bottom Quartile for Silver Bulletin:

Pollster_538	Ranking_538	Rank_Silver
Roanoke College Institute for Policy and Opinion Research	31	478
Rutgers University Eagleton Center for Public Interest Polling	58	418
Mitchell Research & Communications	77	497

Pollsters in the Upper Quartile for Silver Bulletin but in the Bottom Quartile for 538:

Pollster_538	Ranking_538	Rank_Silver
North Star Opinion Research	228	95
GBAO	230	117
co/efficient	237	56
Garin-Hart-Yang Research Group	244	57
Harstad Strategic Research	247	109
Trafalgar Group	273	67

Excluding 538 transparency scores

Rather than using 538’s combined score, if we rank 538’s pollsters by POLLSCORE, with no transparency score, then the two rankings move even further together.  The Spearman Rank Correlation coefficient is 0.641 (p=5.13×10⁻⁵⁴).  Because of the large jump in the number of pollsters that 538 gives a POLLSCORE, from 277 to 539, we now have 455 overlapping pollsters between 538 and Silver Bulletin.

With only one decimal place for the POLLSCORE rating, it’s difficult to specify a rank, but ABC News/The Washington Post poll would now be ranked by 538 between #5-10, while Rasmussen Reports would rise to somewhere between #36-46.  

Conclusion

The 538 and Silver Bulletin ratings for 2024 are generally similar, while Silver is staying the course with his methodology and Morris is definitely changing the direction of 538 now that he’s in charge.  We won’t know until after the election, at the earliest, who was right (or more right).

And both are wrong to call this a “pollster” rating since these are actually “polling firm” ratings.  As the business of politics continues to grow and consolidate, polls from large polling firms cease to be a unified work product.  Much more than Morris’ transparency questions, tracking the individual(s) responsible for a poll would increase the value of the rating system, and the integrity of the polling industry in general.  As long as they can land clients, a bad pollster will survive, if not prosper, in politics.

Additional Information

Silver Bulletin

Silver Bulletin model methodology
Silver Bulletin pollster ratings
FiveThirtyEight pollster rating methodology (2023)
Historic FiveThirtyEight pollster rating data
*Silver Bulletin pollster rating data is behind the paywall.

538

What are the best pollsters in America?
How 538’s pollster ratings work
Pollster Dashboard
538 pollster rating data (2024)

Silver v. Morris 2020

Who Won the 2020 model wars?