Forecasting the 2024 Hurricane Season: Manifold v. Experts

Two play-money prediction markets were set up to forecast the number of hurricanes in 2024. One market was closed to experts, and the other was open to the public.

The first, CRUCIAL Atlantic Hurricane Markets (CAHM), is run by Climate Risk and Uncertainty Collective Intelligence Aggregation Laboratory (CRUCIAL). The market ran from December 10, 2023, to November 30, 2024, and was closed to invited experts:

Of 19 candidate participants selected for their expertise, 15 completed the informed consent form approved by Lancaster University’s Research Ethics Committee. 60% of the participants were based in academia, while the remaining 40% were in the private sector or non-profit organisations. Of the 15 participants endowed with onplatform credits, 10 subsequently engaged in trading on the platform during the hurricane season.
“Tracking evolving views of the 2024 Atlantic hurricane season with an expert prediction market”

The expert traders were given 500 credits to trade and achieved 40,749 in trading volume overall. Participants were then paid £1 for every credit they had accumulated above their original 500 credit allocation. The market itself was operated by the AGORA platform, which was created by Hivemind, a spinout from the Winton Group. CRUCIAL itself is an academic project funded by SCOR, a French reinsurance company.

The second hurricane market, “How many Hurricanes in 2024?” created by this author, was hosted by Manifold Markets and open to the public. 44 traders participated with 28,918 in trading volume using their fictitious virtual “Mana” currency. It opened on April 18, 2024, and closed on December 31, 2024. To give a sense of the users on Manifold, here is Nate Silver’s description of “Manifest”, Manifold’s conference/festival:

I’ve been around a lot of weird people. These people are weird and proudly so, even relative to the other weird people.
Conversations with Tyler podcast

It should be noted that both markets utilized an Automated Market Maker (AMM) with Robin Hanson’s Logarithmic Market Scoring Rule (LMSR). This facilitates trading by providing an automatic counterparty.

Methodology

Comparing the two markets is not apples-to-apples, so adjustments were made:

The markets were open for different lengths of time, so we’re limiting analysis from May 1, 2024, to November 30, 2024 (214 days).
The markets offered different bands or contracts. Manifold offered 14 outcomes from 0 hurricanes to ’13 or more’. CAHM offered 21 outcomes from 0 hurricanes to ’20+’, so options from ‘13’ through ’20+’ were combined into a ’13 or more’ option with the sum of their total probabilities.

Incentives were vastly different between markets, with CAHM traders receiving cash for every credit achieved above 500, while Manifold users traded with valueless Mana (although it could be donated to charity).

To measure the accuracy of the market forecasts, we’re using the average Brier score from May 1 to November 30, 2024.

The binary market outcomes (“resolution”) are quantified as either 1 (YES for the outcome of 11 hurricanes) or 0 (NO for every other outcome). The Brier score is essentially the mean squared error for each day of outcomes. The market error is the difference between the actual outcome and the market’s probability for that day. That error is then squared, which has the consequence of eliminating negative values and penalizing extreme predictions. The Brier score is the simple mean of the squared errors for all the forecasts. For example, given a probability of 0.8 and a resolution NO:

(Probability – Resolution)² = (0.8-0)² = (0.8)² = 0.64

Brier scores range from 0 to 1, and lower is better. A score of 0 means perfect accuracy.

Here is the simple mean of the daily Brier scores from May 1 to November 30, 2024:

Market	mean Brier score
CAHM	0.0556
Manifold	0.0598

Manifold traders performed well, but not quite as accurate as the expert market.

In the chart below, we can see that the first few weeks of June are where CAHM has an edge on accuracy over Manifold; the market only launched on April 18. If we eliminated May from our analysis, Manifold would have a lower Brier score than CAHM from June 1 to November 30 (0.0525 to 0.0549). It’s also worth noting that of the 214 days examined, CAHM (106 days) and Manifold (108 days) had roughly the same number of days with the lower score.

We shouldn’t read too much into one year of data (N=1) but it will be interesting to see if Manifold traders maintain expert-level accuracy in the future. CRUCIAL is hosting another CAHM prediction market for 2025 hurricanes, as is Manifold. Real-money markets are finally joining in, with Kalshi offering a market on 2025 hurricanes, although Polymarket still only offers markets on “named storms during Atlantic Hurricane Season.”

In case CRUCIAL would like to invite them to participate, here are the top five traders on Manifold’s 2024 hurricane market:

#	Name	Total Profit
1	For Truth	545
2	JAAM	281
3	John Ota	262
4	Christopher Randles	216
5	Daniel evans	191

Hurricane markets & data

CRUCIAL market data at Harvard Dataverse
Manifold: “How many Hurricanes in 2024?”
Manifold: “How many Hurricanes in 2024?” market data
Manifold: “How many Hurricanes in 2025?”
Kalshi: 2025 Hurricane market
Polymarket: “How many named storms during Atlantic Hurricane Season?”

Methodology

Further Reading

Hurricane markets & data