Three AI Forecasters Walk Into a Prediction Market, and Copy the Odds

OpenAI has opened its Decisions API in beta, and within hours a developer had put it to work forecasting real-world events. Instead of relying on benchmarks, which models may have already seen during training, the developer scored three AI systems against prediction markets, where prices reflect real money on the line. The three contenders were OpenAI's Decisions model, TypeSafe's Jev and Cloudflare's Clef, which has open weights and can run locally through Ollama.

The setup is straightforward. For any yes-or-no question about the future, the pipeline pulls the last 60 days of news, removes anything that quotes odds or betting sites, and passes the eight most relevant articles to all three models. Each returns a probability. A replay feature then re-asks every model after each article, so you can see which headline moved which system. The developer describes the evidence search as running through Valyu's news API, with live prices taken from the public Polymarket and Kalshi feeds.

The most striking result had nothing to do with forecasting skill. When the market price was left in the prompt, the models simply copied it. On a Federal Reserve rate-cut question, one model gave 63 percent when no price was shown, 30 percent when told the market said 30, 71 percent when told 71, and 88 percent when told 95. The developer's conclusion is that "AI versus the market" comparisons mean nothing unless the odds are stripped out before the models read anything.

With prices hidden, the models behaved quite differently. Decisions was the most decisive and the most willing to disagree with the market. Jev barely moved on any article, which reads as caution. Clef reacted the hardest. It was pushed from 8 percent to 63 percent by an SEO spam page, then corrected itself after the next genuine news story. On an October Fed question where the market sat at 83 percent, the three models finished at 17, 30 and 19 percent.

The developer is clear about the limits. The sample is tiny, and none of the questions have resolved yet, with the Fed market due to settle on October 28. The project is presented as an experiment, not a trading edge, and it is not financial advice. The code is open source under the MIT license. The open question the developer is asking the community to help with is how to score calibration fairly across the three models once the markets resolve.