LivetechWestern cycleAnchors pending
A publicly-released AI model achieves 85% or higher on SWE-bench Verified at some point during 2027?
As of 2 Oct · 10:00 UTCResolves 31 Jan 20283 movers2 pressure moments
The read
The crowd underrates Benchmark Cleared.
Reality leans toward Benchmark Cleared — the forces holding it there outweigh the pull in the other direction. The crowd lands lower on Benchmark Cleared than the evidence supports. The main thing that could shift this is The pace of post-training on code-execution feedback.
Where reality is heading
A, 80% bandCrowd today
Reality crosses toward A in early 2028. Today’s crowd number sits at the top of A’s band, possible but optimistic. View as table
The movers, ranked by leverage
What would change our mind
Signals toward A
- The pace of post-training on code-execution feedback. Running an AI model on real code, letting it see which tests pass and which fail, and training on those ou shifting in the direction that supports Benchmark Cleared
- Compute availability for post-training runs. Code-execution training is compute-heavy crossing the threshold toward Benchmark Cleared
- Whether SWE-bench Verified gets revised. The benchmark has already been curated once (the Verified subset itself was a 2024 revision stripping out bad tasks). I readings confirming the Benchmark Cleared direction
- Open-weights lab contribution. If DeepSeek, Qwen, Meta, Mistral, or a new entrant releases openly-downloadable models with competitive code reasoning, the publi trajectory bending toward Benchmark Cleared
- Lab willingness to submit publicly. Some labs withhold scores from certain benchmarks for strategic or safety reasons. Reaching 85% on the public leaderboard re moving in support of Benchmark Cleared
Signals toward B
- The pace of post-training on code-execution feedback. Running an AI model on real code, letting it see which tests pass and which fail, and training on those ou shifting in the direction that supports Still Short
- Compute availability for post-training runs. Code-execution training is compute-heavy holding against the shift toward Still Short
- Whether SWE-bench Verified gets revised. The benchmark has already been curated once (the Verified subset itself was a 2024 revision stripping out bad tasks). I readings confirming the Still Short direction
- Open-weights lab contribution. If DeepSeek, Qwen, Meta, Mistral, or a new entrant releases openly-downloadable models with competitive code reasoning, the publi trajectory bending toward Still Short
- Lab willingness to submit publicly. Some labs withhold scores from certain benchmarks for strategic or safety reasons. Reaching 85% on the public leaderboard re moving in support of Still Short
Pressure moments
Evidence
Princeton NLP / SWE-bench · SWE-bench Verified public leaderboard (swebench.com) — the specific scoreboard the prediction resolves against. Snapshot archives through 2025 show late-2025 state-of-the-art at approximately 70%, up from about 17% in early 2024.
METR · METR (Model Evaluation and Threat Research) evaluations of frontier AI agent capability (metr.org) — third-party external evaluations that track the gap between benchmark scores and real-world deployed task completion, providing an independent check on whether leaderboard numbers transfer to practical use.
Papers With Code · Papers With Code SWE-bench leaderboard and the arXiv model-release pipeline (paperswithcode.com, arxiv.org) — the research paper trail feeding frontier capability improvements. The post-training recipes that drove the 2024-2025 jumps are all documented in public papers, which makes trajectory estimation possible.
- Metaculus AI-benchmarks category (metaculus.com) — publicly-tradeable question-contracts aggregating analyst views on AI capability thresholds. Serves as the narrative-weighted baseline the prediction compares against where no exact-threshold market exists.No source URL yet