2D Maze
Continuity
Which points are connected to point A?
[E]
View full-size images
SceneHumans 98%Best model 99%
Oral at three NeurIPS 2026 workshops: NEmo, VLM4RWD, BabyVLM
MindTopo Can Foundation Models Reason in Topological Space?
Models can recognize topological relations in static scenes, but struggle to maintain or act on them reliably across a sequence of actions.
- topological properties
- 5
- cognitive levels
- 2
- procedural tasks
- 13
- Core instances
- 3,900
- MLLMs evaluated
- 13
- video generators probed
- 7
The Extended Set adds 8,780 instances with no exact-input overlap with the Core Set. Explore both sets and download the manifests.
02Key findings
Six things we learned
Planning lags behind reasoning 13/ 13 models score lower on planning than on reasoning. Far below humans 76.3% best model on Easy reasoning, vs 94.2% for humans. Open-weight models barely plan ≤0.6% average planning success, next to 0.53% for random legal moves. Training teaches patterns, not plans +37.3vs+6.1 points from SFT + RL on reasoning vs planning (Qwen3-VL-2B). Seeing vs acting 47.5% of reasoning failures are perception; 50.9% of planning failures come later, in state, dynamics and action. Imagination without validity 0/ 1,890 audited video rollouts pass all process-validity checks. Six video models reach a solved final frame in up to 35.2% of rollouts, while every audited rollout fails at least one check.
03Play the benchmark
Try the planning environments
These demos run MindTopo's own engines on input-hash-verified boards selected directly from the Core manifest. Core Easy and Core Hard use the same instances, rules, initial states, and action budgets as the primary evaluation.
Loading the gym…
Reasoning examples · reference scores on Core Easy
3D Maze
Continuity
Which points are connected to point E?
[C, F]
View full-size images
SceneHumans 95%Best model 87%
Assembly
Separation
Which pair of subassemblies exactly reconstructs the complete object? Choose A, B, C, D, or E.
Core Easy example
Humans 83%Best model 75%
Bead
Order
Start at the bead circled in white, follow the indicated direction along the string, and list all bead colours in order.
Core Easy example
GREEN, PURPLE, RED, GREEN, PURPLE, RED, GREEN, PURPLE
View full-size images
SceneHumans 93%Best model 52%
Origami Point
Order
Track the marked point from state 1 to state 14 of the fold.
[F]
Humans 96%Best model 83%
Sheep
Enclosure
How many sheep cannot escape to the outside?
3
View full-size images
SceneHumans 96%Best model 59%
Hole
Enclosure
How many holes are visible in the top board?
13
View full-size images
SceneHumans 100%Best model 67%
Knots
Knots
Which type is this scene?
View full-size images
SceneHumans 92%Best model 89%
04The action gap
Planning lags behind topology recognition
Results on the 3,900-instance Core Set: reasoning accuracy against planning success for every model. Across the evaluated models, identifying topological relations is more successful than acting on them. All Core combines 150 Easy and 150 Hard instances per task. Extended results are reported separately.
Difficulty
13 of 13 models plan worse than they reason
Averages over 8 reasoning and 5 planning tasks. GPT-6 Astra drops from 59.5% to 49.8%; no open-weight model plans above 0.6%.
GPT-6 Astra −9.7
GPT-5.6-Sol −4.5
Claude-Haiku-5.5 −14.5
Gemini-3.1-Pro −15.0
GPT-5.5 −10.6
Gemini-3.1-Flash-Lite −8.8
GPT-5.4-mini −6.6
Llama-4-Maverickopen −6.4
Qwen3.5-397Bopen −6.4
InternVL3.5-241Bopen −6.5
Ministral-3-14Bopen −5.6
Nemotron-Nano-12Bopen −5.4
Gemma-4-31Bopen −3.9
Random chance −7.4
Show data table
Full leaderboard
Select a column to sort.
| Average | Continuity | Separation | Order | Enclosure | Knots | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra | 59.5 | 49.8 | 70.7 | 81.3 | 54.7 | 71.0 | 70.7 | 53.3 | 69.3 | 51.3 | 35.3 | 33.3 | 51.7 | 62.0 | 20.7 |
| GPT-5.6-Sol | 24.3 | 19.8 | 33.0 | 14.0 | 11.0 | 36.0 | 9.3 | 18.7 | 9.7 | 40.3 | 31.3 | 26.3 | 35.3 | 25.3 | 3.0 |
| Claude-Haiku-5.5 | 19.9 | 5.4 | 1.0 | 13.3 | 0.0 | 30.0 | 0.0 | 11.0 | 4.3 | 0.0 | 42.3 | 25.3 | 22.0 | 31.7 | 5.0 |
| Gemini-3.1-Pro | 19.2 | 4.2 | 10.7 | 14.0 | 0.0 | 27.0 | 0.0 | 6.0 | 3.0 | 1.0 | 30.3 | 20.7 | 13.7 | 41.7 | 6.3 |
| GPT-5.5 | 18.6 | 8.0 | 8.7 | 7.7 | 0.0 | 33.0 | 0.0 | 7.3 | 10.3 | 19.7 | 23.7 | 27.3 | 12.7 | 30.7 | 7.7 |
| Gemini-3.1-Flash-Lite | 9.4 | 0.7 | 6.7 | 1.3 | 0.0 | 24.3 | 0.0 | 6.3 | 2.7 | 0.0 | 14.3 | 1.7 | 0.3 | 18.0 | 3.0 |
| GPT-5.4-mini | 7.3 | 0.7 | 7.7 | 2.7 | 0.0 | 22.7 | 0.0 | 5.3 | 2.3 | 0.0 | 5.0 | 0.3 | 0.0 | 12.3 | 3.3 |
| Llama-4-Maverickopen | 6.9 | 0.5 | 8.0 | 3.7 | 0.0 | 24.0 | 0.0 | 4.7 | 3.0 | 0.0 | 2.0 | 0.0 | 0.0 | 10.0 | 2.7 |
| Qwen3.5-397Bopen | 6.6 | 0.2 | 6.7 | 4.7 | 0.0 | 22.3 | 0.0 | 4.7 | 1.3 | 0.0 | 0.3 | 1.7 | 0.7 | 11.0 | 0.3 |
| InternVL3.5-241Bopen | 6.5 | 0.0 | 2.0 | 3.3 | 0.0 | 20.3 | 0.0 | 3.0 | 3.7 | 0.0 | 4.3 | 0.3 | 0.0 | 14.7 | 0.0 |
| Ministral-3-14Bopen | 6.2 | 0.6 | 5.0 | 2.0 | 0.0 | 24.0 | 0.0 | 7.0 | 1.7 | 0.0 | 1.3 | 0.0 | 0.0 | 8.7 | 3.0 |
| Nemotron-Nano-12Bopen | 5.6 | 0.1 | 4.0 | 1.3 | 0.0 | 21.7 | 0.0 | 2.7 | 0.3 | 0.0 | 3.3 | 0.3 | 0.0 | 11.0 | 0.7 |
| Gemma-4-31Bopen | 4.3 | 0.4 | 2.3 | 3.3 | 0.0 | 13.0 | 0.0 | 0.0 | 0.0 | 0.0 | 7.0 | 0.0 | 1.3 | 8.7 | 0.7 |
| Random chance | 7.9 | 0.5 | 4.3 | 3.7 | 0.0 | 20.0 | 0.0 | 7.1 | 2.2 | 0.0 | 5.5 | 6.7 | 0.0 | 14.0 | 2.7 |
05Core & Extended
Core and Extended are evaluated separately
Core is the balanced primary evaluation set. Extended adds exact-input-disjoint instances from the same generators. Scores from the two sets are reported separately and are never pooled.
Core Set
3,900
13 tasks, each with 150 Easy and 150 Hard instances. In total, 2,400 reasoning questions and 1,500 planning episodes.
Extended Set
8,780
Additional instances after removing 2,250 Core overlaps and 5 repeated Swap inputs from the original pool.
Paper and release files
Current manuscript (PDF) · Composition and deduplication record · Extended result counts (JSON)
The manifests identify each input, its source location, and its content hash. Use them with the benchmark source files to select the evaluation instances; the manifests do not contain the full inputs or rendered images.
arXiv v1 and the Hugging Face snapshot describe earlier dataset compositions. The current manuscript and manifests above define the Core / Extended split used on this page.
Per-task composition of both sets
| Task | Core | Extended |
|---|---|---|
| 2D Maze | 300 | 850 |
| 3D Maze | 300 | 846 |
| Pipe | 300 | 450 |
| Assembly | 300 | 890 |
| One Stroke | 300 | 450 |
| Bead | 300 | 850 |
| Origami Point | 300 | 1,000 |
| Swap | 300 | 445 |
| Sheep | 300 | 700 |
| Hole | 300 | 699 |
| Chat Noir | 300 | 450 |
| Knots | 300 | 850 |
| Untangle | 300 | 300 |
| Total | 3,900 | 8,780 |
Extended results for 11 models
Percent correct or successful. Reason and Plan are unweighted means over the eight reasoning and five planning tasks. Coverage follows the manuscript's Extended table, which excludes GPT-6 Astra and Claude-Haiku-5.5.
| Model | Reason | Plan | 2D Maze | 3D Maze | Pipe | Assembly | One Stroke | Bead | Origami Point | Swap | Sheep | Hole | Chat Noir | Knots | Untangle |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-5.6-Sol | 73.39 | 59.77 | 88.71 | 70.92 | 60.00 | 60.67 | 41.11 | 64.71 | 77.00 | 86.74 | 78.29 | 80.11 | 70.67 | 66.71 | 40.33 |
| GPT-5.5 | 60.93 | 30.00 | 67.29 | 51.18 | 2.89 | 57.42 | 13.11 | 39.88 | 60.00 | 50.11 | 67.29 | 79.54 | 39.56 | 64.82 | 44.33 |
| Gemini-3.1-Pro | 58.88 | 26.27 | 36.12 | 58.27 | 7.33 | 53.82 | 4.44 | 51.88 | 41.10 | 22.47 | 78.43 | 76.11 | 45.11 | 75.29 | 52.00 |
| Gemini-3.1-Flash-Lite | 45.48 | 11.36 | 33.76 | 36.05 | 0.00 | 45.73 | 0.00 | 39.76 | 42.20 | 11.69 | 63.14 | 31.62 | 8.44 | 71.53 | 36.67 |
| Qwen3.5-397B-A17B | 30.36 | 7.83 | 14.00 | 38.65 | 0.00 | 44.83 | 0.00 | 21.88 | 42.60 | 12.36 | 9.43 | 34.91 | 11.78 | 36.59 | 15.00 |
| GPT-5.4-mini | 26.72 | 13.34 | 17.53 | 22.34 | 0.22 | 36.18 | 0.00 | 32.47 | 27.20 | 13.71 | 25.71 | 23.75 | 8.44 | 28.59 | 44.33 |
| InternVL3.5-241B | 22.91 | 2.86 | 17.18 | 11.82 | 0.00 | 12.70 | 0.00 | 27.29 | 19.50 | 5.62 | 22.86 | 22.03 | 1.33 | 49.88 | 7.33 |
| Llama-4-Maverick-17B-128E | 22.82 | 4.51 | 25.65 | 17.97 | 0.00 | 35.73 | 0.00 | 30.94 | 17.10 | 7.42 | 15.57 | 3.72 | 2.44 | 35.88 | 12.67 |
| Ministral-3-14B-Instruct-2512 | 20.98 | 6.23 | 20.35 | 12.41 | 0.00 | 28.76 | 0.00 | 16.47 | 18.40 | 5.17 | 16.29 | 7.30 | 5.33 | 47.88 | 20.67 |
| Nemotron-Nano-12B-VL-v2 | 20.08 | 1.56 | 14.12 | 11.82 | 0.00 | 28.88 | 0.00 | 20.47 | 26.70 | 1.57 | 20.43 | 9.87 | 0.22 | 28.35 | 6.00 |
| Gemma-4-31B-IT | 11.83 | 7.67 | 30.35 | 18.68 | 0.00 | 19.10 | 0.00 | 9.41 | 1.60 | 1.35 | 9.29 | 0.00 | 17.33 | 6.24 | 19.67 |
Scroll sideways to see every task.
06Against humans
Even the strongest model trails humans on most tasks
Easy level, where five professional annotators solved the same instances under the same instructions (inter-annotator agreement 0.89).
Compare humans with
Reasoning avg
76.3 vs 94.2
-17.9 pts from humans
Planning avg
81.9 vs 99.9
-18.0 pts from humans
Largest gaps to humans
- Untangle −82.0
- Bead −41.0
- Sheep −37.0
- Hole −32.9
GPT-6 Astra reaches human level (within 1 point) on 2D Maze, Pipe, Swap.
Show data table
07Training
Training teaches visual patterns, not plans
Qwen3-VL-2B-Instruct trained on the Easy level of each task. Answer supervision lifts reasoning sharply; planning success barely moves, and gains on held-out tasks are uneven.
Single-task runs split Easy data into 80% training, 10% validation, and 10% test. Planning predicts a complete action sequence from the initial observation and executes it without further model feedback. Bead uses a simplified variant. These test scores use a different protocol from the Core leaderboard's closed-loop evaluation.
Training method
Reasoning avg (5 tasks)
14.2% 51.5%
+37.3 pts
Planning avg (4 tasks)
0.2% 6.3%
+6.1 pts
Show data table
08Failure modes
Reasoning fails at seeing. Planning fails at acting.
910 failures of Gemini-3.1-Pro and InternVL3.5-241B, sampled uniformly and labeled with the earliest stage that explains them, from following the instruction to planning the action.
Model
Show data table
Instruction followingSheep
“my final answer is that four sheep are trapped”
Output cut off; answers “unclear” (GT 4).
Task understandingAssembly
“…the top shelf floating. Invalid. The subassemblies must be connected.”
Adds an unstated connectivity rule; rejects split A, answers C.
Perception groundingHole
“a cylindrical object inside, … Total is 8.”
That bar makes two holes; answers 8 (GT 9).
Topological invarianceKnots
“The alternating under/over pattern … is a key indicator of linkage.”
After removing brown, red and white are both free; answers ["none"].
Feature state prediction2D Maze
“the green bar appears to be the critical one”
Orange and blue also work; answers [green].
Dynamic violationChat Noir
{"answer": 20} ×3
Three turns on the same cell; illegal, the cat escapes.
Action planningPipe
“I’ve been rotating (1,0) in a cycle.”
Spins (1,0) a full turn; budget spent.
09Visual imagination
Solved endpoints do not certify valid rollouts
Can image or video generation act as the world model that planning lacks? We let generators render the next state, or a whole plan, and then check sampled states and transitions.
Probing uses 105 episodes per task, 35 from each of three generator difficulty settings, including instances outside the Core levels. The video verifier uses a separate sample of 1,890 rollouts. It parses 41 evenly spaced frames, checks static constraints on four frames and dynamics between adjacent sampled frames, and requires every check to pass. Its human validation mainly confirms rejections; it is a strict screen for invalid rollouts.
35.2%
best rate of a solved final frame (LTX-2.5, Pipe)
0.0%
of 1,890 audited rollouts pass all process-validity checks on sampled frames and transitions
Manual audit of 119 Wan2.2-I2V-A14B rollouts
Illegal dynamics 116/119
Topology broken 93/119
Inconsistent frames 90/119
A generated next frame helps only where a local cue suffices
Planner
Knots +31.1
Sheep +38.2
2D Maze +39.7
Untangle +5.7
One Stroke −2.9
Pipe −9.5
Show data table
Manual audit of 464 GPT-Image-2 frames
Illegal dynamics 95/464
Topology broken 92/464
Inconsistent frames 68/464
A solved last frame is not a solved task
One Stroke
LTX-2.3 0.0
LTX-2.5 0.0
Veo-3.1-Lite 0.0
Wan2.2-I2V-A14B 0.0
MiniMax-H3 0.0
Seedance-2.5 0.0
Pipe
LTX-2.3 0.0
LTX-2.5 0.0
Veo-3.1-Lite 0.0
Wan2.2-I2V-A14B 0.0
MiniMax-H3 0.0
Seedance-2.5 0.0
Untangle
LTX-2.3 0.0
LTX-2.5 0.0
Veo-3.1-Lite 0.0
Wan2.2-I2V-A14B 0.0
MiniMax-H3 0.0
Seedance-2.5 0.0
10Citation
Cite MindTopo
@article{ge2026mindtopo,
title = {MindTopo: Can Foundation Models Reason in Topological Space?},
author = {Yunfei Ge and Anbang Liu and Qineng Wang and Johnalbert Garnica and Jianwen Lyu and Zihan Wang and Reuben Tan and Jianfeng Gao and Ruohan Zhang and Yining Hong and Jiajun Wu and Manling Li},
journal = {arXiv preprint arXiv:2609.11900},
year = {2026},
url = {https://mind-topo.github.io/},
}