Source: https://spec-kitty.ai/blog/benchmarking-open-weight-models-swe-bench

# Benchmarking 12 open-weight models on Mirantis TokenFactory and Exe.dev

> We ran 12 open-weight models through all 2,294 SWE-bench tasks, 27,528 attempts in all, and published every score, method and attempt. Here are the top-line results, what our own limits did to them, and why this is the baseline for Brain Kitty and Spec Kitty.

By [The Spec Kitty team](https://spec-kitty.ai/about.md) · Published 2026-11-02 · Benchmarks

This is a dated blog post. It may use retired names (Charter, Carta, doctrine) or describe features that are not in the product today; on any conflict, /llms.txt and the product pages win. Current product model: https://spec-kitty.ai/llms.txt

We ran 12 open-weight models through the full SWE-bench benchmark, one attempt on each of its 2,294 real-world bug-fix tasks. That is a total of 27,528 attempts, every one graded with the official SWE-bench harness. We're publishing the scores, the method and every attempt we recorded, so you can check our work or use it for your own.

[Read the full results →](https://benchmark.spec-kitty.ai/)

## The top-line results

On the full 2,294-task benchmark, `deepseek-v4.1-flash-1017` came first. It solved 1,261 tasks, or 54.97%. `glm-5.3` solved 52.01%, and the other four top models landed between 48.7% and 52.6%. At the bottom, `nemotron-3.5-lightning` solved 17.00% and `minimax-m3.1-flash` solved 16.39%.

![Bar chart of the share of the 2,294 SWE-bench tasks each of 12 models solved, with 95% confidence intervals. deepseek-v4.1-flash-1017 leads at 54.97%; minimax-m3.1-flash is last at 16.39%.](https://spec-kitty.ai/cms/blog/swe-bench-baseline-leaderboard.webp "Share of the 2,294 tasks each model solved, with 95% confidence intervals.")

Each model got one attempt per task. The short lines are 95% confidence intervals. Neighbouring models in the top six overlap, so we wouldn't call a winner among them from this run alone. [Explore every model, including the Verified and Lite subsets.](https://benchmark.spec-kitty.ai/)

On the 500-task Verified subset, `deepseek-v4.1-flash-1017` solved 79.4% and `glm-5.3` solved 78.8%. On the 300-task Lite subset, those two tied at 66.3%. The Verified and Lite numbers are on the [leaderboard page](https://benchmark.spec-kitty.ai/) too.

The models fail in different ways giving us different insights than the rank alone. Most attempts that missed did so because the patch failed the tests. For some models, a large share ended with no patch at all.

![Stacked bars showing how each model's 2,294 attempts ended: solved, patch failed the tests, no patch submitted, or ran out of time. 5,594 attempts ended with no patch and 120 ran out of time.](https://spec-kitty.ai/cms/blog/swe-bench-baseline-outcomes.webp "How each model's 2,294 attempts ended.")

Where every attempt ended. [The methodology page](https://benchmark.spec-kitty.ai/methodology.html) explains exactly how we count each outcome, including timeouts.

> **A note on the "no patch" bars.** Those models weren't crashing or looping. They hit limits we set: a cap on conversation length (131,072 tokens) ended about two-thirds of the empty attempts, and a one-hour clock per attempt ended most of the rest. The cap counts a model's earlier reasoning, which the gateway's own token counts seem to leave out, so reasoning-heavy models hit it sooner. Read the bottom of the leaderboard as what these models did inside our limits, not as a ceiling.

Models also don't solve the same tasks. Together, the 12 models solved 1,586 of the 2,294 tasks. 708 tasks (31%) stumped every model, 82 were solved by all twelve, and 149 were solved by exactly one. Some models owned a handful of tasks no one else solved.

![Bar chart of tasks solved by exactly one model. kimi-k3 has 34, glm-5.3 has 23, deepseek-v4.1-flash-1017 has 22.](https://spec-kitty.ai/cms/blog/swe-bench-baseline-unique-tasks.webp "Tasks where exactly one model passed the tests.")

Tasks where only one model passed the tests. [Browse the tasks](https://benchmark.spec-kitty.ai/instances.html) to see which ones stump everyone.

## How this sits next to the official leaderboard

We put our results beside the public SWE-bench leaderboard for context. We ran a simple, fixed setup, [mini-SWE-agent](https://github.com/SWE-agent/mini-swe-agent), with a single attempt per task, and we are not an official submission. Official entries use different agents, budgets and tools, so this is context, not a controlled comparison.

![Dot plot placing our 12 models against 24 official SWE-bench leaderboard entries for the full test set. Official entries run from 0.2% to 52.6%, with a median of 18.5%. Our models run from 16.4% to 55.0%.](https://spec-kitty.ai/cms/blog/swe-bench-baseline-official-leaderboard.webp "Our 12 models beside 24 official SWE-bench leaderboard entries.")

[See the comparison for each subset.](https://benchmark.spec-kitty.ai/compare.html)

## Why we needed a baseline

We're about to benchmark Brain Kitty and Spec Kitty on these same tasks. We can only say something honest about what they add if we've first measured the underlying models without them: same tasks, same grader, one attempt each. This run is that baseline. It also means that when our next numbers appear, you can hold them against these and see the difference.

**Next, the top three with our limits lifted.** Because our caps cut some attempts short, we'll first rerun the top three standard models (leaving out the uncensored and abliterated variants) from this benchmark on the 300-task Lite subset with the caps we set on conversation length, steps and time removed: `deepseek-v4.1-flash-1017`, `glm-5.3` and `kimi-k3`. That shows how much our limits mattered, and it gives Brain Kitty and Spec Kitty a cleaner baseline for multi-turn workflows that can run for hours.

We'd rather publish the baseline in full than quote a single number. You can see exactly what we ran on the [methodology page](https://benchmark.spec-kitty.ai/methodology.html).

## What ran it

**Mirantis TokenFactory** gave us one OpenAI-compatible endpoint and one API shape for all 12 models, with tool calling working across every route. That sounds small. In practice it meant we wrote our request logic once and pointed it at a dozen different models, instead of maintaining a dozen integrations. It wasn't flawless: we hit a serving problem on one model route, worked around it, and sent the gateway team the evidence so they could act. We wrote that up in the [incident notes](https://benchmark.spec-kitty.ai/methodology.html). You can get access through the [Mirantis TokenFactory portal](https://portal.trytokenfactory.dev/).

**exe.dev** supplied the machines. We created 16 worker VMs from the command line, and their creation times span 211 seconds. Each was a named Linux machine we could reach over plain SSH, with Docker and systemd, which is what our deployment scripts already assumed. We started on 8 vCPU / 32 GB and resized the same VMs in place to 16 vCPU / 72 GB when the run needed more room, without throwing away our queues. Its credential integration meant our gateway key never had to sit on any worker. The run covers about 63 hours of recorded run time. The scheduling, recovery and grading logic is ours; what exe.dev gave us was a place to run it that behaved like ordinary Linux. [Details on the infrastructure are on the methodology page](https://benchmark.spec-kitty.ai/methodology.html), and exe.dev is at [exe.dev](https://exe.dev).

![The exe.dev dashboard during the run, with 17 machines under the swebench tag: the 16 workers and the controller.](https://spec-kitty.ai/cms/blog/swe-bench-baseline-exe-dev-vms.webp "The exe.dev VM list during the run: 17 machines under the swebench tag.")

## All the data is public

We're publishing everything we recorded, because a leaderboard you can't audit isn't worth much.

- **Every attempt.** All 27,528 attempts, each as one compressed file with the model's full conversation, its patch, and the grading logs. They're about 3 GB in total, in the [repository](https://github.com/spec-kitty/swe-bench-trytokenfactory/tree/main/evidence).
- **Checksums.** A [manifest](https://github.com/spec-kitty/swe-bench-trytokenfactory/blob/main/evidence/MANIFEST.sha256) lists a SHA-256 for every file, and we verified all 27,528 attempts against our source data.
- **Predictions and results per model,** ready to load.
- **The code** that ran, graded, packed and verified all of it, under the MIT licence.

If you do research on agents, model behaviour or benchmark design, [the researchers page](https://benchmark.spec-kitty.ai/research.html) lists what you can do with it. The [evidence page](https://benchmark.spec-kitty.ai/evidence.html) shows what's in the package and how to check it.

The attempts contain text from open-source projects' issues and code, so the usual rules for that material apply to anything you build from it.

## Look for yourself

Pick any model and read its attempts: [start here](https://benchmark.spec-kitty.ai/). If you spot something we got wrong, [open an issue on the repository](https://github.com/spec-kitty/swe-bench-trytokenfactory/issues). We'd rather fix it in public.