Until recently, a serious agent needed a closed model. Open models could chat and code, but give one a browser, a shell and forty tool calls in a row and it lost the thread. This summer that changed. Moonshot released the weights of Kimi K3, and independent leaderboards such as BenchLM now score open and closed models on the same agentic benchmarks. The gap at the top is a few points, not a league. That is why botfleets runs on open-weights models served in the EU.
What the Kimi K3 paper shows
Kimi K3 is a 2.8-trillion-parameter mixture-of-experts model with 104 billion active parameters and a one-million-token context window, with weights on Hugging Face. Its post-training is reinforcement learning in agentic and coding environments where one rollout runs “over hundreds or thousands of tool calls”. The abstract concedes it “still trails the most powerful proprietary models, namely Claude Fable 5 and GPT-5.6 Sol”. The agentic rows of its evaluation table, all at maximum thinking effort, say how much:
| Benchmark | Kimi K3 (open weights) | Claude Fable 5 | GPT-5.6 Sol |
|---|---|---|---|
| BrowseComp (web research) | 91.2 | 88.0 | 90.4 |
| MCPMark-Verified (MCP tool tasks) | 94.5 | 87.4 | 92.9 |
| MCP-Atlas | 84.2 | 84.7 | 83.6 |
| Toolathlon-Verified | 76.5 | 77.9 | 74.9 |
| AutomationBench | 30.8 | 29.1 | 29.7 |
| Terminal-Bench 2.1 | 88.3 | 88.0 | 88.8 |
| JobBench | 54.3 | 57.4 | 45.4 |
| GDPval-AA v2 (Elo) | 1686 | 1747 | 1736 |
On web research, MCP tool use and automation tasks the open model is level with or ahead of both closed models. It trails on JobBench and GDPval, the broad “do a professional’s job” evaluations. These are Moonshot’s own numbers, so they need a second source.
How it benchmarks against closed models
BenchLM aggregates published results into one weighted score per model and never interpolates missing scores. Its agentic category covers Terminal-Bench, BrowseComp, OSWorld-style computer use, tool calling and workplace tasks. From its 21 September 2026 snapshot:
| Model | Weights | Agentic | Overall | Price in / out |
|---|---|---|---|---|
| Claude Fable 5.1 | closed | 80.2 | 84.6 | $10 / $50 |
| Claude Opus 5 | closed | 79.1 | 81.9 | $5 / $25 |
| Kimi K3 | open (BenchLM status: pending) | 72.1 | 74.4 | $3 / $15 |
| GPT-6 Astra | closed | 70.6 | 83.8 | $10 / $50 |
| GPT-5.6 Sol | closed | 69.9 | 80.7 | $4 / $20 |
| GLM-5.3 | open | 68.7 | 67.1 | not listed |
| Qwen3.8 Max | open | 67.5 | 73.3 | not listed |
| Qwen3.8-27B | open | 63.9 | 64.2 | not listed |
Kimi K3 scores above GPT-6 Astra and GPT-5.6 Sol on agentic work at 30% of the leader’s output price, and BenchLM calls its overall score “conservative” because coverage is still partial. Only the three Claude models sit above it. Further down, Qwen3.8-27B, a model that fits on one node, scores 63.9, ahead of Claude Opus 4.8 and GPT-5.5. The best open-weights models now sit eight to thirteen points behind the best closed model and above most of the closed field.
Why good enough is enough
An agent is a long loop of small steps: call a tool, read the result, decide, call the next one. A scheduled automation runs that loop every hour, forever. Once a model clears the reliability bar for those steps, three things matter more than the last eight benchmark points:
- Cost per run. Kimi K3 output tokens cost 30% of Claude Fable 5.1’s at list price; smaller open models cost less, or nothing if you host them.
- Where the data goes. Open weights can be served by any provider, including EU providers under GDPR, or on your own hardware.
- Freedom to swap. Build for the upgrade: with open weights a model swap is a download, not a contract.
The remaining gap matters for open-ended, multi-day tasks with no test to run and a human expert as the only judge. It matters little for a task with a success condition a machine can check. An automation is the second kind.
How botfleets uses open-weights models
botfleets runs AI automations that run themselves. You describe a recurring job in plain words, such as “fetch the receipts from my inbox every Monday and filter out personal expenses”. botfleets drafts it as a graph of tasks, tests it against the real services, runs it on a cron-style schedule and repairs it when a run fails. All inference runs on open-weights models served by GDPR-compliant providers in the EU, and the platform is hosted in the EU.
- Tools over MCP. Automations connect to tools over the Model Context Protocol, plus the web, email and HTTP APIs. That is where the open frontier is strongest: Kimi K3’s 94.5 on MCPMark-Verified is the highest score in the table above. While drafting, botfleets picks the best-ranked server for each step on MCP Toplist, preferring ones that need no key.
- Test, then schedule. A draft is tested against the real services before it is saved. “Is the model good enough?” becomes a pass or fail per automation.
- Repair, not retry. When a run fails, botfleets diagnoses the step that broke and redrafts it. Each run leaves notes that improve the next draft.
- Isolated branches. Tasks form a directed acyclic graph; a failing branch does not stop the branches that share nothing with it.
None of this needs a model that can pass the bar exam. It needs one that calls the right tool with the right arguments a few hundred times a day and keeps your invoices in Europe. Two years ago no open model could. Now several can.
Caveats
The paper’s scores are self-reported at maximum thinking effort, and BenchLM still lists Kimi K3’s weights as pending verification. The eight-point gap at the top is real: for “refactor this 200-file codebase over a weekend”, the closed leaders remain the safer bet. For “every morning, do these twelve things and tell me if one fails”, they are not.
botfleets is opening access gradually. Join the waitlist, or read the botfleets project page.
Sources: Kimi K3: Open Frontier Intelligence (arXiv 2607.24653, v2, 7 August 2026); BenchLM agentic leaderboard and Kimi K3 model page, snapshot of 21 September 2026; botfleets.com. botfleets and MCP Toplist are built by BIFF.ai.