Kimi K3 Is Competitive with Fable; Kimi K3 and Fable Is SoTA

Summary

K3 is a frontier quality open model at a fraction of the cost. Even bigger is that it complements Fable predictably, which makes it possible to get the highest quality intelligence by routing tasks.🧭 tl;dr: We ran Kimi K3 (open) against Fable 5 (closed) on ~1,000 agentic tasks finding:We achieved 93% accuracy with routing between K3 and Fable.Results were up to ~50X more cost effective than Fable alone on long agentic loops, and consistently lower cost across every use case.How We MeasuredWe averaged benchmarks, each aimed at a different kind of work, and ran K3 and Fable 5 through the same harness. About 1,030 tasks in all, in real agent loops.FamilyWhat it testsTasksSWEReal repo bug-fixes (SWE-bench style)460TerminalLong agentic ops: security, crypto, reverse-eng, sysadmin89AlgorithmicLeetCode / AtCoder-style problems100Multi-LanguageImplementation across six languages225LegalA legal-agent benchmark (lawyer-graded tasks)120One quick definition before we get into the results. Oracle routing is a method for measuring the best theoretical performance by running the task through each model and then picking the cheapest correct option (the cost/performance ceiling). In a practical router, you don’t get to run your task against multiple models. The router makes a prediction of which model has the best cost and quality trade off, but ultimately it’s a guess.In this study, oracle routing demonstrated K3 is selected for 72-96% of tasks. This suggests a near-perfect router might be achievable, by learning the difference between day-to-day tasks and the true long tail of frontier work. It will require an order of magnitude more routing data, and real world performance to say definitively.K3 is a good model.From a 10,000 foot view, it can be easy to look at both models and call the head-to-head a tie. For example, if you look at SWE, the headline benchmark, K3 gets 92.4%, Fable 92.6%. Across the five types of tasks we benchmarked on, the two models tend to stay within a few p...