Are these comparisons based on API price? If so, then the OpenAI/Anthropic subscriptions (which are a 15-40X discount on model pricing) turn the comparison into a bloodbath, even more in favor of the closed source models.

These general intelligence/capability benchmarks don't apply to your specific task. If you are coding, you want just benchmarks for coding. I don't see performance+price at this benchmark for coding. When you look at just coding, DeepSeek is Pareto optimal for a good portion of coding benchmarks. For example here, although even this is very flawed because it only benchmarks max/xhigh effort: https://artificialanalysis.ai/agents/coding-agents

Closed models might be better generalists because they don't have that much constraint on size and come from well funded labs that can afford to better train for generalization. If a model's actual goal is to be open then smaller is better.

I see this take frequently; that coding does not require general intelligence. Perhaps I am in the minority, but rarely do I find myself in coding situations that are completely divorced from other subject areas. The code is modeling or interacting with the world, either through business processes, or (in my case) simulating the physical world, and I need the agent to have some working knowledge of not just the code, but also physics, materials, or social conventions. Are you all really able to just "write code" without dealing with all the externalities?

Yes, 100% agree. Without more details on how they got their data, it's very hard to understand why they don't value DS Flash 4.1 as much as pretty much every other analysis I've seen (and my own experience).

I'm just going to paste my comment from another post since it is applicable here too, maybe even more so.

I've had really great results with cheap flash agents using a system that has a parent agent that spawns 2 sub agents. They remain active throughout the whole session and don't lose context.

The 2 sub agents get the exact same prompt and tools as each other and they independently answer but before responding back up to the parent they exchange their findings which must be backed up by a citation and then debate their findings and citations repeatedly until they can agree on correctness of all points. They are also heavily encouraged to never assume and always verify with the prompt and tools and guards where they can be realistically enforced.

It churns through tokens but tokens are crazy cheap with some of the plans out there. Could Opus one shot most of those same queries in like 1/4 time? Yep, but that's not the point for me.

DeepSeek Flash for parent then DeepSeek Flash and GLM 5.3 flash as the debating subagents gives best overall results in my experience.

For context I do equal parts sysadmin/devops and having them write software

What harness are you using to organize them? How are they kept from losing context as the session grows?

Without knowing which dots are which, this is useless.

Let’s say I want to run qwen 27B or flash next on a machine I own, off solar panels that are also sunk cost.

How did they price the marginal cost of a token? Depreciation on the box + reverse billing opportunity cost with the power company?

This article isn’t about that. It’s about comparing hosted models.

IMO then it is missing the forest for the trees regarding open weight models. Qwen 3.8 Flash Next or 27B, for example, are accessible enough to run at home/office at very usable speeds. Provided memory companies don't collude to keep margins at 80-90% they are at the moment, it seems to me this will become a lot more accessible for many use cases while models at this size still have room to improve.

That's to be expected since all the proprietary models were just released. Give it six months and the Chinese labs will distill from the proprietary models like they did last time.

Yes, but then at that point the newest proprietary models will dominate the price-performance frontier.

The article itself goes against this intuition, open models have taken up as much as 40-50% of frontier for capable/affordable. Right now is very much a low point. The fact there's a lot of churn as new models are released is not unexpected, I think.

If required, so far I'm not impressed with newest proprietary. i have a claude sub mostly rotting at this point, so i come back from time to time until the year long sub expires, but at this stage i will not renew and see no point in doing so.

Have you tried Opus 5.5 and Fable 5.1? They're still better than any of the available open-weights models. (And rumors are Fable 5.5 is coming out tomorrow or next week.)

Despite its simplicity, price-performance is not the only metric; this obsession is a typical consequence of extreme overextension of the Friedman doctrine, and it is why so much software and hardware has been qualitatively shitty even when it matters that it’s not. That something can be measured doesn’t automatically make it preferable as a decision-making basis over intangibles.

It’s pretty obvious that local AI has a higher TCO if you look only at money. But there are other things to be considered.

True but let's think about whether this actually matters in practice going forward, given that today, Deepseek/any OS-27B variant can do all the things that most devs need it to.

Let's scale this out 3 years into the future: - Opus 6.7 costs an arm and a leg to run for 2 prompts - Astra 7.89 literally requires you to buy-now-pay-later because it's so expensive - As a bootstrapped dev, your only option is serving local Qwen-4.04-27B that runs on your old RTX (that you bought in 2024)

Sure, it's worse performing. But if you can't afford the best ones in the first place, what's so bad about having a "pretty decent" model that you run essentially free (only paying electricity) if it does what you need it to?

... and if we're all in the permanent underclass anyway, what does it matter that your local model is dumber than the SOTAs? At least you have one! Some people who can't afford GPUs still have to type emails by hand.

(jk, this is actually a cool dataset though ^)

Even as the article points out, this fluctuates constantly. In a few months, we're guaranteed to get a new batch of open models, and the balance will shift again.

What's more important is that open models can be inspected, they can be customized, and you don't have to worry that a company will decide that you're not allowed to use them for stuff like reverse engineering things or do security work.

That's why open models, and especially ones you can run locally will always be important. When you're renting closed models, you're basically a serf who has no say in how the model can be used.

Oh really? I have tasks that DeepSeek 4.1 Flash is absolutely murdering everything else on price–performance.

Not to mention the insane discounts I'm getting with subscriptions on it

I'm running full swarming on a 90$ subscription lol

Maybe that’s in the 3% mentioned in the article.

For general use-case, closed-weight frontier models dominate.

Open-weights excel primarily at coding.