> [...] Let's maybe add a helmet? It could improve riding theme, but may obscure head. Maybe a small cycling cap or helmet? The user didn't ask; can add red helmet? Might be cute. But pelican with big beak; a helmet might obscure. Better maybe no.
Simon’s karma exceeds yours by about 1,300x — so most of us disagree with you. Like me, I disagree with you. I like the pelican benchmarking, and I like it when Simon stays on top of models for both public good reasons and because it saves me time and effort.
Hy4 apparently has ludicrous traction on OpenRouter already (https://openrouter.ai/tencent/hy4-preview), with trillions of tokens processed in a couple days: more than GLM 5.3 in a week. That said, it's relatively cheap with a 5% cache cost when everyone is still doing 10%/20% cache costs, so Hy4 may be more compelling.
Btw I still haven't came across any decent model that is <$0.01/MTok cache costs apart from deepseek thru their official API (even with the price increases).
Seems like a bit of an opportunity for someone to take - drop cache read costs significantly.
That's because Deepseek invented the paradigm of prompt caching, they are the SOTA when it comes these techniques. Despite them open sourcing all their research, nobody beats them.
edit: I do wish openrouter would let you sort providers by Cache Hit % and Cache cost. These are the only things that matter to me at this point when choosing a provider.
Caching was always here, you don't need to do anything special to get it on a single user local backend running a base model or a chatbot in the first place. Among commercial providers, OpenAI adopted it in 4o first.
OpenRouter randomizes which provider gets your request by default right? I think you have to pass a specific provider in the request to prevent that. (Or set up a preset or something.)
This behavior makes it so you don't benefit much from the caching, unless you pin it to a single provider.
This is not true, there isn't even a way to see a cache hit % model for a specific model, that wouldn't make any sense. You are confusing what I'm saying with cache cost, that has nothing to do with effective cache hit %. I'm talking about when you click on a specific provider for a specific model, you can scroll down on the view and see their cache hit % for that model [0].
These cache Hit % are accurate, I've done a ton of testing of this myself. The cache hit % is one of the most important metrics as far as estimating cost. There are many providers with cheap cache reads, but have an effective cache hit % of 30%, making their cheaper cache pricing meaningless compared to another provider who charges more but has a 85% cache hit percentage.
scroll down on the provider/model card and you'll see a field called cache hit %, its different for every provider/model.
I don't use routing on openrouter, I strictly use models with a single provider and no fallback, at least for use with harnesses its pretty dumb to route requests to multiple providers you are busting your cache every other request and increasing costs by 20-50%.
You can sort by the cost cache cose, but you cannot by cache hit %. You have to click on the provider and see what their cache hit % is. A provider could have a super low cache cost, but a 50% cache hit percentage, making the cheap price of cache read's meaningless.
I experimented with Hy3 for a project and was surprised with how good it was. I don't know if it's good for coding, but as a general purpose agentic model, it was only beaten by deepseek4-flash in our tests. It was so close to deepseek behaviour I kept thinking it must have been forked from it.
> Notably, Hy4 preview also contributed to its own development process, participating for the first time in the automated optimization of training methods, data strategies, evaluation frameworks, and low-level operators. The model proposed approaches, ran experiments, and iterated based on the results, with the resulting code, logs, and feedback feeding into subsequent rounds of exploration. This established an early-stage recursive self-improvement loop.
This reminds me of one of the predictions from https://ai-2027.com/ . Only that there it's "OpenBrain" doing this, not the Chinese. And the authors of that paper were also slightly wrong about "Mid 2026: China Wakes Up": China woke up already a while ago. And:
> But China is falling behind on AI algorithms due to their weaker models. The Chinese intelligence agencies—among the best in the world—double down on their plans to steal OpenBrain’s weights.
No need to steal anything, they have already caught up.
And then there's this prediction for February 2027:
> Officials are most interested in its cyberwarfare capabilities: Agent-2 is “only” a little worse than the best human hackers
If the distillation "attacks" created useful inputs to open weight models, ai-2027 was directionally correct that the Chinese would find ways to extract IP from western firms. (Scaled account creation and grinding outputs etc is not a dramatic story element as spies, though!)
Whether the distillation has constituted "attacks" or has or will meet the bar of "stealing" IP is not super interesting to me, though.
Is anyone here working on a problem for which current generation LLMs are inadequate, but that could possibly be solved by the next release of a first tier LLM?
Or is it like bicycles? Unless your problem is named Tadej, you don't need a $13,000 bike.
My experience is that even Opus 5 still tends to write buggy or low-quality code and makes serious mistakes when analyzing code. It's a lot better than before but still not something I trust. I've had less experience with Fable since I can't use it at work; I hear it's a step up but still has its limits.
For large tasks like a web browser or a compiler, even expensive swarms of frontier LLMs have not been shown capable of producing codebases that actually work. (Anthropic built a C compiler with Opus 4.6 but it lacked optimizations and apparently hit a complexity wall.)
I also want to use LLMs for reverse engineering, but apparently it's pretty hit-or-miss, especially if you're forced to use open-source models to avoid restrictions.
This reply is particularly interesting to me because most of my experience with actually using LLMs to get work done is with coding agents. But I only have a fairly narrow set of experiences: two pretty large solo Flutter projects. I am currently really pleased with Gemini as a coding agent. It could improve, but I think improvements are going to come from marginal gains in the harness and training material so it can catch things like misconfigured permissions in platform specific areas.
It's also interesting because, while coding agents are important and are a notable success, they are never going to be a multi trillion dollar business. And are there any other domains where LLMs have such a large impact?
Scientific physics simulations - even the frontier models just engage in rationalization of obviously unphysical results instead of understanding the system. They have the rote knowledge but fail to apply it unless their hand is held through the process.
Yes. Most of us are, still. The frontier is currently both at expanding ‘common sense’ / non-cheating outcomes for imprecisely specified software (that’s all software), and at expanding autonomy - ability to work longer unsupervised with success, oh, and also at expanding outside contextual reasoning about what’s being built, as in “hmm, that doesn’t look right or make sense, let me explore that.”
For me personally, the answer is no. Fable is adequate to do basically anything I want to do. My perspective, broadly speaking, is that we've saturated most of the benchmarks because we've largely saturated our capacity to verify models' work at scale. What's left is context-bound verification, i.e. the problem of ensuring that output matches intent and ambiguities in prompting were resolved correctly. Further advances in autonomy do not make that latter verification problem easier. If anything they make it harder as the output per task becomes more complex and therefore more taxing for a human to verify.
The solution to that (to my mind) would be not a better model but a basic shift in architecture beyond the current paradigm and into a setup where agents have durable, plastic memories and undergo contextual individuation over time. But at that point agents start to become quasi-persons and not tools.
I saw a laptop earlier in the train that I asked ChatGPT, Claude and Gemini what it was, providing a brand, screen size and ports description. Gemini could never figure it out, Claude and ChatGPT eventually did, after multiple rounds of indirection, giving completely wrong answers (there was a perfect match for the problem statement, they all explored alternatives first).
LLMs are (probably) amazing at things I don't care about, and still suck at the mundane stuff you would have the marketing tell you they excel at.
The results are boring. Not because the content is boring, but because you can so easily remix the results. Human curation is what create values with these, not dumping and consuming. A personal perspective of a human being ups the respect, where the exact same sentences generated by an LLM carry no such value.
That sounds like an interesting challenge. Have you seriously considered solving it? Because in about 10 seconds I came up with a process that should work, provided enough compute power. Simply model the traditional film making process by starting with a script, character stories. Design your world, then design the storyboard, and all the scenes. Create a list of all the visual elements that need to be replicated between all the scenes. Then you have to built prompts and reference art of the objects, faces, people. Make sure to do multiple takes of each scene, and have the vLLM critique and analyze the performances and technicalities. Should work?
I think, also, like in the traditional film makers career, this process should be built iteratively, start with a fast food commercial, then do a music video, then you can probably do a short film. Continue to improve the process, and one day I’m sure the LLM film studio can make you any movie you want, provided you have enough tokens.
On that topic, check higgsfield cinema studio 4; they already provide amazing tech for the cinematic experience, somewhat similar to what you are describing.
I'm getting a Poe's Law feeling. I'm genuinely unsure about whether this post is a stone cold parody or not. I think I need to turn off the internet and go to bed.
I was given a picture cube, which is like a Rubik's cube but every side is a unique picture. It came scrambled and I don't have an original reference image. I like to take videos of it and give it to llms to solve. I call it my agi test because it hasn't been solved yet
Yes, lots – I think that folks will hopefully discover more of these as they scale up their ambition, now that LLMs make a lot of previously difficult things far easier.
This is the exact same type of comment I heard about computer hardware upgrades for three decades in a row.
“Very few people actually require a Pentium workstation, a 486 is perfectly adequate for the majority”
The logical fallacy is taking an extant distribution of “product capability” that is priced to fit what the market will bear and assuming the “next upgrade” simply tacks on a little bit more to the right hand rail of that curve.
No!
It shifts the entire curve!
Everything for everyone gets better and the top 1% of the most demanding users will continue to pay the same-ish premium.
“Nothing” will change.
Look at it this way: you can buy a $200 laptop for your kid or a $20,000 Mac with an M5 Ultra processor.
BOTH are vastly more powerful than either a $200 PC or a $20,000 “workstation” from 20+ years ago.
Intel didn't just surf some natural wave of demand for higher power personal computers. Intel found new needs for powerful PCs, especially in gaming, and they put a lot of marketing and industry relations dollars behind PC gaming.
In other words. PC users didn't figure out that they could buy super powerful PCs and play games on them, that was a carefully managed market transition.
If the models stay open, it seems like everybody but anthropic/openai wins. i literally can’t see a downside. We can post-train the models to know about tienanmen square.
I used Hy3 quite a bit for the type of tasks it was suited for. Excited about this. My one concern over Hy3 was speed. In theory, it could be served much faster as a smaller model but it was relatively slow everywhere I could get it (including from Tencent directly) but also several other inference providers.
In my evals, I saw an unprecedented jump between preview and final release on Hy3, from unusable to competitive. Did you see similar in preview vs release version?
Oh yes! I forgot about that. Yes, you can see this in benchmarks about hy3 preview and hy3 release still today because they measured them separately - it was significant.
The first column has both the Hy4 and Hy3 scores overlaid on one another (Hy4 is darker blue and the taller one), with both scores written below the top of the respective bar - maybe you're seeing that?
Vague and without substance. It easily passes as sarcasm, which means criticism but without any commentary, else it is sincere... but doesn't have any commentary.
Im not particularly fan of the chinese, but no chinese model asked for my citizen card yet to complete a task. And apparently no chinese provider uses persona to manage this kyc information. OpenAI does, in EU space. Just saying.
Claude and OpenAi are not allowed in Venezuela, so I thank China too and I swear to god I'll never use them and will be rooting for chinese models forever
> Tencent has released and open-sourced Tencent Hy4 preview, a next-generation large language model with 770B total parameters and 49B active parameters, and a context window exceeding 1M tokens.
There are no open source models, at least not useful ones (yet) [0]. Open weight is not the same as open source. The current "open weight" models are just opaque binary blobs you can run on your own computer instead of through a web API.
Imagine thinking that running a Photoshop binary on your own computer instead of through a SaaS web app means that it's "open source". Of course you think that's ridiculous.
Parent poster is technically right - open “source” implies the source used to make something is open. The model source is training data and code, not just weights.
But the reality is, the weights are a useful artifact that you can use to create derivative works. So, dismissing it as a photoshop binary is as technically wrong as calling it open source.
> Maybe add sunglasses? no.
> Maybe add water? no.
https://tools.simonwillison.net/markdown-svg-renderer#url=ht...
Just tap on the [-], and upvote what you find more interesting :)
https://martinalderson.com/posts/watch-out-for-cache-read-co...
Btw I still haven't came across any decent model that is <$0.01/MTok cache costs apart from deepseek thru their official API (even with the price increases).
Seems like a bit of an opportunity for someone to take - drop cache read costs significantly.
edit: I do wish openrouter would let you sort providers by Cache Hit % and Cache cost. These are the only things that matter to me at this point when choosing a provider.
Caching was always here, you don't need to do anything special to get it on a single user local backend running a base model or a chatbot in the first place. Among commercial providers, OpenAI adopted it in 4o first.
This behavior makes it so you don't benefit much from the caching, unless you pin it to a single provider.
I thought you had to actively manage caches, do you not?
These cache Hit % are accurate, I've done a ton of testing of this myself. The cache hit % is one of the most important metrics as far as estimating cost. There are many providers with cheap cache reads, but have an effective cache hit % of 30%, making their cheaper cache pricing meaningless compared to another provider who charges more but has a 85% cache hit percentage.
[0]: https://openrouter.ai/deepseek/deepseek-v4-flash-0731?endpoi...
scroll down on the provider/model card and you'll see a field called cache hit %, its different for every provider/model.
I don't use routing on openrouter, I strictly use models with a single provider and no fallback, at least for use with harnesses its pretty dumb to route requests to multiple providers you are busting your cache every other request and increasing costs by 20-50%.
wouldn't trust they dont do Capitalism like the rest of the AI field.
Like lobbying the US president to harm their competitors?
This reminds me of one of the predictions from https://ai-2027.com/ . Only that there it's "OpenBrain" doing this, not the Chinese. And the authors of that paper were also slightly wrong about "Mid 2026: China Wakes Up": China woke up already a while ago. And:
> But China is falling behind on AI algorithms due to their weaker models. The Chinese intelligence agencies—among the best in the world—double down on their plans to steal OpenBrain’s weights.
No need to steal anything, they have already caught up.
And then there's this prediction for February 2027:
> Officials are most interested in its cyberwarfare capabilities: Agent-2 is “only” a little worse than the best human hackers
I think we're past that point now, too…
Whether the distillation has constituted "attacks" or has or will meet the bar of "stealing" IP is not super interesting to me, though.
- if you're gonna order the rest of the bar chart by rank, order your model accordingly.
- if you're gonna highlight a winner in a table of benchmarks, don't highlight your entire model row in the table.
Etc etc
They can do the difficult small level optimization, the boring but tedious code but cannot be tasteful.
That means I'm more valuable and more productive. Good stuff
Or is it like bicycles? Unless your problem is named Tadej, you don't need a $13,000 bike.
For large tasks like a web browser or a compiler, even expensive swarms of frontier LLMs have not been shown capable of producing codebases that actually work. (Anthropic built a C compiler with Opus 4.6 but it lacked optimizations and apparently hit a complexity wall.)
I also want to use LLMs for reverse engineering, but apparently it's pretty hit-or-miss, especially if you're forced to use open-source models to avoid restrictions.
It's also interesting because, while coding agents are important and are a notable success, they are never going to be a multi trillion dollar business. And are there any other domains where LLMs have such a large impact?
The solution to that (to my mind) would be not a better model but a basic shift in architecture beyond the current paradigm and into a setup where agents have durable, plastic memories and undergo contextual individuation over time. But at that point agents start to become quasi-persons and not tools.
Both animated and live action results would be acceptable.
Unfortunately most existing LLMs lack the capability to maintain context across tens of thousands of frames.
I think, also, like in the traditional film makers career, this process should be built iteratively, start with a fast food commercial, then do a music video, then you can probably do a short film. Continue to improve the process, and one day I’m sure the LLM film studio can make you any movie you want, provided you have enough tokens.
EDIT: Your username doesn't help, either.
“Very few people actually require a Pentium workstation, a 486 is perfectly adequate for the majority”
The logical fallacy is taking an extant distribution of “product capability” that is priced to fit what the market will bear and assuming the “next upgrade” simply tacks on a little bit more to the right hand rail of that curve.
No!
It shifts the entire curve!
Everything for everyone gets better and the top 1% of the most demanding users will continue to pay the same-ish premium.
“Nothing” will change.
Look at it this way: you can buy a $200 laptop for your kid or a $20,000 Mac with an M5 Ultra processor.
BOTH are vastly more powerful than either a $200 PC or a $20,000 “workstation” from 20+ years ago.
Look at: https://arena.ai/leaderboard/text?q=openai&utm_source=chatgp...
The “budget” 5.5 Instant model beats o1 and o3 which were “pro” models at the time of their release!
In other words. PC users didn't figure out that they could buy super powerful PCs and play games on them, that was a carefully managed market transition.
What is going to do the same for LLMs?
The first column has both the Hy4 and Hy3 scores overlaid on one another (Hy4 is darker blue and the taller one), with both scores written below the top of the respective bar - maybe you're seeing that?
But, what bars are clearly off? I couldn't spot any.
EDIT: I don't know why I'm being downvoted by AI bots.
There are no open source models, at least not useful ones (yet) [0]. Open weight is not the same as open source. The current "open weight" models are just opaque binary blobs you can run on your own computer instead of through a web API.
[0] https://allenai.org/
Imagine thinking that running a Photoshop binary on your own computer instead of through a SaaS web app means that it's "open source". Of course you think that's ridiculous.
Models are lossy compressed datasets you can pick up and amend (fine tune / continue training / alter) according to license they were released under.
Hy4 is released under OSI approved Apache License 2.0.
But the reality is, the weights are a useful artifact that you can use to create derivative works. So, dismissing it as a photoshop binary is as technically wrong as calling it open source.