I think the quality of Codex CLI (and Claude Code) is an honest reflection of the status quo of LLM-driven development. The future doesn’t look really promising.
To be fair we've gone from barely capable at writing code to what we have today in barely 3 years. I wouldn't put money on the counter bet that they won't get beyond this hump.
It has pretty rapidly reached the point where I do not think any acceptable LLM-harness is widely available. Every single one I've tried has broken in bizarre ways on even minor updates.
I used codex the first time yesterday and quickly decided I need to sandbox it and came up with a pretty easy and straightforward solution (I think).
I use nixos and use a little wrapper that wraps Codex CLI in Docker. Run codex-box /path/to/project and it starts or reuses a container with only that project bind-mounted, keeping login and history in a separate volume. It uses tmux for detaching and reattaching
I've used the CLIs for Codex, Gemini and Claude and they all give off the feeling of being on an very rickety, very fast runaway train.
You install an update. Oops, they added a sandbox that doesn't work inside a docker container, it's stopped working.
You install an update. Oops, you can't log in inside a docker container any more, it's stopped working.
You install an update. Oops, they started catching the copy+paste shortcut and replaced it with one that doesn't work.
You revert a broken update. Oops, the update changed the format of your login details, the old version can't read them, gotta log in again.
You install an update. Oops, can't scroll with the mouse wheel any more.
Not showstopping bugs (if you're used to persuading broken oauth logins to work) but it certainly makes updating an "ugh, what have they broken this time?" experience.
Seems like its just a bunch of young engineers making bad architecture decisions. Not sure I would blame the culture of vibe coding everything. It might just be bad taste.
The daemon has many bugs. For example, when auto-updating, it would randomly ruin the permissions and block the goal as .git became read-only. Generally, it is extremely annoying how many bugs Codex has. I basically can't trust it to execute on any goal since it would often get blocked every few hours due to random bugs.
It also fills up my HD with old versions.
I investigated and it took up something like 12 gigabytes in total on my system; which seems insane to me, it’s just a harness?
People moan about how badly made Claude code is but Codex seems much worse on every metric
All of the harnesses I've used are terrible at cleaning up after themselves. They leave all kind of clutter behind, to the point where on my dev VM, I now have an LLM run once a day to just look through the disk and delete everything that looks like it's leftover garbage.
Can't wait until we get really well designed harnesses written in a compiled language that doesn't bloat my computer. Will probably have to be written by a company other than OpenAI because codex is like the exact opposite of everything I just listed
it's minimal and somewhat opinionated, not integrated well with memory subsystems (although that's kind of the point)
my only real criticism is that last I played with it it deals with slow provider responses very poorly , so lots of locally hosted llms require a lot of tuning to get it to handle streaming/non-streaming/timers well.
way less minimal (but still pretty minimal) I prefer Pi; but different markets I think.
hax fit my openbsd laptop well, espescially since Pi and related derivatives are kind of un-spokenly incompatible with the BSDs.
It's because they statically linked it, and probably rely on a trillion pointless crates (as is standard in Rust development). A good statically linked harness can easily be ~5MB MAX
I did, and the weird things you have to do to make work at least decent. I’ve been doing evals and I got some things better than opencode but not as good as pi. Though my goal is to optimize for a browser and browser based llm so it can create stuff in a web container or at least generate ui by requesting json schema (a2ui style).
I suspect a lot of people wouldn't use AI if they had to pay per token at current rates. Yes, it's technically a different discussion, but "just write your own harness" isn't really an answer for the vast majority of us riding on the investor-subsidized subscription plans.
> People moan about how badly made Claude code is but Codex seems much worse on every metric
Also, it feels that chatgpt.com and aistudio.google.com are the buggiest websites I have seen for long time. Why it is always something related to LLMs?
for some reason I had 11 separate codex and codex-code-mode-host binaries in my ~/.bun/install/cache/@openai . each weighing about ~300 MB. might be worth a look to anyone running low on disk space.
Yes, my DEV VM tended to fill up with VSCode backups as well. I finally expanded the disk so it's less of an issue today, but for a while ENOSPC was resolved by cleaning up ~/.vscode-server/cli/servers.
I’ve been using GPT models via hax[1], a bare bones harness with minimal tools on my system and have the models rely on bash for the most part the last few months. It’s been a breath of fresh air amidst all these overengineered harnesses.
If I need special actions, I just make them CLI tools on my Mac and tell the AI model that such a tool exists(via a single line in its system prompt) and that’s it. It’s been very nice.
You seem to care, so I wanna point out a semantic nit; "such a tool exists" could be interpreted as a harness provided tool, rather than a command line tool. (Edit, on a re-read its pretty obvious you mean cli tool, but keeping this statment here as im sure it'll open someone's eyes to the difference)
I've gotta give hax a go, as a pi user who came on-board due to the no frills direction pi takes. I say takes rather than took, because regardless of the (heated) discussion surrounding mcp and code mode support, I cannot fight the numbers- augmenting the harness provided tools is a notable boost to execution round trip performance.
There aren't any documented settings to disable this, but there is a feature flag it currently sits behind. You can flip it in your ~/.codex/config.toml[1]:
[features]
daemon_auto_start = false
There are also some files that it writes to in ~/.codex at startup: pid files, lock files, things like that. You can delete them and replace them with symlinks to /dev/null, then mark those unwritable with `chattr +i` or similar. Presumably then the background daemon will crash on startup, if the feature flag ever goes away.
For whatever reason, the daemon has been refusing to start and Codex has been grumbling to me. I was wondering where the daemon had come from, and thought maybe I missed something previously and that it had just broken.
What makes ACP deficient for headless uses of LLM agents? Maybe fixing that or standardizing on another protocol designed for pluggable LLM agents (I guess ACP is more oriented towards pair programming in an IDE, at least originally) is a better idea imo.
Yep, the real issue is that it was something they didn’t have before, and then with Codex 0.157.0 enabled, they installed this persistent nonsense [1] that survives after the last instance of Codex has exited on your machine, which is just gross.
Off-topic: I think the front-end quality of ChatGPT/Codex Desktop and Android app is simply dreadful—hard to imagine how such a large number of frequent failures made it into production. Do they not use human testing at all?
You can just copy the executable then run it with --no-daemon, or edit the config equivalently. No autoupdates and no background process.
The recent terminal UI changes were more fastidious, but even those are easily disabled with CODEX_TUI_DISABLE_KEYBOARD_ENHANCEMENT=1 and the /tui command.
It's right about now that having a operating system distro model is great - you get folks who will package the software & test it out, make generally sensible defaults, and generally keep shenanigans like this from going on
Ordinary package managers are totally capable of updating thousands of programs without manual effort, including automatically and in the background if that's a thing you want. But they do it without violating users' sovereignty over their own computers or turning every app into a special snowflake that gets to install software (namely itself) in its own bespoke way.
Yeah this seems like a decision yielded by a model. One of countless examples of the model’s outputs careening into individually defensible but holistically nonsensical results
A human seems less likely to embark upon a path that most humans know leads to something a human is unlikely to want
LLMs produce amazing things, but only one token at a time. Anything beyond that is retrospectively emergent at best
Firefox does not only not prompt me for updates (it only updates if I manually run an update), even if I do update with Firefox running, it does not stop me from continuing to use Firefox without restarting it. All I did was install Firefox from flathub (packaged by Mozilla (a default configuration provided by Mozilla)).
I am running Firefox Developer Edition on macOS, where it is the default option. Have to set "Check for updates, but choose when to install" to get the similar behavior.
I've never seen that behaviour, it downloads updates in the background and then gives me a little notification that I should restart it to update, but it's dismiss-able.
Being able to turn auto-updates off is okay, but that being the default would certainly not be okay in Codex, giving the reach, what it can do and how immature it is.
Serious tools shouldn't do this. Sure, you can offer an opt-in rapid release beta update option, but if you can't go more than a month without forcing me into a "stable" update that changes the UI and behavior I'm getting used to, then I'm not going to consider your software stable and reliable enough to be worth developing expertise in using.
Software Engineering clearly is solved, that's why Google has a tiny animated banner on the AI Studio sidebar (advertising subscription upgrade), which shows a simple animated border around it and utilizing exactly 100% of the GPU to render just that banner. And there is a browser extension removing that styling element (without banner GPU load is obviously under 0.1% as it should be) released way back in May. So the almighty AI didn't manage to fix a major UI blocker for a flagship AI product in half a year now. :)
When I had a Claude account (cancelled ~May time) I vividly remember the web chat having broken editing for backtick-quoted monospace text. If you typed <space>`foobar`<space> the last space would move the cursor backward into the closing quote, and the space would be emitted inside the monospaced region. It was like this for several months (on Firefox). I always found this baffling for a product marketed to programmers.
Products moving fast doesn't mean that they're moving in the direction I want, or that I want them to move underneath me while I'm in the middle of a project.
Auto-update systems have value but are not a suitable default for professional tools.
It's not necessarily hostility. We considered different updating approaches with our legal IDE where security, stability, etc. are paramount, and having a buggy application out in the wild is user-hostile. Sure, in a dream world we'd say "don't ship until it's perfect", but that's just not realistic.
We considered the daemon thing, but ended up with a hard-update check on every boot. It's thus a user choice: if you don't like the automatic updating, you can just not run the application anymore.
And even with all that, it's hard to stay on the right side of AV algorithms, etc.
> It's thus a user choice: if you don't like the automatic updating, you can just not run the application anymore.
That's not a choice. It s just what most apps do these days, but it's the first time I've seen anyone stating this so directly, or trying to frame it as a choice.
(And behaviorally, this makes your product just a SaaS website with extra steps.)
I guess I mean it's a choice in the sense that it's a transparent part of the product that one chooses how and when to use (or not). That's opposed to something running in the background on its own schedule, with its own agenda whether the user is running the application or not.
[Edit: yes, that is kind of the idea. Native version has better features because it can access them.]
I use it at work. On Windows it also launches a new console window every it executes (?) a command. You should definitely disabled this using --no-daemon.
I don't know how and why they ruined this CLI with needless "features". It was perfectly usable two weeks ago.
I discovered this while benchmarking the tool I'm developing. I realized Codex was surprisingly fast after the initial run. Some other harnesses do similar things, so I decided to use bwrap around these tools for accurate benchmarking.
This is the same operational model as Claude Code.
ls ~/.local/share/claude/versions/
ps auxww | grep "claude bg"
I see a few prior versions of the Claude Code binary and several running background tasks, without any copy of Claude Code in the foreground. I regularly get a "Update available" message in my Claude Code UI too.
This kind of stuff often breaks for applications packaged with Nix, so it's often removed from Nix packages for compatibility reasons. Both behaviors you mention are helpfully patched out of the Claude Code package I use: https://github.com/numtide/llm-agents.nix/blob/dc285bfd94ec7...
As a result, I never knew that Claude Code does a nasty thing like this, either. :-\
Fwiw, packages in that repo get version bumped each day, so if that's a fast enough pace for you it may be a good alternative.
Refusal to adhere to the XDG specification is a big signal that the app is going to be a general bull in a china shop. It's not that hard to do OS- and distribution-appropriate package management.
A well-behaved application should not only use XDG on Linux, but it should use platform-native conventions on macOS as well: "~/Library/Application Support", etc.
Yeah, and for the contrary I spotted Claude app being ugly. When I turned off Claude, then put my computer to sleep and later woke it up, it made my fans spin loudly, and after I checked, there were like 2 dozen Claude processes and a lot of Python. What was it doing - I have no idea, but it was crazy. I'm setting up a VM with nothing but Claude and necessary tools. I feel Claude is not honest with everything it is doing.
Can you share some more detail of how you would implement this? Are you thinking something like VirtualBox (?), running a virtual machine, and you're using - for example, OpenCode terminal in the VM...?
We all know all the valuations are fake. Elon taught us this lesson well. Let’s not pretend they don’t have a gorillion users though, including every enterprise on earth.
https://finance.yahoo.com/sectors/healthcare/articles/godfat...
Progress is spiky and not guaranteed. The capital costs have also ballooned to the point where the rate has to level out
We don't see any indication that the LLM companies are slowing down and go for better code.
Instead it just looks like a cash grab until the AI bubble bursts.
I use nixos and use a little wrapper that wraps Codex CLI in Docker. Run codex-box /path/to/project and it starts or reuses a container with only that project bind-mounted, keeping login and history in a separate volume. It uses tmux for detaching and reattaching
https://gist.github.com/entropie/8f30b9e33e9412fda3adf2f7507...
You install an update. Oops, they added a sandbox that doesn't work inside a docker container, it's stopped working.
You install an update. Oops, you can't log in inside a docker container any more, it's stopped working.
You install an update. Oops, they started catching the copy+paste shortcut and replaced it with one that doesn't work.
You revert a broken update. Oops, the update changed the format of your login details, the old version can't read them, gotta log in again.
You install an update. Oops, can't scroll with the mouse wheel any more.
Not showstopping bugs (if you're used to persuading broken oauth logins to work) but it certainly makes updating an "ugh, what have they broken this time?" experience.
The problem is hardly anyone gives a shit if they can pump out stuff that makes them look like they are good at stuff.
Vanity before engineering.
Legit bugs still open 6 months on.. It's a bit ironic
https://github.com/openai/codex/issues/52251#issuecomment-60...
People moan about how badly made Claude code is but Codex seems much worse on every metric
It's like fighting fire by employing an arsonist.
...are you sure it wouldn't consider your personal files garbage?
Anyone experiences with it?
it's minimal and somewhat opinionated, not integrated well with memory subsystems (although that's kind of the point)
my only real criticism is that last I played with it it deals with slow provider responses very poorly , so lots of locally hosted llms require a lot of tuning to get it to handle streaming/non-streaming/timers well.
way less minimal (but still pretty minimal) I prefer Pi; but different markets I think.
hax fit my openbsd laptop well, espescially since Pi and related derivatives are kind of un-spokenly incompatible with the BSDs.
https://github.com/numtide/llm-agents.nix/
Also, it feels that chatgpt.com and aistudio.google.com are the buggiest websites I have seen for long time. Why it is always something related to LLMs?
If I need special actions, I just make them CLI tools on my Mac and tell the AI model that such a tool exists(via a single line in its system prompt) and that’s it. It’s been very nice.
[1] - https://usehax.dev/
No affiliation whatsoever. Just a happy user.
I've gotta give hax a go, as a pi user who came on-board due to the no frills direction pi takes. I say takes rather than took, because regardless of the (heated) discussion surrounding mcp and code mode support, I cannot fight the numbers- augmenting the harness provided tools is a notable boost to execution round trip performance.
--
1: https://github.com/openai/codex/blob/0c329ad2c9d5a2592128a1d...
The recent terminal UI changes were more fastidious, but even those are easily disabled with CODEX_TUI_DISABLE_KEYBOARD_ENHANCEMENT=1 and the /tui command.
A human seems less likely to embark upon a path that most humans know leads to something a human is unlikely to want
LLMs produce amazing things, but only one token at a time. Anything beyond that is retrospectively emergent at best
Maybe this kind of stuff is standard for some people but I absolutely loathe auto updating software for many reasons
I travel for work, and often have a tech demo, solution, tool, or something to show - or am running a workshop.
I usually do NOT update things right before my trip, as I want to minimize the potential for disruption.
Jokes aside I do the same I tend not to upgrade my Mac, my phone unless I really have to.
The “update” button on the top of my brave install has now turned from green to orange and I’ll likely wait until it goes red
What? Pinning a browser?
First thing it'll want to do after starting is update itself to newest version anyway.
Firefox does not only not prompt me for updates (it only updates if I manually run an update), even if I do update with Firefox running, it does not stop me from continuing to use Firefox without restarting it. All I did was install Firefox from flathub (packaged by Mozilla (a default configuration provided by Mozilla)).
Auto-update systems have value but are not a suitable default for professional tools.
That's not odd whatsoever. /s
We considered the daemon thing, but ended up with a hard-update check on every boot. It's thus a user choice: if you don't like the automatic updating, you can just not run the application anymore.
And even with all that, it's hard to stay on the right side of AV algorithms, etc.
We wrote about it here: https://tritium.legal/blog/update
That's not a choice. It s just what most apps do these days, but it's the first time I've seen anyone stating this so directly, or trying to frame it as a choice.
(And behaviorally, this makes your product just a SaaS website with extra steps.)
[Edit: yes, that is kind of the idea. Native version has better features because it can access them.]
I don't know how and why they ruined this CLI with needless "features". It was perfectly usable two weeks ago.
As a result, I never knew that Claude Code does a nasty thing like this, either. :-\
Fwiw, packages in that repo get version bumped each day, so if that's a fast enough pace for you it may be a good alternative.
TLDR: They don't care.
~/.codex/config.toml
hell, I also have this redundancy configured in bash: