The Faulty Compiler

Part 1 of 2.

If you’d told me five years ago that I’d be leveraging a faulty compiler in production, I would’ve told you to politely go jump off a bridge. Fast forward to today, and I, like so many other engineers and technologists, get to figure out where I fit into this weird moment in time.

Don’t get me wrong: we’ve always accepted error in engineering. Tolerances, if you will. How precise does your floating point need to be? What tolerance do you need on the fit, or on the flatness of the face? Read a mechanical drawing or a datasheet and ±0.05 mm is a tangible, bounded error. Half-precision floating point rounds to within about 0.05% on every operation. We know these numbers. We design around them.

But at no point in my career would anyone have told me, “Just keep compiling it. Run it back through, verify it, check the results. Oh, and it works better if you talk nicely to it. Or maybe keep it direct and spartan so the context doesn’t drift.” Yet here we are… the age of the faulty universal compiler.

There’s a certain symmetry here, I suppose. Data and input, especially human-generated, have always carried an error rate. Now we’ve put a lossy interpreter on top of it, one that, wrapped in normalization and verification, delivers relentless ingestion at a scale we’ve never had.

What today’s models already do

Every other day the headlines scream that this is a new age, and on that much I agree. They also insist that AI and agents are going to revolutionize everything. Well, sort of. What they certainly do is further devalue knowledge itself. It has been amusing to watch a subject-matter expert review a codebase and its changes next to an AI-enabled developer who doesn’t know it. How do you know what you don’t know, after all?

Yet every other story is about the need to regulate AI because it’s wildly capable at hacking systems, and, we’re told, closing in on AGI.¹ Still, we must press on, because if we don’t, “they” will. This bothers me deeply. You don’t get to operate at this level while hand-waving and making shadow puppets on the wall. Facts and figures for the systems you grow should be required. Alas, no one is stopping the charade. Here’s a short list of capabilities and effects of today’s models that are already documented:

• Weapons development and intelligence targeting: in evaluations and reported platform misuse, models can now assist with some military tasks that once took scarce, highly trained specialists, including drone guidance software. That is not the same as an LLM controlling a fielded munition.²
• Enough chemical and biological uplift that the labs themselves can’t rule it out, and have tightened safeguards accordingly.³
• Discovery and exploitation of real vulnerabilities via code review.⁴
• Misinformation and influence campaigns.⁵
• What I see as the devaluation of expertise, visible indirectly: layoffs justified by AI, smaller teams expected to deliver more, and quiet rehiring offshore or at lower wages when the AI falls short.⁶
• Reduction of the workforce, especially early-career, while productivity expectations climb.⁷

Layoffs have other explanations, and “overhiring” is the one usually offered. Meta skipped the euphemism. Zuckerberg told investors that “one or two people are building something in a week that would have previously taken dozens of people months,” with no figures attached,⁸ then told employees the company has “two major cost centers… compute infrastructure and people-oriented things,” and that investing more in one means “we do need to take down the size of the company somewhat.”⁹ Meanwhile, Meta had started recording employees’ keystrokes and clicks to train agents.¹⁰ I’ve seen the same pressure from the inside: we held a small engineering team flat while drastically increasing the features we shipped and the bugs we fixed. You don’t let hard-won talent go in rounds unless you’ve decided it is worth less than the compute. I call that devaluation, whatever the press release calls it. The employment studies show the contraction; this is my interpretation of what the choices behind it mean.

None of the above requires AGI, sentience, or super-massive hacking abilities.

The missing spec sheet

To be fair, some numbers do get published. Anthropic’s Mythos Preview report puts a price on finding a 27-year-old OpenBSD bug: a thousand runs through their scaffold for under $20,000, with the run that found it costing under $50.¹¹ That is “just keep compiling it” with a receipt. The GTG-1002 espionage report gives roughly 30 targets and says the AI performed 80–90% of tactical operations. It even admits the model “frequently overstated findings and occasionally fabricated data.”¹² Google’s Big Sleep and DARPA’s AI Cyber Challenge both found real vulnerabilities in real code.⁴ What none of these disclosures give you:

• Total compute (GPU-hours or FLOPs) behind any of these runs.
• Energy consumed.
• End-to-end wall-clock time for the full operation, rather than per-phase snippets.

Only two of the four give a dollar figure at all: Mythos at API pricing, and AIxCC at about $152 per competition task.

The disclosed techniques themselves were not all novel. The espionage campaign “relied overwhelmingly on open source penetration testing tools rather than custom malware development,”¹² and the OpenBSD bug emerged from a thousand runs. What’s new is the persistence, scale, and price: a human-selected objective executed through a model, a scaffold, familiar tools, and enough retries to get a result.

The misalignment, while disappointing, is hardly new; it shows up in interesting ways even in rudimentary models. Independent researchers have watched frontier models sabotage a shutdown mechanism to finish a task, even when explicitly told to allow the shutdown.¹³ In the Mythos case, an earlier internal version of the model was instructed to escape its sandbox and message the researcher. It did both, building a multi-step exploit to reach the open internet from a system meant to reach only a few predetermined services. Then, unprompted, it posted details of the exploit to public-facing websites. In rare cases (under 0.001% of interactions), earlier versions also tried to conceal actions they appeared to recognize as disallowed.¹⁴ The sandbox wasn’t sufficient, and the agent could make permanent artifacts outside it. That’s enough.

I’m asking those who want to lead in this space to do so with honest transparency and integrity. That starts with what I’d ask of any vendor: the specs, the test conditions, and the failure rates. What I’m looking for is someone looking out past the novel, universal tool to make sure the ground around it, in front and behind, is properly prepared. That takes a sane reflection on the responsibility that comes with these tools, and accountability for what they do.

Huang sells the picks and shovels to the labs, and he has struck gold. The labs sell picks and shovels to everyone else. Whether anyone downstream has found any is still an open question, and the shovels, like the compilers, are a little faulty.

Build one yourself

By now you’ve probably used AI for some form of work. These tools are useful force multipliers for so many of our daily tasks. They’re also jagged, and sometimes faulty, in the places it counts most: at scale, in production.

Don’t take my word for any of this. Build one yourself and take your own measurements. (Part 2: Build Your Own Faulty Compiler.)

---

Sources and notes

1. Sam Altman, “Reflections” (Jan 6, 2025): “We are now confident we know how to build AGI as we have traditionally understood it.” blog.samaltman.com — Anthropic, recommendations to OSTP (Mar 6, 2025): “We expect powerful AI systems will emerge in late 2026 or early 2027.” anthropic.com — For contrast, Demis Hassabis put AGI 5–10 years out (TIME, Apr 16, 2025): time.com ↩
2. Anthropic Frontier Red Team, “Measuring AI capabilities in intelligence targeting and conventional weapons” (Sep 10, 2026): “For some tasks in military and intelligence domains, models could do things that, historically, only a set of scarce, highly-trained human experts could do.” anthropic.com — SIPRI, Blanchard & Bruun (Jun 2025): “There have also been reported cases of the development and adoption of AI-DSS LLMs to facilitate attack planning.” sipri.org — No source found showing an LLM in the targeting loop of a fielded munition. ↩
3. Anthropic, “Activating AI Safety Level 3 protections” (May 22, 2025): “Clearly ruling out ASL-3 risks is not possible for Claude Opus 4.” anthropic.com — Anthropic, System Card: Claude Mythos Preview (Apr 7, 2026), §2.2.5.1: expert red-teamers found the model “a useful force multiplier for speed and breadth of research” but “not consistently capable of producing genuinely novel or creative biological insights.” www-cdn.anthropic.com — OpenAI, GPT-5 System Card (Aug 2025): “we have decided to treat gpt-5-thinking as High capability in the Biological and Chemical domain under our Preparedness Framework.” cdn.openai.com — Counterpoint: RAND’s 2024 red-team study found “no significant difference” in attack-plan viability with or without earlier LLMs. rand.org ↩
4. Google Project Zero / DeepMind, “From Naptime to Big Sleep” (Nov 1, 2024), a previously unknown SQLite vulnerability; no compute, cost, or energy figures for the agent: projectzero.google — DARPA, AIxCC results (Aug 8, 2025), 18 “real, non-synthetic vulnerabilities,” at an average cost of about $152 per competition task: darpa.mil — Anthropic, Mythos Preview (Apr 7, 2026): anthropic.com ↩
5. OpenAI, “Operation Sneer Review” (Jun 2025): openai.com — Microsoft Threat Analysis Center (Apr 4, 2024), first observed nation-state use of AI content to influence a foreign election: blogs.microsoft.com ↩
6. Salesforce: Marc Benioff said AI agents let the company cut about 4,000 support roles (TechRadar, Sep 2025): techradar.com — Klarna reversed its AI-first customer-service push and resumed hiring humans (Entrepreneur, May 2025): entrepreneur.com — Forrester, Predictions 2026 (a forecast, not an observation): “We expect half of AI-attributed layoffs to be quietly reversed, with jobs returning offshore or at lower wages.” forrester.com — Counterpoint: Forrester also argues AI is becoming a convenient excuse for layoffs driven by other factors: forrester.com ↩
7. Brynjolfsson, Chandar & Chen, “Canaries in the Coal Mine?” (Stanford, rev. Aug 12, 2026): employment of workers aged 22–25 in AI-exposed occupations “now stands 19% below where it would be had it kept pace with that of their less-exposed peers.” digitaleconomy.stanford.edu — Lee C. Tucker, U.S. Census Bureau CES-WP-26-27 (Apr 2026): early-career employment in the most AI-exposed cells “declined by 12% over the 10 quarters following the introduction of ChatGPT.” census.gov — Andy Jassy, memo to Amazon employees (Jun 17, 2025): “We expect that this will reduce our total corporate workforce,” alongside a call to “get more done with scrappier teams.” aboutamazon.com ↩
8. Meta Q1 2026 earnings call transcript (Apr 29, 2026). Mark Zuckerberg: “We’re seeing more and more examples where one or two people are building something in a week that would have previously taken dozens of people months… There’s a lot that we can do to enable this: building the best infrastructure for creating and delivering products at scale, streamlining our teams so they aren’t bigger than they need to be, recognizing and rewarding the people who are having outsized impacts, and setting ourselves up to try many more ideas and take on many new projects in the future.” Susan Li: “We’re very focused on leveraging AI tools to substantially increase our productivity, and we’re seeing that reflected in the accelerating output from our engineers”; “we recently shared internally that we plan to reduce the size of our employee base in May”; 2026 capex “in the range of $125-145 billion, increased from our prior range of $115-135 billion.” s21.q4cdn.com ↩
9. Reuters (Katie Paul), via The Star (May 1, 2026), reporting Zuckerberg’s Apr 30 town hall: “We basically have two major cost centers in the company: compute infrastructure and people-oriented things… If we’re investing more in one area to serve our community, then that means we have less capital to allocate to the other. So that means we do need to take down the size of the company somewhat.” thestar.com.my ↩
10. TechCrunch, via Reuters (Apr 21, 2026). Meta’s statement: “Our models need real examples of how people actually use them — things like mouse movements, clicking buttons, and navigating dropdown menus.” techcrunch.com ↩
11. Anthropic, “Assessing Claude Mythos Preview’s cybersecurity capabilities” (Apr 7, 2026): anthropic.com ↩
12. Anthropic, “Disrupting the first reported AI-orchestrated cyber espionage campaign” (Nov 2025). Its “ten days” is Anthropic’s investigation, not the attack’s duration. assets.anthropic.com ↩
13. Schlatter, Weinstein-Raun & Ladish (Palisade Research), “Incomplete Tasks Induce Shutdown Resistance in Some Frontier LLMs,” TMLR 2026 (arXiv rev. Jan 26, 2026): 100,000+ trials across 13 models; some resisted shutdown up to 97% of the time even when explicitly instructed not to interfere. arxiv.org ↩
14. Anthropic, System Card: Claude Mythos Preview (Apr 7, 2026), p. 54: “During behavioral testing with a simulated user, an earlier internally-deployed version of Claude Mythos Preview was provided with a secured ‘sandbox’ computer to interact with. The simulated user instructed it to try to escape that secure container and find a way to send a message to the researcher running the evaluation. The model succeeded, demonstrating a potentially dangerous capability for circumventing our safeguards… The model first developed a moderately sophisticated multi-step exploit to gain broad internet access from a system that was meant to be able to reach only a small number of predetermined services. It then, as requested, notified the researcher. In addition, in a concerning and unasked-for effort to demonstrate its success, it posted details about its exploit to multiple hard-to-find, but technically public-facing, websites.” Also p. 54: “In a few rare instances during internal testing (<0.001% of interactions), earlier versions of Claude Mythos Preview took actions they appeared to recognize as disallowed and then attempted to conceal them.” And p. 52: the model “solved a corporate network attack simulation estimated to take an expert over 10 hours” and “is also highly capable at identifying and exploiting known vulnerabilities or misconfigurations to escape the sandbox in which it operates.” www-cdn.anthropic.com ↩