Trust the work, verify the answer

A conjecture, a cheesecake, and a changed capability curve

Where were you when the Jacobian Conjecture got solved?

Me? I was sunk deep into my parents’ couch, eating a slice of strawberry cheesecake while my three young kids slept blissfully in another room.

I was thoroughly enjoying the completely unearned post-World Cup bragging rights that come from successfully hitching yourself to the winning team as a bandwagoner.

Spain had just beaten Argentina 1-0. I had contributed nothing to this outcome, but I was prepared to be insufferable about it anyway. And if you were to ask my brothers, who are actual real soccer fans and Lionel Messi stans, indeed I was.

In between bites and as I often do, because keeping up with AI is apparently now a full-time job layered on top of my actual full-time job, I checked X to see whether anything important had happened in the 15 minutes since I had last checked.

Scrolling, I saw a post from Levent Alpöge, a Harvard mathematician and Anthropic researcher.¹ It opened:

“hello there the jacobian conjecture is false thanx . . .”

Shocked, I tried to share the news and started choking on the cheesecake.

Literally.

For a few seconds, the Jacobian Conjecture became the second-most urgent technical problem in the room.

Alpöge would go on to thank one friend for suggesting the problem and another friend, “fable,” for working on it during the World Cup final. That second friend was Claude Fable 5.

If you’re a normal person and haven’t heard of the Jacobian Conjecture, the short version is that it said a certain kind of coordinate rule, one that can always be undone locally, must therefore be undoable globally. A conjecture that sat there, unresolved for 87 years. It was considered a problem, according to Smale in 1998, to be worthy of adding to his eighteen problems for the next century. It is not an obscure problem.

And since we’re all marketers here and love a good story, I would highly recommend you look up Yitang Zhang and his relation to this problem. I won’t spoil it. But, because I can’t help myself, let’s just say it ruined him into greatness.

The counterexample Alpöge posted was just 216 characters long. A character count short enough to fit inside the post announcing it, that appeared to knock down a conjecture that had been open since 1939.

Appreciate the insanity of the moment that such a thing happened the way that it did and was announced in such a casual manner.

Though to be honest, maybe this shouldn’t have been quite so shocking as it was. After all, just two months earlier, an internal OpenAI model disproved the Erdős unit distance conjecture, one of the most famous problems in discrete geometry, open for 80 years.²

Tim Gowers, a Fields Medalist and one of the mathematicians who reviewed the result, said that if a human had submitted the work to the Annals of Mathematics, he would have recommended accepting it without hesitation.²

Okay, so the obvious question that you probably have at this point: why am I telling you any of this?

Because like me, these models used to choke on strawberries.

The strawberry mental model expired

In the not-too-distant past — the models were an internet punchline for being unable to reliably count the number of r’s in the word “strawberry.” The failure got fixed in OpenAI's o1 generation of reasoning models (rumor has it the codename was Strawberry as a nod to the meme, though OpenAI never confirmed the joke), but the meme never died. And it gave a lot of people cover to stop paying attention.

Folks. We are not in that world anymore.

I keep meeting people whose mental model of AI is anchored to a bygone era. They tried ChatGPT or Claude once, it hallucinated something, and that one bad experience turned into a hall pass to dismiss the entire category and its exponential progress.

Meanwhile, the models are producing work capable of surviving intense scrutiny from some of the brightest people on earth.

Whatever you think about AI, whether you are AGI pilled or not, your picture of what it can do has a shelf life. Right now that shelf life is measured in months and it's shrinking.

Which brings me to the actual argument of this blog. It has two parts that look like they contradict one another. Bear with me.

Part one: I submit to you, dear reader, that you trust these models with far less work than the current record justifies.

Trust the models with more work

If you are still opening a chat box and merely chatting with the model, you’re testing an interface, not the model. It’s akin to buying a humanoid robot and using it like Alexa.

Like sure, the robot might have a good real-time voice model, but you’re leaving a lot of capabilities unused on the table.

Chat is no longer the ceiling. The work these models can do now involves your files, your tools, your data, multiple steps, your browser, the ability to run something and look at the results and try again until it's right. I’ve seen tasks these models take on go for over a hundred hours.

Do me, and yourself a favor, and look up the METR benchmark on time-horizon work.⁴ The length of task a model can complete has been doubling on a schedule.

This is a different category of activity than asking a question and reading a paragraph.

If you want a concrete next step, a verification loop if you will, ask your preferred model to explain the difference between chatting with it and working with Codex, Claude Code, ChatGPT Work, or Claude CoWork.

Then, if you’re up for it and want to be a hero at work, ask it to find you training videos on YouTube. You should know, and will know if you do what I said above, that things have shifted dramatically over the last seven months. The gap between what the models are capable of and what most people are asking them to do has never been wider.

Part two: even at the frontier, verification hasn’t disappeared.

Verification moved up a level

This might seem like a contradiction to part one but that would be a lazy claim and not defensible.

The lazy version of this read might say, “You see! Experts still check everything.”

They really don’t though. A single example among many, Boris Cherny, who created Claude Code, hasn’t written a line of code by hand in more than eight months and says Claude Code itself is now “100% written by Claude Code.”^5,6 He’s described his workflow now as managing armies of agents, with agents prompting other agents in trees that run into the thousands.⁷ No human reads that much output. He doesn't try. Nor should he.

One of his highest-leverage recommendations for working with these tools is to ensure the model has a way to verify its own work. A feedback loop. A method that improves the results by a factor of two or three according to him.⁸ So his verification got abstracted and encoded into carefully designed agents with incredibly thoughtful verification loops that reviewed other agents’ code.

The work didn’t disappear. It just got moved up a level.

Verification does not mean a human has to inspect literally every line the model produces. For obvious reasons. It means staying close enough to the work at a level where your judgment matters to be able to understand what happened so that you can improve what happens next.

This is happening everywhere at the frontier. The math results I shared earlier weren’t just believed. The Erdős result was released alongside a companion paper by nine mathematicians, Gowers among them.³ Their abstract describes their paper as “a short, digested, human-verified version” of the model's output. The Jacobian counterexample was accepted within a day because 216 characters is just insanely cheap to check.

Trust in these outputs arrived exactly as fast as verification allowed.

So yes, verification guards against models being wrong. But so what. Error-catching is the least interesting thing verification does.

Verification is for understanding, not just error-catching

Terence Tao, arguably the most famous mathematician alive, published a detailed walkthrough of the Jacobian counterexample the next day.⁹ More importantly, and something that went almost as viral as the math itself: he shared the raw ChatGPT session he used while working through the result.¹⁰ I highly recommend reading both. A full back-and-forth of a Fields Medalist probing someone else's proof with an AI as his sounding board is something to behold. It is inspiring, fascinating and an excellent way to feel catastrophically underqualified. I understood maybe 10 percent of it, and that is probably being overly generous. But notice two things about the walkthrough.

First, what he called it: a digestion. A digestion is a summary. So is a headline. The difference between these is not merely in the size of the output. It is actually more significant than that. A digestion is produced BY someone who worked through the underlying material and is accountable for what got dropped. A headline is produced FOR someone who will probably never read the underlying material, and is accountable for nothing.

Second, notice when he wrote it. By the time Tao published either, the counterexample had already been settled. Checked by hand, confirmed, done. His digestion caught no errors because there were none. It existed for a different reason. Understanding. His own, and much to the chagrin of Plato, everyone else who reads it. Perhaps the most famous verification artifact of the whole affair did zero guarding.

Ironically, this blog has already demonstrated this.

I opened by asking where you were when the Jacobian Conjecture “got solved.”

That is cleaner than what actually happened. The counterexample is a map in three dimensions, which makes the conjecture false in three dimensions and above. In two dimensions, the original case, it's still open.

Compression is lossy

If you walked away after the first sentence, you would leave believing something broader than the evidence supports.

AI did not do that to you. I did. Or rather, compression did.

Which brings me to my own work, where I ran into the same thing from the other direction.

At StudioNorth I built our Synthetic Audience Intelligence methodology. Our synthetic audiences are made of personas grounded in four layers of input data: demographic and firmographic, psychographic, behavioral, and decision criteria. Our methodology forces these personas to react to stimuli in isolation from one another and we record every one of these reactions. The result is an incredibly dense artifact.

Then we have a synthesis step that will take those artifacts — and compress them into a single output. A single set of findings.

The synthesis is very good. I mean, I built it.

But, synthesis is compression. And compression, by its very nature, is lossy.

So I read the individual reactions too. Partly because verification has become a habit at this point, and partly because, honestly, the reactions themselves are fascinating.

And I keep finding gold the synthesis has dropped. Not because it failed. Because it couldn’t have kept it.

If you’ve ever read the CoT — or the reasoning traces — of a reasoning model, I think you’ll find the above just obviously true. Sometimes the internal reasoning of the model holds far more profound and useful insight than the actual chat that you see as a response.

But here’s the cool thing. Here is where reviewing the work starts improving the system itself. If I repeatedly notice the same kind of valuable detail getting lost, I can name the pattern and build a check for it. The next version of the methodology can deliberately surface low-frequency objections, unexpectedly strong language or other details that deserve human attention.

Like not only is this improving the current output. It is teaching me, and my agent, how to improve the next run.

There’s a design decision here I want you to notice. If my pipeline simply threw the reactions away, if I simply took the output and didn’t verify the in betweens, then not only would I be missing gold, but I would’ve missed the opportunity to start building out that feedback loop.

If you build AI workflows, this is the cheapest trust decision you will ever make. Keep the intermediate work. Leave yourself not only a path back but a path forward.

So. Back to the couch.

Spain won, and I got to be insufferable about a victory I contributed nothing to. That's what being a fan is, and it's fine. Bandwagoning is a perfectly healthy relationship to have with a soccer team.

It's a terrible relationship to have with your own work.

The models can carry far more of that work than you're currently giving them. Give it to them. And when the answer comes back, don't just consume it like a final score. Read the work. That's where your judgment enters.

How does verification improve future AI runs?

Repeated review reveals patterns: objections that disappear, claims that need stronger sourcing, steps that fail, or signals that deserve more weight. Once named, those patterns become prompts, tests, routing rules, or review gates.