WEBVTT 1 00:00:00.000 --> 00:00:10.004 My review queue has never been this full of code nobody wrote, and this year an OpenAI model overturned a conjecture Paul Erdős made in 1946. Nine mathematicians published a human-verified version. 2 00:00:10.004 --> 00:00:17.195 And in October the same company posted seven hundred and twenty-two machine-written papers, then pulled three the next day over a sign error. 3 00:00:17.195 --> 00:00:23.448 Same company, two very different results. The difference is the referee, and I'm about to bet my roadmap on it. 4 00:00:23.448 --> 00:00:26.887 And I'm about to tell you why that's a bad bet. 5 00:00:26.887 --> 00:00:35.641 Before we start, my week. Digiday ran a piece on agencies finding that AI overuse is leading to what they call brief bloat. Briefs getting longer, not better. 6 00:00:35.641 --> 00:00:37.517 Longer, not better. For a brief. 7 00:00:37.517 --> 00:00:50.022 And I send briefs to two agencies, so I read that and thought, am I the problem? I went back through last month's, and one had a section called "strategic north star" that I did not write and cannot explain. 8 00:00:50.022 --> 00:00:54.086 Nobody can explain a north star. That's what makes it a north star. 9 00:00:54.086 --> 00:01:00.652 Who reads page forty? Who checks it? Nothing fails when a brief is bad. It just quietly costs us a month. 10 00:01:00.652 --> 00:01:05.029 Which is the whole episode, and you handed it to me before the welcome. 11 00:01:05.029 --> 00:01:22.224 Welcome to Dan's AI Intel, where we try to get past the hype and the fear, and ask what the AI revolution means for the economy, for politics and for the race between the labs. I'm Alex, I run engineering at an activewear brand, and nobody has ever explained a north star to me either. 12 00:01:22.224 --> 00:01:31.603 I'm Sam, I run brand and lifecycle marketing at an online pet retailer, and I've now been told I wrote my own north star. Follow the show wherever you listen. 13 00:01:31.603 --> 00:01:50.674 Today's question is why the strangest AI results of the last few years keep landing in maths. On the map: which fields AI has really transformed, the climb from guessing formulas to overturning a famous conjecture, the moment the bottleneck flipped to checking, why a referee turns guessing into learning, and how much of the maths lead is just a scoreboard. 14 00:01:50.674 --> 00:02:06.306 My honest starting position is that a lot of this is scoreboard. I've been burned by a vendor deck that only reported the wins, and maths is the one field where you can count wins on a public list. Wake me when it ships in something my team can buy. 15 00:02:06.306 --> 00:02:16.936 Mine is the opposite. I think the lead is real, the reason is a referee, and once you see the reason you can predict which fields fall next. I'm planning my roadmap around it. 16 00:02:16.936 --> 00:02:19.437 Your roadmap, built on a theory about Erdős. 17 00:02:19.437 --> 00:02:21.000 On a theory about checkers. 18 00:02:21.000 --> 00:02:25.240 Then start with my objection. Whose checker, and who checked the checker? 19 00:02:25.240 --> 00:02:30.540 Start with the title. "The secret is the referee." Slogan, or does it mean something? 20 00:02:30.540 --> 00:02:50.327 It means something, and it's personal for me. Every pitch I sit through says AI will speed up science, and maths is the first place that visibly happened. If the cause is verifiability, I can tell my board which of our jobs are next and which wait, and that's a budget conversation, not a science one. 21 00:02:50.327 --> 00:02:55.980 But hold on. Isn't the biggest AI science result a protein thing? Not maths at all. 22 00:02:55.980 --> 00:03:06.580 AlphaFold. Demis Hassabis and John Jumper shared half the Nobel Prize in Chemistry for it. So maths leads on volume, but AlphaFold is the single most consequential result. Both true. 23 00:03:06.580 --> 00:03:11.527 So why does one field get a Nobel and the other gets a list? 24 00:03:11.527 --> 00:03:23.893 AlphaFold was graded in a blind contest against structures labs had solved but not yet published, and it scored above ninety, which is competitive with experiment. A prediction could be checked against a known answer. 25 00:03:23.893 --> 00:03:24.600 A referee. 26 00:03:24.600 --> 00:03:40.853 A referee, plus decades of banked answers to learn from. Weather is the same. Google DeepMind's GraphCast beat the leading conventional forecast on most test targets and makes a full forecast in under a minute on one machine. And the atmosphere grades every forecast within days. 27 00:03:40.853 --> 00:03:44.033 Weather's the easy one. Show me where it breaks. 28 00:03:44.033 --> 00:03:47.567 Materials. DeepMind's GNoME predicted hundreds of thousands of stable crystals. 29 00:03:47.567 --> 00:03:58.167 I did see that one. And then two materials scientists, Anthony Cheetham and Ram Seshadri, looked at a sample and found nothing strikingly new and no use for the list. 30 00:03:58.167 --> 00:04:09.120 DeepMind stands by it and says hundreds have since been made in labs. But a material has to be made before anyone knows. The checking runs at the speed of chemistry. 31 00:04:09.120 --> 00:04:15.127 This is where I get impatient. A list of predictions sounds like a demo that never shipped. 32 00:04:15.127 --> 00:04:26.433 Then medicine will make you furious. No AI-discovered drug has been approved. Insilico's rentosertib, for lung fibrosis, entered a late-stage trial in July, and the company's own timeline says approval around 2030. 33 00:04:26.433 --> 00:04:34.913 That's a brand campaign whose verdict lands two quarters later, and I've been the person waiting on it. Days to design, years to know. 34 00:04:34.913 --> 00:04:45.867 And no amount of computing shortens the trial. Which is why I'd never let an agent near our returns forecasting. Our checker is next season, and I can't retry a season. 35 00:04:45.867 --> 00:04:56.113 Which gives me a better question for the next time a vendor tells me AI will fix everything. I ask what the checker is and how long it takes. 36 00:04:56.113 --> 00:05:00.000 That's a better question than most of the slides I get. 37 00:05:00.000 --> 00:05:04.545 But AlphaFold's checks took a lab months. That breaks your own rule. 38 00:05:04.545 --> 00:05:16.667 It does, and I should have caught it. The answers had been banked for decades before it arrived. So the rule isn't just fast checks. It's abundant, trustworthy answers to learn against. 39 00:05:16.667 --> 00:05:21.591 Fine. I'll take the shape. I'm not taking that it's the whole story. 40 00:05:21.591 --> 00:05:37.879 The maths record is a ladder, and each rung deserves its own grade. Bottom rung, the Ramanujan Machine, from the Technion in Israel with Google, published in Nature. It matched numbers to guess new formulas for constants like pi and e. No proofs. 41 00:05:37.879 --> 00:05:39.015 So it guessed. 42 00:05:39.015 --> 00:05:44.318 Some guesses were later proved, some were easy in hindsight. Modest result, real idea. 43 00:05:44.318 --> 00:05:54.167 Participation trophy. In my world that's the vendor with a clever demo and no way to measure whether it helped. We hand those out every quarter. 44 00:05:54.167 --> 00:06:05.152 Next rung, AlphaTensor. DeepMind treated matrix multiplication as a game and found a way to multiply small binary matrices in fewer steps than the classic method from decades back. 45 00:06:05.152 --> 00:06:07.424 Why do I care about matrices? 46 00:06:07.424 --> 00:06:21.818 They sit underneath almost everything, including the computers that train AI. But it was narrow. The trick only works for binary arithmetic, and within days two mathematicians in Linz beat one of its records with ordinary computer search. 47 00:06:21.818 --> 00:06:28.636 So a machine got outrun by ordinary search in a week. Very on brand for my budget meetings. 48 00:06:28.636 --> 00:06:33.939 The next one is my favourite, and you'll see why in about ten seconds. 49 00:06:33.939 --> 00:06:37.348 I can already see it. Your shoulders just moved. 50 00:06:37.348 --> 00:06:53.258 FunSearch. A language model writes programs, a scorer runs them, the best survive. It found a set of points in eight dimensions where no three line up, and DeepMind called it the biggest jump in that kind of set in twenty years. 51 00:06:53.258 --> 00:06:55.152 There it is. The face. 52 00:06:55.152 --> 00:06:56.288 There's no face. 53 00:06:56.288 --> 00:07:00.455 There's a face. You get faster and your hands start moving. 54 00:07:00.455 --> 00:07:11.439 Fine, the face is because it's elegant. Nobody proved a theorem. The scorer just wouldn't let a bad program through, and that scorer, in DeepMind's words, guards against hallucinations. 55 00:07:11.439 --> 00:07:14.470 A nervous intern with a very strict manager. 56 00:07:14.470 --> 00:07:18.258 I'm using that with my board and not crediting you. 57 00:07:18.258 --> 00:07:30.000 And the strict manager is the part I'd pay for. I don't want a cleverer intern. I want somebody who reads every page and says no, quietly, before it reaches me. 58 00:07:30.000 --> 00:07:39.470 Except the manager has to say no in seconds. A manager who takes a week is just a slower intern, and I've hired that manager. 59 00:07:39.470 --> 00:07:44.015 So the speed of the no matters as much as the no. 60 00:07:44.015 --> 00:07:47.803 That's the whole thesis, yes. I'll stop saying it now. 61 00:07:47.803 --> 00:07:55.000 Where do the contests come in? I keep hearing gold medal and I can't tell if it's a stunt. 62 00:07:55.000 --> 00:08:08.071 AlphaGeometry solved more than double the olympiad geometry problems the previous best had. And it trained on millions of theorems it generated itself, because a geometry engine could certify each one. No human examples. 63 00:08:08.071 --> 00:08:11.147 It made its own homework and marked it. 64 00:08:11.147 --> 00:08:20.758 Then AlphaProof and a second AlphaGeometry reached the top of silver at the Olympiad, and AlphaProof solved the hardest problem, which only five humans cracked. 65 00:08:20.758 --> 00:08:23.450 Where's the catch? There's always a catch. 66 00:08:23.450 --> 00:08:36.905 Humans translated the problems into a proof language called Lean first, and some took days of computing. I've sat through that pitch from vendors: a landmark, not something my engineers could pick up and use. 67 00:08:36.905 --> 00:08:40.366 A landmark that needs a translator is a demo. 68 00:08:40.366 --> 00:08:42.288 For about a year, yes. 69 00:08:42.288 --> 00:08:53.821 Let me tally the ladder so far. Guessed formulas, a record, a scorer, a contest. Every rung has somebody else doing the grading. Never the model marking its own paper. 70 00:08:53.821 --> 00:08:56.897 Hold that thought for later. It gets weirder. 71 00:08:56.897 --> 00:09:01.895 So a year ago that was a demo with a translator. What changed? 72 00:09:01.895 --> 00:09:24.578 Last year, AlphaEvolve. Think of matrix multiplication as a recipe where every step costs real money on a chip. It beat the classic recipe for complex numbers, the first improvement in fifty-six years. Across a big set of open problems it matched the best known answer about three times in four and beat it about one time in five. 73 00:09:24.578 --> 00:09:26.116 Matching is the baseline? 74 00:09:26.116 --> 00:09:32.267 Matching is my baseline vendor. Beating it one time in five gets a pilot from me. 75 00:09:32.267 --> 00:09:38.418 My CMO forwarded me the gold medal headline with three exclamation marks. Is that a stunt? 76 00:09:38.418 --> 00:09:50.336 Later that summer, an advanced Gemini Deep Think scored gold, officially confirmed, and wrote in plain English inside the time limit. An OpenAI model matched the score, outside the official contest. 77 00:09:50.336 --> 00:09:55.334 Plain English, not Lean. I'll file that, because I think it matters later. 78 00:09:55.334 --> 00:09:56.872 It does. Hold it. 79 00:09:56.872 --> 00:10:15.326 Then the Erdős problems. I saw two people outside any lab solved one with Gemini. No, sorry, the OpenAI one. I mix the names up, it's practically my job. That's the part I care about, because every small shop I know assumes this stuff needs a lab's budget. 80 00:10:15.326 --> 00:10:39.931 Kevin Barreto and Liam Price, in January, with GPT-5.2 Pro. The proof was checked in Lean by Harmonic's system, Aristotle, and Terence Tao called it the first solved largely autonomously, in the spirit the problem intended. In May, DeepMind's AlphaProof Nexus, a language model looping against Lean, resolved a batch more. Days later an OpenAI model disproved Erdős's old conjecture about distances between points. 81 00:10:39.931 --> 00:10:43.006 The one from the cold open. Disproved how? 82 00:10:43.006 --> 00:10:58.000 Picture dots on a page, with a stick of exactly one length joining any two that are that far apart. Erdős guessed you can have only barely more than one stick per dot. The model built arrangements with more. 83 00:10:58.000 --> 00:10:59.752 And nine mathematicians checked it. 84 00:10:59.752 --> 00:11:09.915 Including the Fields medallist Timothy Gowers. They traced its key moves to algebraic number theory, a field far from the problem, and within weeks humans had improved the construction. 85 00:11:09.915 --> 00:11:13.068 So it disproved Erdős's bound. Not the whole problem. 86 00:11:13.068 --> 00:11:21.128 Not the whole problem. But a famous conjecture, overturned by a machine and confirmed by outsiders. That made me rewrite my roadmap slide. 87 00:11:21.128 --> 00:11:25.684 So what makes you less excited? You've been suspiciously happy this whole time. 88 00:11:25.684 --> 00:11:43.205 The checking. Lean is a language where a proof either compiles or fails. Tao says that means you don't need to trust the people you're working with. He calls AI plus formal verification water filters that filter out undrinkable water, and his summary of the models is unreliable but powerful. 89 00:11:43.205 --> 00:11:56.872 Unreliable but powerful. That's my entire vendor list. And water filters I understand. Mine are legal and brand safety, they slow everything down, and nobody loves them. But I'd take a slow filter over a pulled paper any day. 90 00:11:56.872 --> 00:12:05.983 Right, and that's the catch for me: it filters, it doesn't find. So somebody still has to produce the thing worth filtering, and the strain shows. 91 00:12:05.983 --> 00:12:15.094 Twenty-five Fields medallists signed a statement in September saying the labs' goals and mathematics' goals are severely misaligned. And then October, the papers that got pulled. 92 00:12:15.094 --> 00:12:36.470 Only some of them formalised in Lean. So the machines write proofs faster than people can read them. I know that from my side. Addy Osmani put it as the bottleneck moving from writing code to proving it works, and a study this year found most pull requests from AI agents, sixty-one percent, got no review before they merged or closed. 93 00:12:36.470 --> 00:12:37.872 No review at all. 94 00:12:37.872 --> 00:12:55.393 My engineers write less and read more now, and reading someone else's diff is harder than writing your own. If the referee is a person, the referee is the line that never shrinks. That keeps me up, because my review queue is the one thing I can't buy more of. 95 00:12:55.393 --> 00:13:08.009 Mine is the copy queue. My two directors can't read as fast as the tools write, and I've caught myself approving things I skimmed. That's worse than not reading, because now there's a signature on it. 96 00:13:08.009 --> 00:13:12.214 And I explain to the board why we shipped something nobody read. 97 00:13:12.214 --> 00:13:17.470 You love the tech and you're the one with the most to lose from it. 98 00:13:17.470 --> 00:13:20.274 That is a fair description of my job. 99 00:13:20.274 --> 00:13:24.128 Does it get worse with practice? People get tired of checking. 100 00:13:24.128 --> 00:13:42.000 Elena says exactly that, and she'd know. She's an emergency doctor and she proofreads her AI scribe's notes after every shift. A big UC Davis pilot found a small number of notes with errors that could do serious harm if nobody caught them, and the authors said clinician review stays imperative. 101 00:13:42.000 --> 00:13:45.546 Which is why you build it so tiredness doesn't matter. 102 00:13:45.546 --> 00:14:02.923 Robert Wachter's worry was that after a long run of accurate notes, the reviewer stops really looking. I know that one. My reviewers at six on a Friday are not the same readers as at ten on a Tuesday, and a proof checker is the same reader every hour. 103 00:14:02.923 --> 00:14:20.300 In English, please. Why does a checker make a model better, not just safer? Let me test it on my own desk first. If I had a thousand subject-line drafts a day and a button that told me instantly which ones were garbage, I'd be a genius by Friday. 104 00:14:20.300 --> 00:14:39.095 That's it exactly, you've just described the loop. A language model is fluent and unreliable, my junior engineer on a Friday afternoon. Alone, it hallucinates. Paired with a checker, every hallucination gets thrown away and every success becomes training data. AlphaProof Nexus attempted three hundred and fifty-three open Erdős problems and proved nine. 105 00:14:39.095 --> 00:14:41.222 Nine. That's a miserable batting average. 106 00:14:41.222 --> 00:14:55.053 And the failures cost nothing but computing time, because Lean rejected every wrong proof before a human had to read it. My engineers' wrong guesses cost me a review slot each. That's the difference I keep losing sleep over. 107 00:14:55.053 --> 00:14:57.890 Okay, but who pays for that computing time? 108 00:14:57.890 --> 00:15:04.628 The lab. Which is the point. They can afford a million bad guesses because nobody has to read them. 109 00:15:04.628 --> 00:15:15.267 That I can follow. A bad guess that's free is a different business from a bad send that lands in twenty million inboxes and gets my name on the complaint. 110 00:15:15.267 --> 00:15:34.416 Every system we've named runs that loop. AlphaEvolve verifies, runs and scores proposed programs automatically, and DeepMind says that's why it works best where progress can be measured. AlphaGeometry manufactured its own practice theorems. DeepSeek-R1, the Chinese reasoning model, was trained with a reward that simply checked its final answer against the correct one. 111 00:15:34.416 --> 00:15:39.381 And there's a name for this. I swear I saw it on a slide. 112 00:15:39.381 --> 00:15:53.211 Jason Wei, the OpenAI researcher, called it verifier's law. The ease of training AI to solve a task is proportional to how verifiable the task is. Andrej Karpathy put it as, Software 2.0 easily automates what you can verify. 113 00:15:53.211 --> 00:15:56.048 So where does my desk sit on that? 114 00:15:56.048 --> 00:15:57.112 You tell me. 115 00:15:57.112 --> 00:16:08.815 Both ways. My lifecycle team, the autoship emails, we have a referee. People open, click, unsubscribe, and we see it within days. My brand team writing the voice of the company has none. 116 00:16:08.815 --> 00:16:13.780 Which means by your own logic the first one gets automated before the second. 117 00:16:13.780 --> 00:16:23.000 I hate that you just did that. But yes. And it's the other half of the brief problem, because nobody can check a strategic north star. 118 00:16:23.000 --> 00:16:31.781 There's a number behind that. A big survey of marketers found most of them use AI to produce content and well under half say it improved performance. 119 00:16:31.781 --> 00:16:45.115 Faster, not better. That's my quarterly review in one sentence. My brand strategy is the same, isn't it? Experts disagree what good looks like, the verdict arrives years later, and I'm the one who has to defend it in the meantime. 120 00:16:45.115 --> 00:16:53.246 And that's the part I'd hate to be in your chair for. You'd be defending a call that no loop can grade for two quarters. 121 00:16:53.246 --> 00:17:00.726 My daughter Maya has been living this. She quizzes herself with ChatGPT every day, and she's started screenshotting the times it's confidently wrong. 122 00:17:00.726 --> 00:17:03.002 What does she do with the screenshots? 123 00:17:03.002 --> 00:17:12.434 Sends them to me, with a skull emoji. But she knows a wrong answer that sounds specific and logical is the worst kind, because only a referee catches it. 124 00:17:12.434 --> 00:17:16.987 She's sixteen and already working out that the checker matters more than the confidence. 125 00:17:16.987 --> 00:17:31.947 Here's the weirder part I promised. Once a model can generate its own problems and grade its own answers, its training data is limited by computing power, not by human experts. Code is the second domain like that, because a program can be run against tests. 126 00:17:31.947 --> 00:17:38.126 So the model marks its own paper after all. That's the bit that makes me want an outside examiner. 127 00:17:38.126 --> 00:17:51.785 In maths the examiner is Lean, and it doesn't care how confident the model sounds. In code it's the test suite, and that's why I plan around it. Where we have tests, I let the agents run. Where we don't, I don't. 128 00:17:51.785 --> 00:17:56.338 And that I can take to finance. A rule for where agents are allowed. 129 00:17:56.338 --> 00:17:58.940 Boring is good. Boring means it's a rule. 130 00:17:58.940 --> 00:18:08.372 But here's my worry for the people under me. If the cheap-to-check work goes first, that's the work I hand my juniors. Subject-line variants, send copy, the first rung. 131 00:18:08.372 --> 00:18:13.250 Same fight I have with my engineers. The junior rung is the one with tests. 132 00:18:13.250 --> 00:18:17.803 So I have to build them a different first rung, and nobody sells that. 133 00:18:17.803 --> 00:18:27.235 Nobody sells it because I can't write a test for the review I give a junior engineer on a Friday. It's the part of my job no suite grades. 134 00:18:27.235 --> 00:18:30.812 Put that on a slide for my CFO. She'd frame it. 135 00:18:30.812 --> 00:18:50.000 Now my turn, and I'm going to argue this the way I argue a launch deck. Maths wins are cheap to produce, checkable in days, and sit on a public list. A biology win is none of those. So when I see a flood of maths announcements, I count it like a sale email that only reports the opens. 136 00:18:50.000 --> 00:18:58.925 And I'd say the check is the whole point. My agents' tests passing is not a scoreboard. It's a real referee, and it's why I let them run. 137 00:18:58.925 --> 00:19:14.544 Your tests cover what you thought to test. A list of solved problems covers what someone bothered to pose. Tao says AI is clearing the attention-starved long tail, problems posed once and never followed up. One claimed solution turned out to be in a 1977 paper by Erdős himself. 138 00:19:14.544 --> 00:19:17.413 The machine solved something the author solved decades ago. 139 00:19:17.413 --> 00:19:27.931 Princeton's Noga Alon has stopped working on Erdős problems. His line was, once AI started to solve them, there is no point anymore. Picture my junior copywriters hearing that about the easy briefs. 140 00:19:27.931 --> 00:19:37.175 That one lands on the headline, not on the method. My review queue is full of easy pickings too, and the tests still tell me which ones are real. 141 00:19:37.175 --> 00:19:43.550 Easy picking that passes the test is still a pick. I'll give you that. I won't give you the headline. 142 00:19:43.550 --> 00:19:49.606 Next, reporting bias. Labs publish wins, never misses. I know that move, I've sent it in a launch email. 143 00:19:49.606 --> 00:20:08.094 And that's where I have to give ground. Tao put the real success rate on Erdős problems in the low single digits in January, and not much higher later. The cleanest test is First Proof, where mathematicians set unpublished research problems the models can't have seen. In June, the best of four public systems solved six of ten. 144 00:20:08.094 --> 00:20:13.831 Six of ten is great, and nowhere near what the announcements made me tell my CMO last month. 145 00:20:13.831 --> 00:20:18.294 I came in counting the wins. I'd have hated that in a vendor deck. 146 00:20:18.294 --> 00:20:34.231 You told me to hold the English thing, so here it is. The gold medals were written in plain English, not Lean. The voice of the company has no Lean either, and if the models got good at fuzzy work anyway, your referee theory is a theory about the past. 147 00:20:34.231 --> 00:20:47.938 That's the one that could make me wrong in front of my board. OpenAI did say its model used general techniques for hard-to-verify tasks. The referee still mattered in training, but if general reinforcement learning keeps improving on fuzzy tasks, maths' advantage shrinks. 148 00:20:47.938 --> 00:20:51.763 So what you're betting your roadmap on might not be what wins. 149 00:20:51.763 --> 00:20:59.094 I could be wrong, and I've been wrong before. In 2015 I ran a startup whose entire referee was me saying "seems fine." 150 00:20:59.094 --> 00:21:00.050 Every founder's referee. 151 00:21:00.050 --> 00:21:04.513 It failed spectacularly. That's why I trust a real checker over my own confidence. 152 00:21:04.513 --> 00:21:06.744 And the other half of the story? 153 00:21:06.744 --> 00:21:23.000 Maths needs no lab, so a failed attempt costs seconds, and decades of written maths gave the models plenty to learn from. I feel that one in my own shop: my agents retry for pennies, and that matters to me as much as the tests do. So verifiability alone isn't it. 154 00:21:23.000 --> 00:21:39.702 Here's my grade, from my side of the P&L. The referee idea is the best explanation I've heard, and it tells me which of my teams has a checker and which doesn't. But the headline volume is part substance and part scoreboard, and I'm not paying for the scoreboard half. 155 00:21:39.702 --> 00:21:48.053 Substance plus cheap retries plus banked answers. I'd say that grade is fair, and I'd say it's mostly substance. You'd say it's closer to half. 156 00:21:48.053 --> 00:21:48.721 I would. 157 00:21:48.721 --> 00:22:07.094 So where do I land. I came in saying scoreboard. I'm leaving saying half. What moved me is that failure is free when the referee is automatic. Nothing on my P&L works like that, and it tells me which of my teams gets squeezed first. I'll take the thesis and stay annoyed about the scoreboard. 158 00:22:07.094 --> 00:22:23.128 I came in saying the referee explains it. What moved me is the success rate. Low single digits, not much higher later, and a bit over half on problems nobody had seen. I'd been counting wins. That's benchmark theatre, I hate benchmark theatre, and I was doing it. 159 00:22:23.128 --> 00:22:25.132 I'd like that on the record. 160 00:22:25.132 --> 00:22:31.145 You get one. Where I won't move: the order of fields is predictable by who has a checker. 161 00:22:31.145 --> 00:22:37.157 And I still want proof it ships. So the honest split is how much of this is substance. 162 00:22:37.157 --> 00:22:46.177 On Monday I'm listing what my team has a referee for and what it doesn't. Tests, yes. Anything that ends in a person reading a diff, no. 163 00:22:46.177 --> 00:23:04.549 Mine's the same list. Send performance, I'll let agents near it. Brand voice, nobody touches it until someone shows me a way to check it. And when finance asks next quarter what the AI line bought us, my answer is that we pay for what has a referee and we stop paying for what doesn't. 164 00:23:04.549 --> 00:23:06.219 That's a sentence I'd sign. 165 00:23:06.219 --> 00:23:09.226 Then sign it before the holidays, when nobody's reading. 166 00:23:09.226 --> 00:23:21.251 One thing we haven't said. AlphaEvolve's scheduling rule already recovers a fraction of one percent of Google's worldwide computing. That's boring, and it's the kind of work that gets automated first, because the check is cheap. 167 00:23:21.251 --> 00:23:32.274 And the fight that actually worries me is the Fields medallists. They weren't mostly complaining about wrong answers. They were complaining about labs announcing results faster than anyone can check or cite them. 168 00:23:32.274 --> 00:23:41.962 Which means whoever controls verification decides who gets the credit. And that's my vendor problem in a lab coat. If one company owns the checker, it owns the scoreboard. 169 00:23:41.962 --> 00:23:48.643 In my world that's the sending platform. Whoever owns the open rates owns the story of whether my campaign worked. 170 00:23:48.643 --> 00:24:00.000 So I'll be watching who builds referees for biology, materials and medicine, because whoever does will decide where the next big results land, and I'd rather be buying from them than competing with them. 171 00:24:00.000 --> 00:24:14.596 We're an AI-voiced show, and we want to be clear about that. Dan builds a custom stack of AI tools to produce this show, and every claim in it is checked against sources, but the hosts and their lives are fiction, and nothing here is advice. 172 00:24:14.596 --> 00:24:28.875 Follow Dan's AI Intel on Spotify or Apple Podcasts so the next episode finds you. And if someone in your life keeps asking where AI is heading, sending them this one is the kindest thing you can do for both of them and the show. 173 00:24:28.875 --> 00:24:29.827 Thanks for listening. 174 00:24:29.827 --> 00:24:33.000 And I still want to know who checks page forty.