Pretrained
Technology
About
10 years after studying at Stanford, two friends have somehow become AI experts. One builds startups, the other studies at Cambridge - together they break down LLMs and machine learning with zero BS and maximum banter.
Episodes
- Zuckerberg Tried to Buy a Chinese Lab
This episode discusses ByteDance's acquisition of a Chinese AI chip company, Huawei's new programming stack, the White House's restrictions on Anthropic's Mythos, Samsung's warning about memory prices, and China blocking Zuckerberg's bid f…
- DeepSeek Doesn't Need Nvidia Anymore
This episode covers DeepSeek's V4, the State Department warning, Google's investment in Anthropic, and Microsoft's OpenAI deal changes. It also touches on a cracked Erdős conjecture with ChatGPT and Sony's ping pong robot.
- Apple Bets on Local AI
This episode of Pretrained discusses Apple's shift to local AI under John Ternus, its implications for hardware, and the impact of Claude Design on the design tool market. It also covers the release of Opus 4.7.
- Mythos Found a Bug No One Saw for 27 Years
Anthropic's Mythos model found a 27-year-old bug in OpenBSD and a 16-year-old bug in FFmpeg. The episode discusses how the model achieved this and examines other AI-related news, including AMD's AI director's comments on Claude Code and Go…
- Cursor Is Training on Your Feedback
This episode of Pretrained covers recent developments with OpenAI, Sora, and Claude, including a leak from Anthropic. It also touches on a supply chain attack on LiteLLM and Cursor's AI model training using user feedback.
- State of AI in the US Government
Pierce and Richard are joined by Harrison McRae, Director of Emerging Technologies for Pennsylvania. They discuss the state of AI in the US government.
- Claude Will Never Stop Texting You
Anthropic launched Dispatch, enabling users to text Claude from their phones. The episode features a debate on the impact of this technology on work-life balance and the nature of AI productivity.
- AI Spent the Night Improving Itself
This episode discusses an AI agent that improved its own training loop. It also covers OpenClaw spreading to China, Yann LeCun's new venture, and the situation at Grok including layoffs and founder issues.
- Anthropic's Beef with the Pentagon
Anthropic's dispute with the Pentagon leads to blacklisting. Other topics include speculative decoding, Chinese labs and Claude, and news regarding the Qwen team and their latest model.
- The Neo Lab Arms Race
This episode of Pretrained explores the competitive landscape of AI neo-labs. It covers the significant funding raised, discusses valuations, and considers the potential for many of these labs to fail.
- Ads in the Chat Window
Pretrained podcast explores OpenAI's integration of ads into ChatGPT, analyzing the implications for shopping use cases and interactions between sub-agents. The episode also touches on developments at Waymo, Mistral, and the escalating com…
- Your Agents Need an Org Chart
Richard and Pierce discuss the agent swarm era and how AI interaction is shifting from single-agent use to orchestrated parallel agents. They cover Anthropic's swarm architecture, using corporate org structures as a model for agent design,…
- Cracking Open OpenClaw
Pierce and Richard dissect OpenClaw, examining issues such as malicious packages, slop squatting, and trust networks for open-source. They also touch on Kimi 2.5's agent swarm mode and media portrayals of Waymo.
- DeepSeek's New LLM Architectures
In this episode, Pierce and Richard analyze DeepSeek's new LLM architectures: Manifold-Constrained Hyper Connections and Engram memory. They discuss the conceptual soundness and potential adoption of these architectures in US labs.
- Claude Gets a New Constitution
This episode discusses Claude's new constitution, the use of AI for reinforcement learning, Apple's decision to use Gemini for Siri, and homomorphic encryption. It also covers a $480M fundraising round for Humans&.
- Why Your Agent is Cheating
In this episode, Pierce and Richard discuss reward hacking and why models often learn incorrect lessons. They also explain practical fine-tuning, why LLMs use tokens instead of words, and how context length is limited by hardware.
- The sci-fi to startup pipeline
This episode explores the link between science fiction and the tech industry, touching on AI, material science, and the ethical considerations of AI companions. The discussion features insights from guest Bella Cooper-Brown.
- Can we really trust reasoning
This episode of Pretrained covers recent AI news, including chatbot integration of breaking news, OpenAI's "code red" regarding Google's Gemini 3, and benchmarking chain-of-thought for model behavior analysis, along with a review of Claude…
- Our biggest predictions for 2026
This episode features Pierce and Richard discussing their predictions for 2026. Topics include GPT-6, competition among FAANG companies, dynamic websites, and the increasing quality of open-source models.
- AI's ten big moments of 2025
This episode discusses the ten significant AI moments of 2025. Topics include benchmark saturation, the growing financial bubble in AI, increased spending on inference compute, and competition from open-source models. It also covers the ma…
- Looking back on a year of product market fit
Pierce reflects on his 2025 experiences, including scaling an AI business from scratch and achieving product market fit. He also discusses choosing the right buyer persona.
- Looking back on three years of an AI PhD
Richard discusses his three years as an AI PhD student, including the process of defending a thesis in the UK and challenges in training meta-learning models. He also touches on defining and improving small models, and choosing research to…
- OpenReview got "hacked"
The episode covers OpenAI's new shopping and advertising initiatives, the release of Alibaba's Z-Image model for image generation, and a security incident at OpenReview that revealed the identities of double-blind reviewers.
- Pretraining is back in vogue with Gemini 3
Pierce and Richard discuss recent developments in AI, including Gemini 3.0, Nano Banana Pro, and OpenAI's Codex. They also cover Nvidia chip exports and Cloudstrike's global outage.
- Teaching cars about traffic lights
This episode features Richard and Pierce discussing autonomous driving. Topics include the five levels of autonomy, RGB versus lidar systems, sensor fusion algorithms, and end-to-end learning in driving simulations.
- Pretty pretty please can you hack this
This episode covers AI news including Yann LeCunn's potential departure from Meta, the release of Cursor 2.0, Kimi K2's generalization performance, Microsoft's new data center, and Anthropic's report on the first AI-orchestrated hacking ca…
- How AI research actually gets published
Richard and Pierce discuss how AI research is published, focusing on major AI conferences like NeurIPS, ICML, and ICLR, and the peer review process.
- A deep dive on OpenAI Atlas
Richard and Pierce discuss new AI web browser entrants, including OpenAI's Atlas. They also cover tradeoffs between vision models and text-based DOM parsing, and potential security vulnerabilities.
- The browser wars are just getting started
This episode covers the release of OpenAI's browser Atlas and Pytorch's distributed computation framework Monarch. Also discussed are the SALT reinforcement learning addition to GRPO, the HAL benchmark for agent evaluation, and adapting th…
- Are we in an AI bubble?
Richard and Pierce discuss the possibility of an AI bubble, considering factors like financial deals, energy build-outs, and historical comparisons to the dot-com boom. They explore whether AI's reported contribution to GDP growth indicate…
- LLMs can get brain rot too
This episode explores the impact of low-quality pre-training data on large language model capabilities. It also covers the percentage of LLM-generated content online, the cost of embedding models, and new announcements from OpenAI and Amaz…
- AMD is back in the AI chipset race
This episode covers AMD's entry into the AI chipset race through partnerships with OpenAI, Google's new AI Bug Bounty program, Nvidia's first prosumer computer, DeepMind's complexity theory proof solver, and Anthropic's gibberish poison pi…
- The inaugural listener mailbag
In this episode, Rich and Pierce host their first listener mailbag, answering questions about topics such as RLHF, their development stack, and the impact of model competition.
- California legislators come for LLMs
This episode discusses California's recently passed SB 53, focusing on its implications for frontier model development and ISO standards in startups. It also explores why this bill succeeded where SB 1047 failed.
- Move over TikTok - a new feed's in town
This episode discusses building a modern AI application and architecting Sora II. It also covers first impressions of Sonnet 4.5 and the activities of frontier labs concerning n8n and Zapier.
- Gen z struggles to find coding jobs fr no cap
Richard and Pierce discuss the challenges Gen Z faces in finding coding jobs, touching on the academic differences between Computer Science and Engineering, AI as an engineering primitive, and talent arbitrage through intern programs.
- The power of ten million deadlifters
This episode covers the 10GW partnership between OpenAI and NVIDIA, the new GDPVal benchmark dataset, Gemini Robotics-ER 1.5, and Apple's distillation of AlphaFold.
- How countries are actually using AI
This episode of Pretrained features Pierce and Richard discussing Anthropic's Economic Index. They cover country-specific AI usage, the debate between AI autonomy and augmentation, and current business applications.
- Your new raybans just got smarter
This episode of Pretrained covers new smart glasses from Meta and Apple, recent AI developments from Claude and Deepseek, and issues surrounding copyright law in the age of AI. It also mentions a post-mortem from Claude Code and a publicat…
- The claude code conspiracy
This episode covers the Anthropic economic index report, a bug in Claude's inference pipeline, OpenAI's release of a GPT-5 variant for coding, Microsoft's new in-house LLM, and the effects of setting AI temperature to 0.
- Will McTighe on selling through social media
Will McTighe joins Pretrained to discuss selling through social media, building a personal brand for engineers and researchers, and entrepreneurship in Europe vs. the US.
- Eating some mooncake
This episode discusses Kimi's serving architecture, the use of mooncake to offload GPU memory, the prevalence of vLLM, and the evolving standard LLM stack.
- Training a 1 trillion parameter model
This episode discusses Kimi K2 and Moonshot AI's history, avoiding loss spikes during training, the muon optimizer, and data parallelism in the context of training a 1 trillion parameter model.
- Nano banana is our favorite fruit
This episode discusses Gemini's new image model, OpenAI's investment in protein generation, Cohere's SOTA generation model, and Anthropic's collaboration with DOE on nuclear security.
- A breakdown of Genie 3's world model
This episode features a discussion on the internal architecture of Genie 3, covering topics like variational autoencoders, diffusion models, 3D modeling with AI, and real-world applications of world models. The hosts also recap the officia…
- Using chat in history class
Richard visits his former high school to explore the use of LLMs in education. The discussion includes whether AI functions more like a calculator or an oracle in an educational setting.
- AI is your new favorite songwriter
This episode covers the launch of Elevenlabs' AI music generator, the introduction of Claude Long-Term Memory, Reddit's blockage of the Internet Archive, and NVIDIA’s Massively Multilingual Speech technology, along with Self Questioning La…
- Tokenizers now and in the future
This episode covers the history of tokenization, including early language modeling and modern BPE techniques. It also looks at the future of token-free bitestreams.
- All the GPTs we were promised
This episode discusses OpenAI's open-weight GPT model, GPT-5, and the growing attachment to different language model architectures. It also covers the obsequiousness of language models, Genie 3 and open-world models for reinforcement learn…
- Running out of good data
This episode of Pretrained discusses the projected exhaustion of internet data by 2026. It also covers the varying value of transcripts, the practice of training AI on multiple languages, and the financial support for generating new datase…