Sergey Levine - Building LLMs for the Physical World - [Invest Like the Best, EP.465]
Sergey Levine - Building LLMs for the Physical World - [Invest Like the Best, EP.465]
Podcast1 hr 6 min
Listen to Episode
Note: AI-generated summary based on third-party content. Not financial advice. Read more.
Quick Insights

Investors should prioritize Vertical AI platforms like Rogo AI and Vanta, which provide specialized automation for high-value sectors like finance and cybersecurity. Focus on "pick and shovel" infrastructure providers such as WorkOS, which powers the enterprise capabilities for industry leaders like OpenAI and Anthropic. In the robotics sector, the highest value is shifting from hardware to "foundation models" and general-purpose intelligence, making software-agnostic firms like Physical Intelligence key players to watch. Look for B2B SaaS companies like Ramp that offer clear ROI through expense automation, as these are more resilient during economic downturns. Monitor the progress of Tesla and Boston Dynamics as hardware costs continue to deflate, but favor companies utilizing End-to-End Learning and Reinforcement Learning to solve complex physical tasks.

Detailed Analysis

Physical Intelligence (Private)

• Physical Intelligence is a robotics company focused on creating foundation models (the "brains") for any physical robot to perform any task in any environment. • The company's core thesis is that general-purpose intelligence is more effective than building specialized robots for narrow tasks (e.g., just washing dishes). • They utilize Vision-Language-Action (VLA) models, which are Large Language Models (LLMs) adapted for robotic control by training them on text, web images, and diverse robotic data. • Key Technical Approach: * Chain of Thought: Robots "think" through a task semantically before moving, allowing them to handle "long-tail" or unusual scenarios using common sense. * Data Scaling: The goal is to reach a level of usefulness where robots can enter the world and autonomously gather their own data, similar to the Tesla fleet model. * Hardware Agnostic: Their software is designed to work across various forms, from industrial arms to humanoids and even bulldozers.

Takeaways

Investment Theme: Look for the "Scarecrow Problem" solution. While hardware (the body) is becoming a commodity, the value accrues to the "intelligence" (the brain) that allows robots to generalize. • The "Bitter Lesson" of AI: Betting on systems that learn from raw data rather than those programmed with manual physics rules is the dominant trend in modern AI. • Dexterity vs. Logic: Real-world "common sense" (e.g., picking up a plastic bag) is currently harder for AI than complex math. Companies solving these mundane physical tasks are hitting the next frontier.


Robotics Sector & Industrial Automation

Moravec’s Paradox: The discussion highlights that what is easy for humans (walking, folding laundry) is hard for AI, while what is hard for humans (calculus, data analysis) is easy for AI. • Hardware Deflation: Robotics hardware has become significantly more affordable. Costs for robotic arms have dropped from $400,000 a decade ago to roughly $3,000 today. • Humanoids vs. Specialized Forms: While humanoids (like Tesla Optimus or Boston Dynamics Atlas) capture public imagination, the "optimal" robot for a task might be a swarm of drones or a specialized multi-armed machine.

Takeaways

Productivity Gains: Similar to how GitHub Copilot increased software engineer output, robotics will likely act as a "labor multiplier" rather than a total replacement in the near term. • Sector Opportunities: Early adoption is expected in "semi-structured" environments like hotel room cleaning, commercial kitchens, and hospital logistics before moving into the high-complexity home environment. • Risk Factor: The "Long Tail" of physical reality. The primary technical risk is the robot's inability to react safely to unpredictable human environments (e.g., children or pets).


Mentioned Companies & Platforms

Ramp

• A finance automation platform that uses AI to automate 85% of expense reviews with 99% accuracy. • Insight: Focus on B2B SaaS companies that save customers money (Ramp claims a 5% savings) as they are more resilient in various economic cycles.

Rogo AI

• A specialized AI platform designed specifically for Wall Street, investment bankers, and asset managers. • Insight: There is a growing trend toward "Vertical AI"—models built for specific high-value industries (finance, legal, medicine) rather than generic chatbots.

WorkOS

• Provides the "enterprise-ready" infrastructure (SSO, Audit Logs) for top AI companies like OpenAI, Anthropic, and Perplexity. • Insight: In a "gold rush," the "pick and shovel" providers (infrastructure APIs) often see more consistent growth than the individual application builders.

Vanta

• Automates compliance and security (SOC 2, ISO 27001) for high-growth tech companies. • Insight: As AI complexity grows, automated compliance becomes a mission-critical utility for any company handling sensitive data.


Key Investment Themes

End-to-End Learning: The most successful AI systems are moving away from human-coded rules toward "End-to-End" models that learn directly from observation and reinforcement. • Reinforcement Learning (RL): This is the key to "superhuman" performance. While LLMs mimic humans, RL allows a robot to practice a task (like plugging in a cable) millions of times to find a speed and efficiency humans cannot match. • Compositional Generalization: The ability for a model to take two things it has learned separately and combine them in a new way. This is the "holy grail" for general-purpose robotics.

Ask about this postAnswers are grounded in this post's content.
Episode Description
My guest today is Sergey Levine, a professor at UC Berkeley and co-founder of Physical Intelligence. The company is building robotic foundation models designed to control any embodied system to do any task in any environment. Sergey argues that solving robotics at full generality is the right path, and that building systems that learn across many robots, environments, and tasks may be the more scalable approach than building narrow specialists. We discuss how these models can perform new tasks without being trained on them directly, and why everyday human actions remain the hardest problems in the field. He also reflects on how human trust and acceptance may matter as much as technical breakthroughs in determining when robots become part of daily life. Please enjoy my conversation with Sergey Levine. For the full show notes, transcript, and links to mentioned content, check out the episode page ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠here⁠⁠⁠⁠⁠.  ----- Become a Colossus member to get our quarterly print magazine and private audio experience, including exclusive profiles and early access to select episodes. Subscribe at ⁠colossus.com/subscribe⁠. ----- ⁠Ramp’s⁠ mission is to help companies manage their spend in a way that reduces expenses and frees up time for teams to work on more valuable projects. Go to⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ ⁠ramp.com/invest⁠⁠ to sign up for free and get a $250 welcome bonus. ----- Trusted by thousands of businesses, ⁠Vanta⁠ continuously monitors your security posture and streamlines audits so you can win enterprise deals and build customer trust without the traditional overhead. Visit ⁠vanta.com/invest⁠.  ----- ⁠WorkOS⁠ is a developer platform that enables SaaS companies to quickly add enterprise features to their applications. Visit⁠⁠ ⁠WorkOS.com⁠⁠⁠ to transform your application into an enterprise-ready solution in minutes, not months. ----- Rogo is the AI platform for finance. They're building agents for Wall Street that are trained to understand how bankers and investors actually do work: from diligence and modeling, to turning analysis into deliverables. To learn more, visit rogo.ai/invest. ----- ⁠Ridgeline⁠ has built a complete, real-time, modern operating system for investment managers. It handles trading, portfolio management, compliance, customer reporting, and much more through an all-in-one real-time cloud platform. Visit⁠ ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ridgelineapps.com⁠. ----- Editing and post-production work for this episode was provided by The Podcast Consultant (⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://thepodcastconsultant.com⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠). Timestamps: (00:00:00) Welcome to Invest Like the Best (00:02:43) Intro: Sergey Levine (00:03:29) Why Bet on Generality Over Specialization (00:07:24) What if PI succeeds? (00:09:05) Pros and Cons of Humanoid Robotics (00:11:02) Timeline of Major Milestones in Robotics (00:15:47) Sergey's Personal Journey (00:18:22) Making General Intelligence Happen (00:19:57) Understanding Robot Data Collection (00:22:12) Most Surprising Discovery at Physical Intelligence (00:24:48) The Science of Common Sense (00:25:36) Long-Range Tasks in Robotics (00:27:24) Why Wouldn’t We Have A Robot in Our Kitchen by 2050 (00:31:21) Other Interesting Approaches (00:32:38) Cool vs. Useful in Robotics (00:36:48) Form Factor Innovation (00:38:22) Physical Intelligence Analogy (00:39:30) Economic Transformation from Robotics (00:40:48) Controversies in the Robotics Community (00:42:16) Arguments Against End-to-End Learning (00:42:34) Compositional Learning Explained (00:43:25) Last Tasks Robots will Conquer (00:44:30) Dark Parts of the Robotics Brain (00:47:05) What Makes a Great Researcher (00:50:15) Manufacturing and Scale Challenges (00:51:17) How Companies Should Prepare for Robotics (00:53:38) Boston Dynamics' Demos (00:55:43) Converging Technologies Enabling Robotics (00:56:47) How to Stay Up To Date in Robotics (00:59:51) Near Term Objectives (01:00:49) Confidence Level Among Researchers (01:03:31) Google's Experimentation Culture (01:04:24) The Kindest Thing
About Invest Like the Best with Patrick O'Shaughnessy
Invest Like the Best with Patrick O'Shaughnessy

Invest Like the Best with Patrick O'Shaughnessy

By Colossus | Investing & Business Podcasts

Conversations with the best investors and business leaders in the world. We explore their ideas, methods, and stories to help you better invest your time and money. Hear stock market and boardroom insights you can't find anywhere else. If you're a professional investor, CEO, entrepreneur, or business strategist, this is for you. Explore all our episodes and learn more at https://www.joincolossus.com