Anthropic's First PM: Opus 4.5, Rethinking Model Scaffolding & Safety as a Competitive Advantage

Opus 4.5 is a really impressive model.

It's had amazing benchmarks. A bunch of companies are getting great outcomes from it. And on supervised learning, I

from it. And on supervised learning, I got to sit down with Diane, the head of product for research at Enthropic to talk about all things Opus 45. We hit on what's newly enabled by the models, as well as where Diane thinks we are with computer use. We talked about how

computer use. We talked about how Anthropic does research and gets to models like this and how they use these models internally. We talked about

models internally. We talked about what's next for models and the evolution of scaffolding over the past years. And

we hit on how anthropic safety focus actually helps them on the product side.

This is just a lot of fun to get to go deep into the research process, model capabilities, and a bunch of other things. I think folks really enjoy it.

things. I think folks really enjoy it.

Without further ado, here's our conversation.

Thanks so much for uh for coming on the podcast.

>> Thank you for having me. Happy early

Thanksgiving.

>> Yeah, Thanksgiving Eve pod is like a real a real treat to get to do. I feel

like you guys dropped this uh this big model as a as a gift to us all right before uh the holidays. I'm sure a lot I'm sure a lot of founders will be saying they're grateful for it uh tomorrow at the at the Thanksgiving table.

>> Love it. That's that's the vibe we're going for.

>> You guys already had some really impressive models. What does it look

impressive models. What does it look like when you begin the work for something like Opus 45? You know, how do you think about the you know, what actually is required to improve these models and like what does that whole

process end up looking like?

>> I think like in some ways um we have a pretty I would say ambitious and like long range road map around model capabilities that we care about and that

we care about making improvements on. So

this includes things like better instruction following for users, coding advancements, making the models better at memory etc. And really in some ways

every gen generation of claude are like the vehicles by which these capabilities are expressed. And so like even when we

are expressed. And so like even when we are designing new versions of cloud, it's really around what are the advancements overall that we want to

like make sure can be delivered and um and how do we actually package it also position it, price it in a way that like resonates for the types of use cases

that users have today but also they might not even be aware that like AI can help them take on. And so you kind of pick a set of like problems or things that you want to improve the models on and then you know to what extent is it

kind of clear the the research directions to go down to improve that or are you having like a dozen different things and you're kind of figuring out early on which which ones help actually move the needle there.

>> I think it's both. I think we have like had a very um generally uh strong sense of what we think like this technology can do which is that like we can think

we think that it's generally can be extremely transformative across a wide variety of like use cases whether it's like engineering or others. Um I think

like other things like surprise us with like how users and also like builders discover them. So things like making

discover them. So things like making Claude really good at Excel and PowerPoint that was like a relatively small investment earlier on in the year and like what we found is that it really

resonated with financial services customers and so we're doubling down and making cloud really good at that type of like general like Excel work or like PowerPoint work that seems really beneficial. So it's both.

beneficial. So it's both.

>> What's the right way to conceptualize like doubling down? Is that like you know getting more data around a topic or doing more RL around that area? Yeah, I

think it does for like practical applications does start with like users and both users who might be coming to us with a use case but also when we think

of like something like computer use um why should people care like we kind of have to imagine almost a world that we want to go towards and so there's like almost an imagination phase um as a PM

the closest thing I could think of is like product vision docs or PRDS where you're trying to figure out like what is the so what why should somebody like come and use this like solution and then

also then translating that into like practically what are evaluations what are evals we call call them right like what are the evals we want to build to know the model's really good or not good at it maybe there's it's already half

the way there and like it needs to make certain improvements on data or RL for those like last 50% maybe it's um 20% we actually need to figure out like much

more significant changes we want to make And so starting with like sort of that envision in mind is actually very similar to traditional product management. Uh which I think might be

management. Uh which I think might be surprising as to hear.

>> It's really interesting I think especially as like the kind of classic evals feel like they're more and more divorced from like what actually the value is in the real world or the ways that people are using these things. You

have this combination of both customers coming to you with here and now problems that are probably really helpful. Evals,

you obviously have the best seat to what these models might be able to do in the future and really imagining what the what some of those use cases might be.

Yeah.

>> Were there were there any, you know, things that you were imagining that Opus 45 is the first model to be able to do that maybe maybe were in the vision doc like a year a year and a half ago and you're like maybe one day we could do X or uh you know, what were some of the

models capabilities that really surprised you?

>> Yeah, I think some of them have been like continuations of you know being able to do more complex agent coding tasks. uh but also making work that is

tasks. uh but also making work that is like more iterative and actually uh longer running. So like that's been one

longer running. So like that's been one thing like we're starting to see I think more of an inflection point on both like complexity and also um being able to iterate and continuously improve um some

of the deliverables. I think computer use has been another one like um we saw and have been investing in computer use for a long time. Um, last year in

November, I think November, October, we launched a computer use API and since then we've continued to invest in it.

Um, and like I use things like uh cloud for Chrome which is like our browser use extension feature pretty often and like I saw like an improvement just in terms of the quality of that interaction

because now cloud's vision is much better. Um, so I think sometimes it's a

better. Um, so I think sometimes it's a combination of like multiple things kind of working together. I guess there's, you know, a lot of uh builders listening to the podcast that, you know, are curious about computer use and do you

have like a rough mental framework today of like where computer use works and where it doesn't and kind of where we are on the journey? Yeah, I think like computer use have gone from this like

very early stage experimental feature when we launched it. Um and we were initially thinking about it as like probably the first stage is like a complement or a feature on top of

something core like agentic coding to now I think more and more between Opus 4.5 and beyond able to be more of a like end to-end agent just by itself and so

like can it not not just like within a more constrained environment look at the types of like you know QA testing that was a like popular use case but actually

like be able to help monitor manager and be an agent in like a web browser. It's

still more constrained than like an open-ended like let's say your entire like um laptop, but it's like very helpful to be able to have like an agent

that like helps me reschedule my calendar on Google because that tends to get pretty complicated. And yeah, it's I think that's kind of just been kind of the arc like moving from more

constrained environments to towards uh things that are a bit more open-ended.

Was there anything that you uh played around with with Opus 45 when you first had it internally that like was uh was was surprising?

>> It was actually really helpful from a product team perspective to debate like things like pricing and like how our positioning for the model um because it's such a great like leap in many

ways.

>> Um so from a property manager perspective it was actually not just a great writer but it was great thinker.

it came up with like you know different different uh ideas around things like pricing and positioning um where it wasn't just refining my ideas but it came up with alternatives uh more

spontaneously than I had seen with other models >> you know I feel like traditionally a lot of the opus models were like much more expensive and this is both a really good model and like quite cheap honestly uh

what what like drives that and I guess how clear was it from the beginning of this work that like this opus model would be able to be served this efficiently From the beginning, we were hoping that this would be a thing for

Opus models where we're able to have more efficiency gains and we're able to pass it then on to our users and customers and builders. Um, we

intentionally also made it possible in addition to like the core model training for things like the effort parameter which I actually think is like underhyped right now. [laughter]

Um, where you could actually get like set 4.5 level of intelligence at a fraction of the price. Yeah.

>> And I think this is something that like we as an industry haven't gotten really good about which is like just because a model has a certain token price and

sticker price. That's not always a good

sticker price. That's not always a good measure of like the end to end cost to achieve a task. And what we want to do with Opus and what we want to do with things like the effort parameter is to

make it more accessible that like you can actually achieve a higher quality and a lower cost. Um, and you know, we're starting with that w with Opus and I'm really excited about this area going

forward as well.

>> Yeah. Do you feel like most most folks like get that uh intuitively or like what how do you think about educating the market around Yeah. this idea that it's not just the per token cost but obviously the amount of tokens it takes to to complete some of these tasks.

>> Yeah, I think it's something we as model providers can be doing more around. So,

one thing I hear from builders is the smaller models actually just take longer to do a task or it might actually not even get the task done, but you've spent a bunch more tokens than if you just use

an Opus model. Um, you know, for a lot of this year, we were traditionally focused around like Sonnet and having a very core flagship offering. And as we move into like different tiers of

models, it actually is becoming more important for us to educate developers and users on it. Um, and so like I think this is an area we're going to be investing in from like a marketing perspective.

>> You all have a very good intuitive sense of what these models can do, but then when you put it out there, I'm sure they start getting used in like a million different ways that you that you couldn't possibly have conceived of internally. In these first days of

internally. In these first days of release, like what have been some of the things that have surprised you the most?

>> Yeah, I think it's like very early.

We're also in the middle of Thanksgiving week, so I'm sure my answer will will change a little bit as like uh people have like tested the systems. I would

say maybe like two things. Um one in our early access with customers um you know, we invested in areas like I mentioned around uh making the models better at

like office um like deliverables. Um it

was really surprising to see how big of an accuracy jump OPUS 4.5 was for customers like uh shortcut which is like um a cell agent. I think customers were

saying something like 20% accuracy improvement just like without changing harnesses without making other changes.

So that felt like resonated a ton cuz then they can pass that intelligence improvement to their users. [snorts] Um

I think other things that I've seen just like generally um in the early days I think people tend to test the models on like gaming use cases and so like 3D games have gotten better [laughter]

u in particular. Um so that's always really cool visualizing some of this have just been an easy way for people to kind of see intelligence bounds. Um and

I'm really excited also for just like the feedback that we're getting around quality. So customers and users saying

quality. So customers and users saying like, "Oh, it's actually helped me clear out my entire backlog of bugs."

>> Now with with Opus 45, as you think about kind of the broader ecosystem of applications and and you know, I'm curious your mental model of like what does have product market fit or like

really works today uh on top of these models. I think the big ones are, you

models. I think the big ones are, you know, agent coding is like very visibly um something that's like here to stay.

Um I think we continue to actually have more like enterprises reach out about solutions like cloud code and like coding capabilities. I think like

coding capabilities. I think like synchronous agents um have pretty strong uh uh reason for

PMF generally beyond coding. I think

it's more that like we've not figured out the we as like a like a um general like industry haven't figured out the right like harness plus product features

to build on top like what is the you know agent coding but for like you know web monitoring and like personal agents a lot of use cases are still very like

chat focused and so I feel like we're kind of at an inflection point that we need a bit more intelligence to kind of really boost the neck large set of use cases >> and do you think that like we have that level of intelligence now for those use cases?

>> I think people will be surprised by how good Opus 4.5 is and I think PE there will be new things that are born from from Opus 4.5. There will be new features and products.

>> So more of these like proactive experiences or like agents going out and doing things on the beha on your behalf and then surfacing it up to you versus you prompting like via chat. Yeah, that

was one thing we heard a lot from like internally where it felt like the model just got it without having to have in explicit human instruction. And I think

when you couple that plus the fact that um one the context is getting high the quality of the context is getting higher and also things like memory start

working better. really you get a system

working better. really you get a system that can do not just one deliverable but like do things that are not possible before like monitoring maintenance like

those are actually things that are like really valuable. Um it's not really just

really valuable. Um it's not really just about shipping an MVP. It's about also like how do you maintain um something that you build.

>> I'm curious you brought up that example of of shortcut and like really improving Excel. I imagine there's like an endless

Excel. I imagine there's like an endless set of people that come to you and they're like we would love you to improve your model on like X domain or Y task we have. How do you prioritize that and like do you think that that the

foundation models end up going different directions just based on what they prioritize here?

>> I think a little bit yes right like there's you know we do have customers that ask us about things like image generation or video and we're very intentional

um and I've been here about like two and a half years. who've been very intentional about like focusing on like expanding intelligence and so I I think

you do see that a little bit with the labs already. Um in terms of your

labs already. Um in terms of your question of like feedback like um I think it's kind of a two-way flywheel, right? Um sometimes there are like pain

right? Um sometimes there are like pain points or paper cuts we hear from customers. What's really good about like

customers. What's really good about like our shipping velocity today is if a customer comes to us or a potential pro customer comes to us, we can make changes pretty quickly because if it's

um in 4.5, maybe it's in cloud five, right? Like you have like multiple

right? Like you have like multiple chances to get your feedback heard. Um

and then how do we also automate more of that? like how do we build the systems

that? like how do we build the systems on our side to like more organically figure out the right environments to build and like build that like loop

ourselves um and then how do we deliver like new advancements uh more more generally so like things like computer use and beyond so I think it's it'll be birectional >> yeah it's interesting you talk about

building some of these tools internally like uh you know more easily spinning up environments and being able to uh lower I guess probably the amount of effort that is required to, you know, to improve the models in these specific categories. And I'm sure there's like

categories. And I'm sure there's like internal tooling that really helps with that.

>> Yeah, it's definitely a continuous um investment area. We just have a lot that

investment area. We just have a lot that we could be doing and so we have to be really intentional about like where we spend like to the point how do we get Claude to help us? [laughter]

>> Yeah.

>> When we're done.

>> I mean, it sounds like multimodal like you know obviously has been below the the the you know the focus area. Are

there any other things that you think of of like, yeah, we could, you know, we've explicitly chosen not to, you know, spend too much time on X or Y?

>> Um, I think those are some of the big ones. I think like we've also from a

ones. I think like we've also from a product perspective been very intentional about focusing on business use cases generally, right? And so like

this like means that like a lot of our focus and investment in product is around things like data security privacy meeting like enterprise requirements um for them to be able to

adopt and use cloud over like let's say like consumer use cases.

>> It feels like there's been this like really interesting discourse around enterprise agents over the last few weeks. I think largely driven by the

weeks. I think largely driven by the podcast ecosystem. Uh where you know you

podcast ecosystem. Uh where you know you had I guess Andre Carpathy going on Dark's pod and being like you know hey I don't know if these this is really it like it feels like we're still a decade away from from real enterprise agents

and then I guess Ilia most recently came on and was like this current paradigm only gets us so far like we're we're going to kind of hit a wall you know you then obviously square that with all the improvements in something like Opus 45.

Where do you kind of what what's your opinion on all this discourse and like the the feeling I guess maybe within anthropic around it? I I think um intelligence and improvements and model

improvements what I've learned is like improvements are not necessarily like always a smooth line the way that you see in an eval benchmark of like it's always like this and I think they're

more jagged edged and I think depending on what eval or personal test that you have maybe it feels like a small jump or maybe to someone else it feels like a

large jump. So I think there's like a

large jump. So I think there's like a level of like what is the exact like framing you're looking at. Um I think like from a customer and user perspective

what we hear from like a rakuten or a lovable etc is that like capabilities have actually still continue to to improve their team's

productivity and I think like really from like a is AI transforming technology and we talk about like less AGI but more like transformative AI

internally and anthropic like are we making technology as that is transformative. I think we're very much

transformative. I think we're very much on the path on that. Claude is like transforming every generation of Claude transforms internally how anthropic employees work. And so we feel that

employees work. And so we feel that because we're adopting it in a different degree. And so maybe if you're not

degree. And so maybe if you're not adopting it to the same degree, it might not feel like it's meaningfully like make making advancements.

>> You've obviously gotten very far and I imagine some of the things that were on the board are off now. But what's on like the next board? Nobody can predict the future. But I think like some of the

the future. But I think like some of the areas that I'm really excited about is like this move towards like longer running intelligence. So again, not just

running intelligence. So again, not just it a human gives and delegates a specific task to Claude, but actually like Claude taking responsibilities that

are more open-ended. So, not just like build me this like portion of my website, but maintain it, refactor the code when you think it's correct and not

needing so much of like uh handholding.

Um, I'm also really excited about capabilities like Office uh improvements that I mentioned, things like Excel, PowerPoint, there's a lot more to go.

Um, and also computer use. I feel like computer use is actually um on the way of a like adoption and quality where

like it could really be transformative um from like um you know enterprise and also just like general user perspective the way that like a a coding could be

like computers like being able to navigate computer is how most of us interact with each other. We're on a podcast right now, right? So like uh being able to have claw that like can

interact in these environments where like people work makes them much more useful. Um so I'm really excited about

useful. Um so I'm really excited about like where computer use will go.

>> I mean you mentioned longrunning agents and I feel you know it feels like this is certainly where the world's headed and you know maybe even having a fleet of agents doing work on your behalf. It

feels like in in many ways, you know, obviously the models will get better and but in many ways part of this is solving the product side of it and like what does it actually look like to have a long running agent uh going on in the background and what does it look like to

have a set of a dozen of them that you're managing. Uh you're obviously a

you're managing. Uh you're obviously a product guru like how do you think about what the product surface area might look like here and and you know have you seen anything that that you thought was particularly interesting?

>> Yeah. Um I think longunning agent and like longunning intelligence um generally is like a it's not a use case in itself right we need to like

actually have like a user problem that it's like a good fit for um so I think things that are like really valuable are around like being able to maintain

iterally improve right like let's say you have like um a like you you're an investor and you're you want to like understand and the latest like stock movements or like how you should adjust

your portfolio. That's not like a

your portfolio. That's not like a one-time thing. And I think like what we

one-time thing. And I think like what we kind of lack today is like having these like really long horizon task in a way that makes it easy to eval quality

improvements. So like the closest one

improvements. So like the closest one that I think we're starting to like um track and it might not be the right one because um I think you know no eval is

perfect is like vending bench which is >> um the the Claudius like run a vending machine business right but that's >> I love that you guys do this [laughter] >> or or or you know Claude plays Pokemon

right like these just you know Claude played Pokemon when we first >> I kind of missed the Pokemon eval that I like I don't know if I just missing it in the reports I liked when that was like the one one of the main ones people were discussing.

>> Okay, I am I will take a note for like potentially bringing it back. Um uh but like >> maybe the models are all too good and they beat the game really quickly, but uh >> well well it's also not just about completing the task, but like how long

does it take, right? And like can you actually complete the task in much less steps because you're not brute forcing the problem and like you remember the fact that like Ash Ketchum was like in

Pallet Town at this time and already talked to like some user, right? Sorry

to nerd out for a second. We can go deep on Pokemon. I'm I'm I'm all all game for

on Pokemon. I'm I'm I'm all all game for that.

>> Okay. Yeah. But like I guess the the the part of what I want to know here is just like you know intelligence isn't just about like can I complete a task and check the box more and more it's also

the quality of the judgment and the quality of like how what intuitions the models can have in a situation. And so

can we actually dramatically reduce the time it takes or the amount of effort to get to a certain outcome? And sometimes

this really shows up in things like around longunning tasks, right? Um but

really I'm just like excited about this area. I think what we need more as an

area. I think what we need more as an industry is better eval around it >> and what are the eval?

>> Oh, it's a very good question. Um I

think we would need to evolve beyond things like Sweet Bench and like Taen. I

think we're especially with Opus 4.5 um I think we got to like 80.9% and like these DC valves are so saturated.

>> Also love the example of like finding a way to upgrade the ticket through like you know changing a fair category and uh that was a very fun way to uh to to to to get around things.

>> Yeah, it's like it's crazy cuz like when I shipped the tool use API last year I think we were in like maybe below 50% on accuracy. It's just again there's so

accuracy. It's just again there's so much advancement. It's really how do we

much advancement. It's really how do we measure and how do we productize it. Um

I think the evals that matter continue to be for us like what are the areas where people are using claude and how well does it work. Um so I think like end user measurements of quality

continue to matter or feedback that we get from customers. I do think we will probably see more movement towards evals that are more open-ended like a bending

bench sort. It's not going to look

bench sort. It's not going to look exactly like that cuz I don't know how realistic running a vending machine is all the time, but like that type of direction of

like it's open-ended. There is some quantifiable way to measure quality, but it's not just like yes, no, cuz how much of like most things in the world are

yes, no tasks, right? if we're moving beyond coding into other types of like impact like it's hard to say like is

there a yes no like SWE equivalent in like biology >> but you've kind of alluded a few times now to you know the harnesses that people are putting around these models and obviously you work very closely with teams that are at the cutting edge of

using Opus 45 how would you like you know do you feel like there's a typical kind of set of scaffolding people are building around models today >> I think similar to model intelligence scaffolds have evolved

Um, I would say in like 2022, 2023, even 2024, a lot of the scaffolds are more like training wheels to keep like

the model on distribution and they tend to be of the form do not do this, always do this, right? Just like uh

instructions and like 20 rules. Um and

uh I think more and more this year what we've seen is scaffolds become much more around augmentations for intelligence rather than training wheels or like the

best scaffolds tend to be and uh iteratively removing the parts of the scaffold that are no longer intelligence amplifying. [snorts] So for example on

amplifying. [snorts] So for example on like cloud code our scaffolds are relatively lightweight. Um the types of

relatively lightweight. Um the types of tools we give it are things like batch tools that are not like specific very unique and the point of that is

that we want to maximize autonomy of like the work as it's being done by the model and so the like um types of scaffolds that I think will

continue to be valuable might be things that are intelligence amplifying so things that are like giving it like generic sets of tools uh multi- aents I

um have started to be like more viable this year um where it's like not just like having one one model but like orchestrating a set of models to like amplify and improve on things like

context quality etc. >> I think a bunch of builders are asking themselves you know what set of this stuff becomes obiated by the next generation of models uh versus you know continues to amplify intelligence for

that next set. Is it obvious from the inside or is it also like you know we'll we'll see when we get to those models. I

think there is a bit of like this is why having like a thinner layers of harnesses and like scaffolds is important. Um I think that like how much

important. Um I think that like how much does it change like it's a little bit hard to predict but we do see

improvements in in quality by some level of iteration on scaffolds by model. Um

part of it also users requests also change right like what we see is like you know when I use like cloud code or use cloud um if I see that it's really good at editing a document I might give

it a large set of things and like hey come up with an alternative strategy right and so like we're constuously also as users pushing the products that people build and so it's not just in

service of like new models to update your scaffold but actually in service of like user behavior and users will naturally push your product to be slightly more complex and

how do you as like a developer meet that demand is like I think the the the the right way to think of it.

>> You've kind of been at anthropic since the early days and so uh I I feel like it must just be been a fascinating journey these past uh two and a half years. I I wonder maybe to to start like

years. I I wonder maybe to to start like how do you kind of compare the culture of Anthropic to like other places that you've worked? I would say a lot of how

you've worked? I would say a lot of how like you know our leaders um show up in and and uh how people um talk about the

mission and the goal. It's like very much actually how we internally a big part of how internally we make decisions like um it's actually a lot of like

walking the walk and I think anthropic has been the most like authentic um and as much from like the first day I joined to like now uh of a company that

I've been in. I think it goes without saying but like the talent caliber and talent density has been incredible. It's

been the most like talent dense like environment that I've gotten to work with. And um I think people are like

with. And um I think people are like just deeply take like radical ownership, deeply thoughtful, kind, but direct and

also in the service of making products, models, capabilities, research better.

>> I'm sure our listeners will be curious like what does your day-to-day look like? I think like every 3 months my job

like? I think like every 3 months my job changes cuz it's almost like we're a different company. I think we were like

different company. I think we were like 150 people when I joined and I like two years ago around this time I was setting

up like our first AB tests like and emailing prospective customers to like try new models right it was like extremely hands-on you know like as a PM

you do whatever it takes to get the thing done. Um, I think now more

thing done. Um, I think now more day-to-day as we have like more product managers, like more researchers, more engineers, a lot of my day is spent like

helping the team and a lot of my PMs are more embedded with their research counterparts. And so like coaching,

counterparts. And so like coaching, supporting them is a big part of my day.

Maybe a third of the time with calls with customers just better getting a sense of like all of the different use cases particularly [snorts] what's emerging like what is not working today

that like is really uh could be really pivotal. um so spending a lot of time

pivotal. um so spending a lot of time with like either businesses or startups and then uh I think you know especially in this environment thinking about what's ahead

>> obviously what a wild last two and a half years you know I guess as you reflect back on that like journey are there some like key decision points that stick out to you is like oh that that really like tipped things uh or or was

pretty pivotal >> I think we were very intentional um in the early days I would say in 2023 Three, the most

common user request we got was you guys need to have an embedding model because we're doing rag and rag is all the rage. And I think we were in very

the rage. And I think we were in very intentional about what LLMs and AI could do and what cloud could be. And we

focused on things like investing in agentic coding. Um, and so this is a

agentic coding. Um, and so this is a little piece of like being userled versus user centric, right? Like what is the thing that they're not even asking for? like nobody's coming down our door

for? like nobody's coming down our door in 2023 being like gave us injected coding. [laughter]

coding. [laughter] Um so so I think that's one like just the early days of like being very clear focused around what is the bigger like

opportunity. Um I think another one

opportunity. Um I think another one personally is when we did choose to ship computer use um on the API and we shipped it as an a

beta feature. Um, we knew that it didn't

beta feature. Um, we knew that it didn't quite work completely yet, but we thought that this was really helpful for like showcasing what AI could do. Um,

and how it's just a different form factor. I I think we're still on that

factor. I I think we're still on that journey, but like bring that to the world. And there was a lot of

world. And there was a lot of discussions around like, hey, how do we make sure like everything is like very safe, etc. And so, we did a lot of safety work, but it's impossible to

capture every edge case. And so we had to put it out into the world a bit to like then figure out what else we need to like make safe as the capabilities

expand. And so I thought that was like

expand. And so I thought that was like just like a very authentic um moment making a bold decision um that I think was the right one. And

yeah, I think those are those are probably like two of the large things.

Um there's a lot of other fun moments, but they're not like I think pivotal in my mind. Well, what's what sticks out as

my mind. Well, what's what sticks out as a fun one?

>> Um, so my team and I worked on Golden Gate Claude, which was an >> Oh, yeah. [laughter]

>> of our interpretability work. And that

was very cool because um I think the company was still less than 500 people, but it was starting to feel a bit bigger. And we shipped Golden Gate

bigger. And we shipped Golden Gate Claude from model to the UI within less than a day.

>> [laughter] >> Did you know it was going to be this like kind of viral thing?

>> It went a little viral internally. So I

think like usually anthropic employees have good like product sense and taste.

So we were like, "Okay, if we like it, let's just try it. Maybe it's like we get like maybe a couple hundred people who like geek out with us." Um and uh

but we we didn't know. Um but yeah, it was like very grassroots. some

engineers, researchers, uh PM and designer. Um we were like, okay, let's

designer. Um we were like, okay, let's figure out how do we show this to users and we needed to do it in the week because we had published a paper. Um so

that was probably one of like the proudest moments.

>> It has been a fascinating conversation.

We always like to end our interviews with a quick fire around where I basically just stuff in as many questions as I can before we uh we run out of time. Um and so maybe to to start, what's one thing you've like

changed your mind on in AI in the last year? I actually think we are closer to

year? I actually think we are closer to transformative longunning AI than I expected even starting the year. Like it

actually feels like the building blocks are kind of there more I feel that more now than than than in the beginning of the year.

>> You obviously have a a a range of sophistication of of end users of of the cloud API. Are there things that the

cloud API. Are there things that the most sophisticated folks do that you wish that just all of our listeners uh would would do with these models to use them more effectively?

>> I think there's like maybe uh two two big things. Um, I think the

boldest like builders and users um are constantly thinking about like not just what's working today, but actually might have things off the shelf that like did

not work before or some prototype that doesn't quite work. Um, that like they still kind of put together and uh test

with new versions of like models that come out. Um so like we call this kind

come out. Um so like we call this kind of like prototyping like just having library uh user centric prototyping and I think like that has been something that's like been really valuable to see

of like because a lot of times these systems are not planned it's almost like you have to discover the capability and if you don't have a prototype then you're not going to discover the thing

you're always going to wonder oh I I wonder when it's going to get really good at drug discovery but if you don't have like some way to actually like test that each time like it's always going to

be too to to to abstract. So I think like um builders having very ambitious like prototypes, product ideas or features that might not work in the past but just like having things like

hackathon where you could actually test these ideas is really important. And

then I think like being willing when you have a new version of a technology to like invest in actually uh potentially

changing your product experience to like meet the intelligence tailwinds.

>> Yeah. What is model taste?

>> I think of model taste as like just like you know product sense and like product taste is kind of develop over time. It's

really like a contin continuously honed sense of like model capabilities and a willingness to discover the capabilities

uh by being hands-on and also by understanding what users are trying to do and like continuously like iterating on that. What do you think model tastes?

on that. What do you think model tastes?

>> Yeah. No, I mean I love the I love the idea of like um you know getting your fingernails dirty with the models, right? And just like you know developing

right? And just like you know developing some intuitions to uh both you know what they can and can't do and also the right ways to push them or build scaffolding around them uh you know in order to uh get the most out of them. There's just

some people that uh you you give everyone the same tools and they always seem to just get way more out of them.

>> Yeah. I think it's like your willingness to like experiment like I think um kind of creatively problem solving is not the perfect word but people who are

willing to kind of like approach models as like a new discovery >> and you're constantly like trying new things um that's kind of a way of developing model taste. Well, I'm struck

by too, you talked about, you know, folks having, you know, being able to try new things as the models come out.

You know, I imagine it's not as simple as just like plugging in Opus 45 and seeing if it works because obviously there's a lot of scaffolding one has to build. And so I imagine part of the art

build. And so I imagine part of the art is also being able to have some intuition as to, you know, I'm not just switching out the API key and up it doesn't work. Uh, but I'm but I'm

doesn't work. Uh, but I'm but I'm actually have some idea of like things to tweak that that actually might make it work.

>> This is why we actually internally have things like hackathons pretty regularly.

I think we do a hackathon maybe every three or four months internally at Anthropic just because people have these like pentup ideas of like I wonder if cla can now do X and you really need to

give um builders, engineers, product people a place to do that creative work and do that discovery work outside of their day-to-day. Um because yeah like

their day-to-day. Um because yeah like you know nobody asks for a thing like agentic coding to work >> and it has to be a bit discovered. It

has to be a bit, you know, hands-on uh to to to figure out if like something is possible.

>> Yeah.

>> Um and giving people the place to do that is really important. You know,

>> I guess you you talked about obviously longrunning agents um you know, being being closer uh and all these things.

What's like one thing you think of as a society like we're not talking about enough or one of the implications of of all this um that maybe is under discussed? I think we don't talk enough

discussed? I think we don't talk enough about the benefits of safety, not just of the fact of hey it's to make sure the model

doesn't do bad things but more also like what is the benefits of an aligned model. One of the key issues that is

model. One of the key issues that is like an active research area is around things like sync fancy. Yeah.

>> Right. Um that like models are just tell you what you want to hear and actually a well-aligned safe model.

It's not it's actually the opposite. So

it's actually independent thinker, right? And independent thinkers are

right? And independent thinkers are actually how we have breakthroughs and how we have better ideas. And I don't think we really talk enough about why

safety is actually really good for higher value intelligence too. It's not

just to constrain um AI. It's actually a way to like amplify quality of intelligence if it works well. Like

coming back to my like example in the beginning, like I asked Claude like, "Hey, how do we think about pricing?

Here's two options." And it comes up with a third that was really good that pushed my thinking. And if we had a very sacrifantic version of Opus 4.5, it might not have done that. It would have

just agreed with me, right? And it's

just like I don't think sacrifancy is solved by any means, but just like by investing in like making AI that's aligned, we could actually get to a better intelligence. I feel like we

better intelligence. I feel like we don't talk about.

>> I love that example. You know, obviously I I think folks at Anthropic famously have like very uh aggressive ASI timelines. Like do you do you fit into

timelines. Like do you do you fit into that bucket or or uh how do you think about that? I would say my timelines

about that? I would say my timelines have probably moved up this year uh based on things of what I'm seeing with thing uh models like Opus 4.5. Um I feel

like the building blocks are actually closer than we think. Um and that it's actually more of like a product overhang or like product opportunities to express it.

>> Um and how do you like again build the right scaffolds or build the right ways to harness like the model quality? Um,

and so I'm more of like a barbell. I

think like there's like a very large probability in short term and then like very large probability in long term.

>> Well, this has been fascinating. I feel

like I I will delay the uh the launch of the future generation of models if I keep you any too much longer here, but um I I just want to make sure to leave the last word to you like where can folks go to learn more about Opus 45, about you, about anywhere you'd like to

point them. Uh the mic is yours.

point them. Uh the mic is yours.

>> Yeah. Um I think our blog and our website. Um, I think that, you know, we

website. Um, I think that, you know, we are pretty like, um, we like to let the worst speak for itself. So, I think following on on our website is the right place.

>> Yeah. I guess you guys famously don't do the like coming tomorrow or like the cryptic, you know, tweets before model launches that everyone else seems to do.

>> No.

>> Well, awesome. This has been fascinating. Thanks so much for uh for

fascinating. Thanks so much for uh for the time.

>> Thank you so much. It was great chatting.

>> [music] [music]