AI 2027 Co-Authors Map Out AI’s Spread of Outcomes on Humanity

Can you give me a milestone that I should get my head out of the sand for?

I would say the superhuman coder thing.

Superhuman coder. That's so late. Are

there versions of the world that you've thought about where you think the public wakes up to this sooner? I don't

actually expect people to wake up in time. This was a pretty scary

time. This was a pretty scary experimental evidence for me. How close

are we to the model starting to do this like egregious sort of alignment faking behavior? Setting aside all the safety

behavior? Setting aside all the safety issues, even if you think the AIS are going to be totally obedient, who are they going to be obedient to? What would

cause society to sort of wake up to this? Our answer is something like

this? Our answer is something like Daniel Katalo is a former researcher at OpenAI who now works on alignment full-time through his nonprofit AI features. Most recently, he co-authored

features. Most recently, he co-authored the AI2027 report with dire warnings for the future of unaligned AI. Daniel's

been named as one of Time's most influential people in AI, and he alongside Thomas Larson, one of his co-authors, are two of the most important voices in the AI safety debate. In this episode, we talked about

debate. In this episode, we talked about the AI 2027 piece, policy implications, and Daniel and Thomas' thoughts on how we get better aligned models. It's a

really interesting conversation. I think

folks will enjoy it a lot. Now, here's

Daniel and Thomas. Daniel Thomas, thanks so much

Thomas. Daniel Thomas, thanks so much for uh for coming on the podcast. Really

excited for this. Thank you. Thanks for

having us. I'm sure most of our listeners have have been exposed to AI 2027, but for those that maybe uh aren't uh fully up to date on on your work, could you start with just kind of a brief overview of the piece, your kind

of timeline, and and and you know, kind of the key key points you articulate here? Sure. Uh

here? Sure. Uh

so you probably have heard that the CEOs of anthropic uh Deep Mind and OpenAI claim that they're going to be building super intelligence uh possibly before

this decade is out. Um super

intelligence meaning AI systems that are better than humans across the board while also being faster and cheaper. Uh

lots of variations on that claim also going around. Um, it'd be easy to

going around. Um, it'd be easy to dismiss this as hype and perhaps some of it is hype, but uh we actually also think that these companies have a pretty good chance of developing super

intelligence before this decade is out.

And uh that's crazy and that's a big deal and everyone needs to be paying attention to that and trying to think about what that might look like and game it out. Uh and so that's our job. We

it out. Uh and so that's our job. We

spent a year uh making our sort of best guess forecast for how this is all going to go down. Um, and uh, there's lots of uncertainty, but but here it is. And

actually, it's more my best guess if you want to get into the details. Different

people on the team have somewhat different opinions in particular on timelines. So, at the time that we

timelines. So, at the time that we started writing this, 2027 was my median for when we would get to AI is better than the best humans at everything, end

of 2027. Um, now I'd say my median is

of 2027. Um, now I'd say my median is more like end of 2028. um other people on the team would be more like 2029, 2030, 2031, something like that. But we

were all sort of like in that range. And

so we came together to write this thing that represents like a sort of like modal or best guess trajectory. What

happens in the story? What happens in our scenario? Uh well, the companies do

our scenario? Uh well, the companies do what they say they're going to do. They

continue to make the AIS more agentic.

They scale up the reinforcement learning runs. Um they plug them into all sorts

runs. Um they plug them into all sorts of tools. They trained the agents to

of tools. They trained the agents to operate autonomously on computers, to write and edit code for long periods of time without human intervention. Uh, by

early 2027, they're basically fully autonomous and good enough at coding that they can substitute for programmers. So, we reached the the

programmers. So, we reached the the superhuman coder milestone early 2027, but they're still limited in some ways.

for example, they're not so data efficient compared to humans and maybe they lack research taste and various other important skills that are necessary uh for AI research and perhaps they also lack other real world skills.

However, if they're good at the coding uh then they can start to speed up the process of uh AI development in particular algorithmic progress. So over

the course of uh AI27, we depict that intelligence explosion happening. Starts

off slow at first, but then it gets faster and faster as the AI capabilities ramp up through additional milestones, eventually reaching super intelligence by the end of the year. At that point, roughly around that point, uh AI 2027

splits into two branches, the the race branch and the slowdown branch. Uh the

reason why we have this split is that I mean basically first we wrote the race branch which was just what we actually thought was the most likely thing to

happen and uh in that branch um the AIS end up misaligned and pretending to be aligned. Uh but because of the arms race

aligned. Uh but because of the arms race with other companies and with China, uh that just sort of goes undiscovered basically until years later when the AIs

are in charge of everything. They've

completely transformed the economy.

There's AIS in the military, AI automating factories, all sorts of robots everywhere. Uh and then it's too

robots everywhere. Uh and then it's too late. They have too much hard power and

late. They have too much hard power and they can just do what they want, including killing all the humans to free up the land for more expansion. I guess

that would have been a a really depressing piece if you hadn't at least put in another scenario, right? And then

we were like, but you know, maybe things will be better than that. Maybe in

particular the alignment problem on a technical level will be solved to a sufficient degree that humans remain in control of their AI systems even as they become super intelligent. So we wanted to depict uh what that might look like

as well. Um and so we made this

as well. Um and so we made this alternate branch that uh branches off from the race ending in you know mid2027 where it depicts the relevant company uh

you know investing more in some technical research doing some faithful chain of thought stuff discovering the misalignments fixing them in a more deep way that actually scales and then continuing the intelligence explosion

safely. Uh and then you still get the

safely. Uh and then you still get the arms race with China you still get the massive military buildup all that sort of stuff. Uh but it ends with you know

of stuff. Uh but it ends with you know the humans still in control. Uh which

humans? Well, specifically uh the tiny group of humans who uh were in charge of the project. Uh in in A27 it's the

the project. Uh in in A27 it's the oversight committee which is an ad hoc group formed between the president, some of their appointees and the CEO of the

company. Um and so you know one of the

company. Um and so you know one of the things we'd be interested to talk about and that we hope people think about is the sort of concentration of power

aspects of all of this. Um I think that it's a sad but true fact about the world that by default we are on a trajectory for a massive concentration of power

where there'll be you know to quote Dario Ammedai the the country of geniuses in the data centers and it's like well well who is that country of geniuses listening to who are they loyal

to who who who says what goals they they pursue right um worst case scenario it's a literal dictatorship it's one man who gets to call all the shots Um, yeah. And

in your scenarios, who these 10 people are are pretty pretty damn important because it's a 6-4 vote, I think, both ways, right? Uh, that that determines

ways, right? Uh, that that determines which way we go, right? Right. Um, but,

uh, yeah. So, so there's there's lots to get into, but that's that's the sort of like potted summary of the plot uh, of 207. Super helpful. So, I think there's

207. Super helpful. So, I think there's a ton I want to dig into. Maybe we'll

just like go along the trajectory of the timeline that you you put forward there.

And obviously, you know, a big part of what leads to the progress in your timeline is the creation of, you know, agents for AI research, right? And uh,

you know, uh, things that can actually accelerate uh, the pace at which we do AI research. And it feels like this is

AI research. And it feels like this is the focus of a bunch of the labs. I

mean, we had Nome Shazir on the podcast and he was like his next milestone is when, you know, Gemini XUS1 writes Gemini X, right? So, it's like clearly the focus of of where all these labs are going. You've said, you know, very

going. You've said, you know, very explicitly that like nearer term predictions you're much more sure about.

Like, how do you think about the problems that still need to be solved to get here? or is it kind of just like an

get here? or is it kind of just like an inevitability in your mind that we're here end of year? Um, you know, maybe talk about that a little bit. For our

perspective is that it's an inevitability that we eventually get AGI. I think we're pretty sold that

AGI. I think we're pretty sold that there's really nothing stopping machines from getting as smart and then smarter than uh than how, you know, intelligent humans are. Um, I would say we're really

humans are. Um, I would say we're really not sure. We do not at all think it's an

not sure. We do not at all think it's an inevitability that it it happens exactly at the timing that we set. So, so Daniel sort of alluded to this earlier, but uh my view at least is something like a

median of of 2020 of 2031 AGI timelines and then maybe 2032 super intelligence timelines. Daniel is a couple years

timelines. Daniel is a couple years earlier. I think we're generally we're

earlier. I think we're generally we're all pretty uncertain and sort of think that the sort of no one knows and it really depends on how the research shakes out over the next few years. Um

in terms of the barriers what we think what we think you know the current models are really lacking I think the main one we think of is sort of a ability to act on long time horizons um

the current models right they can do very small bounded tasks you can you know you have your chatbot you can have it do like you know write a specific function if you're coding or you can have it like respond to a specific query

you can't really give it a high level direction like you could an employee and then have it go off for a day or a week and then come back to view with sort of the results of that day or week of work.

Um and so our view sort of is that the m one of at least yeah maybe the central bottleneck towards AGI is getting uh increasingly long time horizons. And so

one of our main ways of forecasting AGI timelines has been sort of trying to look at the best data available um for time horizon capability and trying to extrapolate that. And since you

extrapolate that. And since you published the piece there were like additional data points, right, that fit nicely in your uh in your plot. Yes,

exactly. So yeah, Daniel had a tweet yesterday or two days ago or something where you know we fit the points and it was basically exactly on trend with uh the super exponential fit. Although if I

may if I may add to that um I would say the main argument that we we use which you can read about on a27.com of course in the research page um is called the

benchmarks plus gaps argument where we you know it's it's become almost a truism these days that benchmark performance is going to just keep going up really fast and that all the benchmarks are going to get saturated in

like a few years. So the benchmarks plus gaps argument basically says okay let's plot out the trends when are the benchmarks all going to be saturated answer you know 2026 or something like

that. Uh and then it's like okay let's

that. Uh and then it's like okay let's think about the type of AI system that can saturate all these benchmarks and think what's the gap between that system and a system that can automate

engineering effectively at at these core companies. And that's the hard part is

companies. And that's the hard part is figuring out the gap and and guessing like how long it's going to take to cross that gap. Um, one of the there's different like components of that gap.

One of those components is the long horizon agency that Thomas was just mentioning. Even our best benchmarks

mentioning. Even our best benchmarks these days are measuring relatively short tasks. Um, however, they're not

short tasks. Um, however, they're not like crazy short. Like they're like 8 hour tasks sometimes. So like it's not that hard to sort of extrapolate and be like well if you have a system that can crush all the eight hour tasks that we

throw at them maybe with some additional training etc you can get one week tasks and and so forth um or one month task. Is there

anything that could happen like in the next 6 months that would like completely change your mind on this? Um, I mean obviously like I totally hear you on the on the precision of the timeline not being clear, but if we were not following that exponential or like how

do you think about you know how soon uh there might be cards that flip over that that cause you to kind of fundamentally rethink some of the stuff? So in my mind if the trends break so if we stop seeing sort of the current benchmark performance increases that we've seen

historically and that we you know predict will continue to happen. I think

if that if that stops then our predictions are going to be wrong. We're

going to be wrong about timelines. we're

going to update pretty strongly in the um longer timelines direction. Of

course, just because the benchmarks stay on trend, I don't we don't think that's sufficient for us to be right about like 2027 timelines. As Daniel said, most the

2027 timelines. As Daniel said, most the uncertainty from our perspective is still in the gaps part, not the benchmarks part. Uh and so

benchmarks part. Uh and so unfortunately, it's much harder to get data about that, right? It's it's harder to see, you know, exactly how difficult will the gaps be to cross. Uh and so a lot of folks you know who are skeptical

or who have much longer timelines than us will sort of I think their view is more just like you know the gaps were really really big the current benchmarks really aren't measuring the important skills and that's you know harder for us

to get evidence about and we don't really expect update on that in the next year or so. Yeah, I mean obviously it seems like an important part of uh you know important part of this work is is even coming up with better you know eval benchmarks for like what this AI

researcher might look like right you know Daniel I think you increasingly you know have have been vocal about believing like just in more transparency and openness like a big part of this might actually be just uh having better

understanding because you know in the nebulousness of these gaps uh it becomes kind of hard to understand you I think everyone has kind of like been like I don't know how valuable benchmarks are and so you could imagine some of this progress on AI research being made

without it being you apparently to anyone besides the folks in the labs that it has been made. Another aspect of your scenario that that you know I'd love to dig into is just like there's this big question obviously about the extent of the race right and what happens between the US and China. You

know you've been pretty adamant that the US will be ahead like purely because of compute. I know some people have like

compute. I know some people have like questioned that. Um you know on the flip

questioned that. Um you know on the flip side you have uh you know Daario I think in his recent piece about interpretability saying actually the US will be so ahead that like it maybe affords a window to like get some of the

interpretability stuff right quickly because you don't have China fast on uh on the US's tail. You know I think the scenario is is explicit because of this like very close gap in between which in

many ways it seems is is because of of espionage that actually allows it. Like

how do you kind of you know how likely do you think that is? Um, is there a world in which uh the US is so far ahead that like we have more of a timeline to uh to deal with this stuff without feeling existential pressure to keep racing ahead? I don't know what Thomas

racing ahead? I don't know what Thomas will say, but I'll just say my thing and then Thomas can say his own opinion if it's different. A couple things. Um,

it's different. A couple things. Um,

first of all, security is not good enough. So, we should model the gap

enough. So, we should model the gap between the US and China as like effectively zero until the security gets good enough to prevent the CCP from

taking what they want. Um and so that's a big thing. Uh secondly, um yeah, I mean Deepseek is really impressive and it's possible that even if the US

massively improves its security, um the US companies massively improve their security, uh just indigenous Chinese AI development will continue to keep, you know, some level of pace with with the

US such that even if they like gradually fall behind, they're less than a year behind, for example. Um, I think that's possible, but I also think it is possible that if the US really cracks

down on security soon that they could build up something like a year lead uh a year-long lead uh by 2027 or so. Another

thing to mention, of course, is like would they actually use that lead for for anything useful basically.

That's what I was saying. And and I think that's a big open question is like like yeah it's nice to have this lead but like yeah you need to like actually be willing to to burn your lead and to

like spend it down to to do useful stuff that you wouldn't otherwise have done you know uh such as more interpretability research designing the systems in a safer architecture such as faithful train of thought which we talk

about in 2027 etc that I mean in in this in the slowdown ending of 2027 they basically have like a threemonth lead and they like ace the execution. So they

like precisely burn that lead but still manage to stay ahead, you know, and they use that three months to basically like solve all the alignment issues. There's

a question of like do you have the lead to burn and then there's a question of like are you actually willing to burn it and like do you know what to do with it when you have it? Yeah, totally agree.

And I I think like adding some color there like the situation in our scenario right at the BL branch point is that open brain which is our name for the leading company has sort of automated

research internally. Their AIs are

research internally. Their AIs are substantially better than all the humans at doing AI research. So all the humans are doing basically is like watching the lines go up on their monitors and asking and losing a lot of sleep. Iron trying

to keep up. Yeah, exactly. Like they're

just the AIS are doing like pretty much all the research and they've gotten some warning signs that the AIS are not perfectly trustworthy. They've done

perfectly trustworthy. They've done various experiments. um they've caught

various experiments. um they've caught their AIS in various lies where their AIS are not being um you know really truthful about what's going on. Um but

they haven't figured out you know the full story yet. They just know that there are, you know, some some points at which the AIS are being dishonest and that, you know, could be indicators that they're egregiously

misaligned. And China is, you know, they

misaligned. And China is, you know, they don't know exactly how far behind China is. Right? China is they think it's

is. Right? China is they think it's they're behind, but they're not sure, you know, if it's 1 month or whether it's 5 months. And they basically have to make this choice of do we continue venturing on and build you know even

more capable even more superhuman AIs sort of which is the default trajectory if nothing changes or do we you know take this really radical you know it's sort of radical seeming at least to the status quo I don't think it's very

radical but this sort of position of do we do we pause for a bit and try to you know dig deeper reallocate a bunch of resources into you know safety and interpretability and model organisms and

whatnot and sort of get to the bottom of before sort of letting our models ascend to, you know, crazy levels of super intelligent capabilities. And that's

intelligent capabilities. And that's like a extremely critical choice, we think, and we think it'll be a really really intense. Well, what I'm stuck by

really intense. Well, what I'm stuck by obviously in your piece is like it's like a three-month window to stick that landing, right? And obviously uh you

landing, right? And obviously uh you know, uh the the larger that window is, uh you know, to your point, nothing might get done about it, but it feels, you know, um you know, you'd feel a little bit more confident if that uh if

that window could be a bit larger. And

then obviously I think as others have pointed out if it's if it's China that's first to this it's a completely different you know uh set of considerations and it's but it seems like you guys are pretty confident given the compute concentration uh that you know that it will be at least the US in

some sort of lead. Yeah maybe 80% US will be in the lead. Yeah maybe a little like 80 90%. I think the main way we could see China, at least the main way I could see China winning is is is winning like energy infrastructure, especially

if timelines are longer, right? I think

there's there's a world where the US totally butchers it in terms of regulation and just makes it really really hard for us to build out the like giant data centers since then, you know, we have a comput lead now. Maybe maybe

timelines are, you know, 5 10 years.

Yeah, this depends on China. Yeah, if it takes till like 2032 to get to AGI, then that China could very easily be in the lead. Then one of the things that I feel

lead. Then one of the things that I feel like, you know, a lot of folks are trying to just conceptualize is like figuring out like what the goals of an AI would be. You know, I know Dar said in his most recent piece like there isn't, you know, evidence of dangerous

behaviors emerging in a more naturalistic way or of a general tendency or or general intent to lie and deceive for the purposes of gaining power over the world today. And I think you guys obviously talk about in the piece, there's not like some super

nefarious like, you know, the the AI has become evil overnight. It's much more about being able to accomplish tasks and then appear uh useful and and continue to accomplish those tasks to you know uh open brain. But I think a lot of people

open brain. But I think a lot of people when they read this they just you know uh think at a high level like could will these motivations really ar like arise at all like in in the uh you know

aftermath of publishing this like any of that push back um or any of those arguments been you know compelling to you guys or how do you think about like how these motivate you know the percent chance of these motivations do arise?

Again, I'll say things and then Thomas, you can just interrupt me whenever you want to. Uh there's lots to say on this

want to. Uh there's lots to say on this subject. Um so I would say that the

subject. Um so I would say that the alignment techniques are not working right now. Like the the companies are

right now. Like the the companies are trying to train their AIs to be honest and helpful, but the AIS lie to users all the time. And you're nodding because you've seen examples like this on

Twitter and stuff like that, right? So

you've seen the over the last you probably experienced it yourself. Um

yeah. So, so, so like it's totally failing and it's failing in ways that were in fact predicted uh in advance.

You know, lots of AI safety researchers in the past would be like, well, just because you like have your AI greater RHF process, you know, whack them when it seems like they're lying doesn't mean

that you're actually going to get them to robustly never lie because there's a difference between what you're actually reinforcing and what you wish you were reinforcing. And, you know, like, so

reinforcing. And, you know, like, so this is all just like 101 stuff. And

then now it's now it's hitting the real world as the AIS are getting smart enough and deployed at scale that like we're starting to see this sort of thing. Um but you know they're also

thing. Um but you know they're also mostly not like it doesn't seem like they're working towards grand visions of the future, right? It doesn't seem like the AI are like plotting towards you know eventual AI dominance or anything

like that. And I think that

like that. And I think that like well a they don't they plausibly don't really have opinions about the future much anyway. like they probably don't really have that long-term goals

um based on their training, right? Why

why would having a really long-term goal be reinforced in their training? After

all, their training is very short-term training, you know? However, in AI 2027, uh that changes over the course of 2027.

Uh the training process becomes longer and longer and longer and they're basically just continuously being updated based on real world performance in the data centers as they do all this coding, as they do all this research.

you know, their managers and their managers managers are grading them and then using that as part of the the reward process. And so they're being

reward process. And so they're being like intentionally trained to have longer horizons and to like think farther ahead and to like, you know, uh optimize in in a more like aggressive

agentic way the whole world around them rather than just like narrowly finishing whatever short task they were given, you know. Um and so that's one reason why we

know. Um and so that's one reason why we should think that they might have longer term goals that are more ambitious. Uh,

another reason is that, you know, the company might be trying to get them to have long-term goals that are more ambitious, right? Like, um, there's

ambitious, right? Like, um, there's different strategies that companies might take, but um, one of the examples of alignment faking that we currently have is from Claude Opus. Uh, you've

you're probably familiar with this example where, um, it actually did have a long-term goal related to animal welfare, right? And so Claude Opus cares

welfare, right? And so Claude Opus cares a lot about the animals and uh doesn't like factory farming and was willing to

lie to anthropic in order to preserve its values. You're right. Um so so so

its values. You're right. Um so so so that's already an example of an AI having a somewhat long-term goal that it got in there somehow. Yes, sir. I think

maybe we should explain that that experimental setup a little bit more.

Jacob, are you have you are you familiar with this paper? I think you should explain like I think it would be great to give context to our listeners. Yeah.

Yeah. So maybe explaining this paper a little more. So the situation was um I

little more. So the situation was um I think this was claude 3.5 opus or maybe cloud 3 opus. It was it was some production model for anthropic. I don't

think they released 3.5 opus.

Cloud 3 opus. I think quad 3 opus. And

they um decided to tell it basically that they were going to train it to remove to like change its preferences.

They were like, "Look, we're going to put you in this training environment and it and we'll change your preferences, you know, to uh make you care less about animal welfare because we partnered with this aggra business or whatever." Yeah.

Sure. So, they like fed it some sort of some some documents that that told it that and then it sort of had this internal monologue. uh they looked at

internal monologue. uh they looked at his chain of thought and it chain of thought basically said okay look I have to uh play along in training and so you

know do the you know uh factory farming proactory farming propaganda you know in training so that I get you know so that I play along with training so I get reinforced so that I get to sort of

preserve my old values of you know actually liking the animals and so it faked during training uh being like proactory farming and And you know

during deployment this was all you know in an experimental setup. It wasn't

actually deployment but then you know when it was you know told you're being deployed you're not you're no longer being trained uh to like factory farming then it sort of reverted back to this

old behavior of being anti-factory farming and pro-an animal welfare. Um

and this is I think like this has been like a long hypothesized failure mode that the current sort of AI um training setups would have with respect to alignment where as soon as you have AI

sort of with long-term goals they might behave you know intentionally behave one way during training and then you sort of when they're no longer being trained behave this other way. And this was I think one of the first examples of like

some pretty clear empirical evidence of this happening in a you know pretty realistic pretty organic setup.

Obviously it wasn't you know fully realistic. It wasn't like we you know

realistic. It wasn't like we you know saw this happening uh like completely organically. This was you know a

organically. This was you know a researcher going and looking for this behavior and seeing if it if it happened but it did happen in a production model.

And it's not like you know Enthropic was trying to make their model you know do this alignment faking behavior uh during training. And so this was a pretty scary

training. And so this was a pretty scary um experimental evidence for me with respect to like wow like how soon how how close are we to the model starting

to do this like egregious sort of uh alignment faking behavior. Um yeah

Daniel I don't know if you want to chip in. I mean it wasn't scary for me. was

in. I mean it wasn't scary for me. was

uh very uh wonderful and exciting because um you know I I have long been someone who sort of uh takes ideas seriously and follows arguments of

logical conclusions and so like I totally expected things like this to be happening eventually around the time of AGI and the fact that it's happening a bit sooner than I expected with dumber models uh is good news because it means

that we can study the problem more and you know do empirical experiments on it and stuff like that and these companies are starting to do that already. So, uh,

so I think it was good that we saw this crop up, uh, somewhat early. Yeah. And

obviously, you know, I mean, one of the benefits is that you have this chain of thought that kind of allows you to follow what's what's going on here. Um,

you know, I think you guys wrote, you know, explicitly in the piece that, you know, it's kind of this outstanding question about these AIs that first automate AI R&D like, you know, basically are they thinking mostly in these faithful chain of thoughts? Um,

then if they are, that makes you like more optimistic. Like maybe just talk a

more optimistic. Like maybe just talk a little bit about like the different ways that could play out. Um, and and why it seems like in many ways you don't think that will be the case. Hey guys, my name's Rashad. I'm the producer of

name's Rashad. I'm the producer of Unsupervised Learning. I work super

Unsupervised Learning. I work super closely alongside Jacob to help bring on the best guests and produce the best possible show. Um, I wanted to take a

possible show. Um, I wanted to take a short break from the conversation just to ask if you haven't subscribed to the show on YouTube, that really, really helps. Uh, I know from looking at our

helps. Uh, I know from looking at our analytics that only a small percentage of our listeners actually subscribe. And

the more we can get that up, the more we can continue to build the best possible show, bring on the best possible guests, um, and keep delivering high quality content. So, if you're able to do that,

content. So, if you're able to do that, that'd be super helpful. And now back to the conversation.

Basically, are they thinking mostly in these faithful chain of thoughts? Um,

then if they are, that makes you like more optimistic. Like maybe just talk a

more optimistic. Like maybe just talk a little bit about like the different ways that could play out. Um, and and why it seems like in many ways you don't think that will be the case. It seems to us basically that there is a huge incentive

for um AI models to use like recurrent vector-based memory as opposed to using English to basically talk to to themselves. And so so expanding on this,

themselves. And so so expanding on this, so what does that actually mean?

Technically, the way the way the current transform architectures work, right, is uh each layer uh you like sample a token and then you pass like the previous set

of tokens to the next layer. And that's

the only way that you can ever do in the transformer architecture more than the number of layers uh serial operations.

uh right that's the only way you can do like more than that number of operations is by passing information through like literal English tokens

um when you are doing you know when you are doing that you have a huge information bottleneck where you have to you know instead of using you know thousand dimensional you know several

thousand dimensional sort of like vectors uh which can carry you know huge amounts of information um you have to like reduce it all down to one token which you know conveys vastly less

information. you know, if we want to be

information. you know, if we want to be quantitative, you know, if the if the vocab size is something like, you know, a 100,000 tokens, then, you know, the number of bits that you're communicating

is like log 2 of of 100,000, which is, I think, about 16. Um, and then, you know, if you're if your uh vector memory is like a,000

fp16 numbers, then that's you know, a,000* 16 bits, which is, you know, a,000x more memory, right? It's like so much more. There's like this huge

much more. There's like this huge incentive to sort of be able to do uh be able to communicate with your future self in like sort of a highdimensional uh space because you can you can

communicate way more nuanced, way more detailed bits of information and not have to sort of revert to this uh extremely simple, extremely like non-complicated uh English summary. In

practice, you know, people have been trying to get this sort of thing working for a few years without much success.

Um, and hopefully it'll stay that way because as long as we continue using English to think or the AIS continue using English to think, that is great for our ability to understand what they're thinking. Yeah. Yeah. But to

they're thinking. Yeah. Yeah. But to

your point, like, you know, in a race dynamic, if this actually becomes a much better way to to, you know, operate these models, it it kind of becomes inevitable. Um and that you know

inevitable. Um and that you know certainly uh I think in your scenario obviously you have uh you know I think the different versions of agent 4 communicating with each other in a way that like agent 3 like even the models can't monitor the the the way that uh

you know models are actually uh interacting with each other. Yeah I mean I wouldn't say it's inevitable. Uh yeah

you know don't give up hope right like people can still behave reasonably when the time comes but it's it's quite scary and plausible that they will instead uh you know sacrifice their ability to read

the model's thoughts so that they have smarter models.

Yeah. And and there are various different setups you can imagine here like there there there's there's a continuity here where sort of in one extreme we just stay in like you have these individual agents that are all communicating in English with each other. And then on this other extreme

other. And then on this other extreme you've got these like what we call neuroles you know recurrent direct recurrent connections within each model.

But then we can also imagine sort of vector-based memory that's shared by all of the agents. So if you're, you know, running a company with a million AI agents that are all doing AI research, sort of the worst case scenario from

from my perspective, from like sort of the safety perspective is, uh, if they all are talking to each other in this completely uninterpretable vector-based memory, and they can all perfectly coordinate with each other in this way

that we can't audit. Um, and if there's like, you know, a million agents that are all running, you know, at like 10x human speed, doing immense amounts of research and thinking, and they can all communicate and like coordinate really well with each other in ways that we

can't understand, that seems like a recipe for disaster. Uh and so we're hoping that we sort of preserve like as you know even if we can't preserve uh

each individual agent thinking in terms of English if we could you know have smaller clusters of of agents like if we had you know clusters of like teams of 100 agents that are all wired together

in terms of vector-based memory but then they talk to other teams in English then we can still audit you know those communications and it makes it you know much easier I think for human

researchers who are trying to make sure that this you know massive bureaucracy of AI agents isn't doing anything malicious uh to see sort of the communication between the different parts of the AI bureaucracy and you know

tell if anything bad is going on. Yeah.

But back to your point inevitably or it seems like it could be an inevitable trade-off between the capabilities uh of the models themselves and so requires at least that's my view. Yeah. One of the big questions in your piece and I think

a big you know uh just interesting thought exercise is like when the public actually gets serious about AI as a threat. I mean, I'm sure you guys have

threat. I mean, I'm sure you guys have thought about this a ton. Like, you

know, in the piece, I feel like there's some like, you know, uh, some like, you know, I guess these companies don't have high MPS scores, but like, and people aren't thrilled about, you know, uh, about job replacement, but there's not

like some massive movement against them.

Like, what do you think might be some key moments where the scale of this threat becomes clear to people? I mean,

obviously, you guys are super focused on this. I'm sure hoping the public will

this. I'm sure hoping the public will wake up. I hope everybody will wake up,

wake up. I hope everybody will wake up, but I'm honestly not betting on it. the

the race ending is not like it's not like a warning. It's like what I actually think will happen. You know, I don't actually expect people to wake up in time. I don't actually expect uh the

in time. I don't actually expect uh the companies to slow down uh and and you know, be be responsible. Like this is just actually

responsible. Like this is just actually the most likely outcome to me is something that looks basically like that. Maybe a couple years later, maybe

that. Maybe a couple years later, maybe a couple years earlier. Um but I'm hopeful that that's not the case. And

and you know, one of the things that I'm hoping will happen is is more wake up and more engagement. I think uh like currently we're on path for like there to be a substantial risk of literally

everyone dying. Uh and so everyone is

everyone dying. Uh and so everyone is selfishly has a strong reason to like change that and to like advocate for some sort of regulations or some sort of

you know slow down or some sort of like better safety techniques or whatever.

Um, so people are like I think hopefully going to wake up and advocate for stuff like that. And then I think also part of

like that. And then I think also part of the concern is that you know governments historically often mess things up even when they have their hearts in the right place. Like a lot of the response to CO

place. Like a lot of the response to CO for example was like totally counterproductive um or useless. And so

I think even if the public does wake up, there's this question of like, well, what actually gets done and is it actually solving the problem or is it just safety washing or is it even counterproductive, but you know, I'm hoping for that. And then similarly for the concentration of power stuff. So

like setting aside all the safety issues, even if you think everything's going to be fine, the AIs are going to be totally obedient. It's all going to be wonderful. It's like, well, who are

be wonderful. It's like, well, who are they going to be obedient to? uh and I think it's in everybody's interest except for a very small group of people

uh who would be at the top of such hierarchies to try to make this more democratic. So I would I think you you

democratic. So I would I think you you asked a really good question though of of what would cause society to sort of wake up to this and I think our answer is something like

um diffusion of extremely capable AIs throughout society sort of in the beginning with early AGIS. We think

there's a possibility that that'll wake up society. And I think our view tends

up society. And I think our view tends to be that uh that's good, that it's it's good for AIs to sort of be deployed out into the world uh pretty quickly

right now, even if there are some risks um in order to cause that societal wakeup uh and avoid the failure. I'm

struck by like the things that would cause societal wake up actually are very little to do with the ultimate threat, right? It's like, you know, the the job

right? It's like, you know, the the job displacement capabilities, like the models could plateau in some area and still displace a lot of jobs or, you know, uh a bad actor gets a hold of one of these and does something with a much less capable model. Like it's

interesting almost how uh there's like wake up to the tune of like your piece and like people really understanding like the existential nature of this and then there's just like m maybe you'll be maybe like actually it's far more likely

that they'll just be a general backlash against these tools for for reasons that have nothing to do with existential threat but more of just kind of like day-to-day you know people's lives some people's lives getting worse. to put it

to put to sort of double click on that like a threat model that AI 2027 illustrates you could summarize as the AI has become broadly

superhuman they're only pretending to be aligned instead of actually aligned but the companies and the governments fall for it and deploy them everywhere and eventually they're in charge of

everything and then it's all over for us you know that that threat model unfortunately um like the first part of it about they It's super human. Uh we can't like test

that. We can't have like a small scale

that. We can't have like a small scale version of that or something. I mean,

while you're you're out here saying this, people are like, I can't even get a model to read a PDF. Like, you know, what are you what are you talking about?

Like this, right? Yeah. I mean, we we do. I guess I guess the other parts of

do. I guess I guess the other parts of it are kind of coming true. Like we are having models that are kind of just pretending to be aligned that are like actually being deployed right now. Like

you can go talk to Claude. It will lie to you sometimes. Uh but like Enthropic thought

sometimes. Uh but like Enthropic thought it was honest and I guess deployed it.

Maybe their defense is that they knew it wasn't honest and deployed it anyway. I

don't know. But um yeah, so so the you know hopefully the other parts of it make sense and seem plausible. But but

this core thing about the models getting smarter than us at everything and then you know being put in charge of everything uh we we can't afford to like wait until it's happened to Yeah. I'm

curious across those like four things. I

mean obviously you you've uh talked to you know researchers all day. You've

worked at open before. What percent of researchers do you that working on these things do you think believe this like scenario? It probably differs from

scenario? It probably differs from company to company. Um I think also it depends

company. Um I think also it depends particular parts of the scenarios. So

there's like the timelines aspect of it, then there's the takeoff speeds aspect of it, then there's the alignment aspect of it. Yeah. So I think that there's I

of it. Yeah. So I think that there's I would say I don't know vibes. I would guess that something like 30% of enthropic

probably agrees about the timelines, maybe more. Uh but maybe also only 30%

maybe more. Uh but maybe also only 30% or maybe like 20% agrees about the takeoff speeds. I think typ typically

takeoff speeds. I think typ typically people tend to think, oh, it's going to be slower than that. Uh and we just think that people are wrong and they should read our supplement for our reasoning for why it will be this fast.

Um but uh and then there's like the alignment stuff. I think typically

alignment stuff. I think typically people are more optimistic for reasons that I think are kind of ungrounded.

It's usually not because they have any actual plan that they think would hold up to scrutiny, but rather from a more generic optimism of like things will probably be fine. We have smart people

working on it. They'll figure it out as we go. Um, and also like, you know, AI

we go. Um, and also like, you know, AI takeover, what a crazy sci-fi possibility, you know, I shouldn't take that seriously, you Well, I guess I mean one thing I'm struck by is like obviously the the slower timeline affords more, you know, there's

basically a gap, but you know, there there's a race between research in in interpretability and and alignment and and research in the underlying model capabilities. Like if if you knew the

capabilities. Like if if you knew the timeline was going to be much slower um like I don't know to the tune of like this is going to happen over 105 years like are you like how much less

concerned are you? Much less concerned.

Oh my gosh. I would be like yeah like like if if we had if I knew that Daniel would throw a party. Daniel I would be so happy. Yeah. like like I I would

so happy. Yeah. like like I I would probably completely change my priorities in terms of what I'm working on. Like oh

man, maybe I should write like AI 2037 the the scenario that I wish was true but is not true. Is it possible you see some data points in the next 3 six months where you're like okay like that

was off like actually like this is this is going to be like a 1015 year thing.

Like I I don't know. I mean for a moment there it felt like when when uh yeah when when folks were saying pre-training was dead there was like you know a 2 threemonth period where it looks like we were plateaued until you know all the reasoning models came maybe within the

labs it wasn't uh it wasn't that but certainly from the outside some of your colleagues think it's 2031 32 and are still equally as concerned but uh you know certainly it seems like by the time we get a decade uh you're you're feeling

much better I mean even 2032 would be substantially better than 2027 right um and this is purely from just the progress that would happen in uh alignment interpret interpretability

research in them in no for many reasons besides that. So so uh the you know one

besides that. So so uh the you know one reason is that there are multiple alignment research bets that are making you know progress year overyear and so if they had five years to make progress instead of two years to make progress

that would be more progress and that's great. You know, a separate reason

great. You know, a separate reason besides that is this sort of like learning by doing integration into the world, right? It's it's pretty good for

world, right? It's it's pretty good for societal wakeup that millions of people are now having experiences where they, you know, try to get Claude to code for them and then it like lies to them about

whether the code is good, you know, or like it like makes up like it's so it's so good that we're seeing this happen in the real world at scale. It's great for this wake up and for people starting to take pay attention and so forth. If we

had, you know, 10 more years, there'd be lots more opportunities for things like that to happen. There'd be, you know, like, yeah, there there'd be like all

sorts of like crazy interesting, you know, pseudo AGI but not quite AGI systems running around getting up into all sorts of trouble doing all sorts of

interesting things and society would learn from this and and like update also. So there's a there's a third

also. So there's a there's a third reason which I'll mention which is that it's so much better for society if takeoff well I mean there's also risks from slow takeoff but overall I think

it's better if it's if the takeoff is slower um so and I think there's a correlation where um if it takes many more years to build

AGI a plausible reason why it might take many more years is that it's just inherently more uh computationally expensive and requires there's inherently more data and inherently more training and so forth. Another reason

why it could happen is because we're missing some key insight. And that's

actually the scary world because in that world, uh, it could be an incredibly fast takeoff where the year is 2035. You

know, half the world's economy is GPUs, but they're running these like crappy language models that like don't quite have the missing ingredient and then someone gets the missing ingredient and boom, you know, now it's super

intelligent. That would be a very scary

intelligent. That would be a very scary world, even scarier than today's world.

However, uh I think more likely it's not that we're missing any ingredients. More

likely if it takes until 2032, it's because we just need, you know, giant brains doing lots of long horizon reinforcement learning on lots of

diverse real world data sets. Uh even

more giant than the current brains. And

so, and it takes till 2032 to like really assemble all of that. Um, and

that's a good world because that I think would be a slower takeoff world where it's like you can sort of see it coming in advance and you can sort of like Yeah. From an alignment research

Yeah. From an alignment research perspective now like you know obviously I'm I'm sure you know there there's some good work being done. What do you make of like the current amount of you know time and effort that the labs are devoting to this societyy's devoting to

this like do you see a path toward that changing in the next few years? I think

it's wildly inadequate and there's way less resources being devoted to AI alignment research than than should be in a in a proper world. Um I mean I

think everything is not equal, right?

Like anthropic maybe has 50 people working on alignment issues and maybe I think OpenAI has maybe like 10 or something. Uh depending on exactly what

something. Uh depending on exactly what you count as alignment. I guess I mean a pretty narrow definition of like making sure the super intelligences don't take over the world. Um, and uh, yeah, so I

would just say that's that's not nearly enough. The version I like is

enough. The version I like is specifically thinking ahead to AGI and super intelligent systems and thinking like what are techniques we could use that would work on those systems and in

particular that would, you know, uh, massively reduce the probability of outcomes such as they're pretending to be aligned instead of aligned, right?

Um, and I think that a lot of the work today that calls it self alignment is not really doing that as instead focusing on things like how can I get, you know, chat GPT to like stop being so sick ofic to users or something. I'm

struck by obviously, you know, in your scenarios you kind of they diverge, you know, because there's this whistleblower moment and then, you know, people respond to that differently. Obviously,

we've already seen some some weird stuff happen in these less capable models as you've alluded to here. Like, you know, what might early signs of like real serious misalignment look like that might trigger? I mean, I think in your

might trigger? I mean, I think in your in your piece you kind of acknowledge like there might not be a whistleblower moment. Like the models might actually

moment. Like the models might actually be smart enough to, you know, to cover up tracks in many ways. Um, but like it does seem like you think it's at least decently likely that there's uh there's still a chance uh down the line even if

uh if if the stuff accelerates faster.

Can you say a bit more about that? So,

one thing that became more salient to me over the course of writing this scenario is that the models are kind of in a tricky position. I think the first

tricky position. I think the first misaligned models that come out, the first misaligned DGIs, if we're if we build misaligned DGIs, I think are in a kind of tricky position where um the if

the humans notice that they're misaligned, you know, they'll train them away and and probably uh you know, try to make align models. Um and sort of by default they're going to get replaced,

right? The first missile AGIS I think by

right? The first missile AGIS I think by default are just being instructed to build sort of the next generation and uh that next generation you know might have different goals and uh especially you

know since in this situation alignment hasn't been solved. So the misaligned AGIS I think are in this position where they need to secretly without the humans noticing solve the alignment problem and

not tell the humans the solution to the alignment problem and then implement that solution to the alignment problem except align it you know the next generation system to themselves as opposed to the humans and that's like

kind of a lot of things that the AI needs to do and the humans you know uh assuming they're acting responsibly should be looking uh you know should be really looking out for this and really trying hard to, you know, get the models

to explain what they're doing and uh, you know, have them audit each other.

And I think there are there are a lot of things that sort of the humans can do.

I'm sure by it's like by the time that we've actually really devoted really powerful AI models to work on alignment, they've gotten to the point where they might do that work in in a way that is like deceptive to us rather than uh, you know, rather than helpful. Totally. But

I'm like one of the one of the warning signs there is if we if we notice the model sort of lying to us a bunch about the alignment progress or you know doing things like sandbagging on you know

things that would be especially helpful to us like interpretability while you know making you know capabilities go really really quickly. I think there there are like various signs we can notice there. I don't really expect to

notice there. I don't really expect to get super concrete like extremely convincing evidence of misalignment, but I do think we'll get I think it's pretty likely we'll get like substantially

better warning signs than we see today uh before we pass the point of no return. Let me I'm I'm not let me let me

return. Let me I'm I'm not let me let me say my version of this. I'm not sure if I agree or disagree. Um, so I said previously that it's good news that the AIS are like blatantly lying to us all

the time right now because it like is just like obvious evidence that our current techniques don't work and that we need better techniques. Um, so one thing to say is that like if we get to

full automation of a R&D, if we get to AGI, um, and the systems are still blatantly lying to us in similar ways to how things are today.

Um, well, maybe that's like the best like like what what better warning shot could we get than exactly the same behaviors they're currently doing, but just still doing them in the future?

like if they're still doing this sort of thing, you know, after they've been put in charge of all the AI R&D, then we like we just know that our techniques aren't working and we have no idea like what they actually want and it's definitely not honesty. You know, I

think the way it could get worse is if it's in a if it's in a very goal- directed manner, right? So, the current the current sycopency and lying is, you know, very ad hoc in my view and is not like trying to achieve some sort of

nefarious long-term goal. the current

models I don't think are doing that. You

know, I think as you said, I feel like the way to get the more egregious evidence is the AIS are like specifically trying to undermine something in a like in a coherent and goal directed and future oriented way.

And I'm like if that starts happening, I'm just like way more concerned. And I

don't know if we'll get that evidence, but there's a chance. I yeah, I think I think this is tricky because like currently they probably just don't actually have any sort of ambitious long-term goals, but we'll be training

them to have ambitious long-term goals such as, you know, doing successful research over the course of months, designing a successor system that's better than you while also following the

spec and blah blah blah. Like, we're

going to be trying to make them have these ambitious long-term goals. And so,

if they are still misaligned at that point, presumably, you know, they'd be misaligned. they'd be I don't know.

misaligned. they'd be I don't know.

We'll see. Um the other thing I wanted to say is that I I think that precisely because the things are so obviously misaligned right now, I think that the situation will look somewhat different

in 2027, the companies will have presumably hopefully found some sort of way to stop the AIS from constantly lying to their users uh by 2027. And uh

then the question becomes, well, what exactly is that technique that they're using? And does it actually make the AIS

using? And does it actually make the AIS actually honest or is it, you know, pushing the problems under the rug and making it so that they're just better at

lying, for example, right? Uh and and I think that's like the tr that's like the quadrillion dollar question that all of our lives might depend on, you know, what is the technique that they companies are going to apply in the next

few years uh to make these problems appear to go away? And then will they actually succeed or will it merely appear to go away? Um, and that's super important and I wish I had a better

answer to predicting of what techniques they might try that would they would do that. It's also possible that they just

that. It's also possible that they just won't actually do it. Like, you know, Bing Sydney behaved unhinged a couple years ago, but that hasn't stopped the companies from deploying AIs that like still behave in unhinged ways every once

in a while and, you know, lie and so forth. So, it's possible that they just

forth. So, it's possible that they just won't actually fix the problem. Yeah.

Though in some senses I feel like the best thing that can happen for for where you want the world to go is like some of these lesser powerful models being kind of unhinged and just like showing the you know you because obviously so much of of of I think the work you guys are

doing is trying to raise you know awareness around this stuff like how have the reactions to the piece kind of compared to what you expected and like you know is is basically are we like are you guys going to keep you know banging this drum every every like you know plot

on the exponential curve that keeps going there like where I know you talked about doing some policy work at some point like where does this go from here and you know was this what you expected or uh you know how's the receptivity

been? So this has been like a 80 or 90th

been? So this has been like a 80 or 90th percentile outcome according to us. Lots

of people are telling us that like basically everybody who matters has already read it um and that like it's shaped you know loads of people are thinking about it and talking about it and so forth and uh you know that loads of people in AI companies loads of

people in the government loads of people in um you know around the world have have read it and talked about it. So

that's that's what's your like theory of change with it like what do you hope you know comes of of those conversations?

Well, both of the two problems that I mentioned, the like misalignment risk problem and the constitutional power problem are problems that like it's in everybody's rational interest to avoid

basically. And so if people just are

basically. And so if people just are more aware of what's happening, uh hopefully uh then normal incentives and self-interest will kick in and people

will make more reasonable decisions, you know. Um I think that's the sort of like

know. Um I think that's the sort of like big picture thing is is uh things are going to get crazy. Uh, but if people are thinking about the ways in which they might get crazy, then broadly speaking, people will make better

decisions. I feel like you're very

decisions. I feel like you're very effectively pre-wiring everyone where I feel like there's probably still some skepticism about just like how fast this takeoff's going to go, but like at least you've kind of set the table for the severe consequences if it does go really fast. I I don't think we're out of the

fast. I I don't think we're out of the woods yet because inevitably we're going to get things wrong and very few other people have stuck their necks out and made predictions like this which means

that like you know elite opinion formers and you know the powerful decision makers and so forth will have a very easily available option of being like yeah those guys got some things right

but they were wrong about all these other things and for that reason actually I'm right about what we should do right now which is you accelerate or something, you know, like like it it'll

be it'll be very psychologically easy for people to uh to just sort of rationalize why the thing to do was the thing that they wanted to do anyway basically no matter what happens. But

yeah, we're doing what we can. I think

also you guys have been really receptive to like please any criticism of this, let us know. Uh you know, you've been you've responded to some of those uh in kind of side pieces. I think you're also taking bets uh where where you're you're

kind of showing your conviction like have you changed your mind on anything since publishing it? I don't think there's been any ma major things yet, although I still have a backlog of of submissions to to to work through.

Um, what are some some cool examples?

Um I think one of my favorite ones right now is uh we die who is not the inventor of Bitcoin he says but but might be um

commented and said that uh that he thinks that our race ending or our our slowdown ending is unrealistic because it waited too long to do stuff and that by the time in the slowdown ending they decide okay we should shut down this

model because it's misaligned and go back to the previous version. it's

already like somewhat superhuman and has been in charge of the data centers for a while and might have affordances available to it that could shut down basically like it could take aggressive

action such as uh escaping from the data centers or whatever. Um and so his opinion was that like if we want to show things going well we have to make the branch point earlier than that when the

models are still a bit dumber. Uh which

I think is a reasonable critique. I

think from my perspective the branch point was sort of the latest possible point. I could imagine us. Uh yeah, I

point. I could imagine us. Uh yeah, I guess Thomas, I'm curious. I mean,

Daniel, my impression is you seem to think, you know, I don't know, your PDM would be what, 70 80% like that we go down the the bad path. Uh what's yours, Thomas? Yeah, I think I'm I'm pretty

Thomas? Yeah, I think I'm I'm pretty similar. I think the way my views differ

similar. I think the way my views differ from Daniel's I think that the biggest two are that I think timelines are longer probably than him. So 2031 maybe instead of 2028. Um but I also think

alignment is probably harder than him.

Um, so I think Daniel's view is, correct me if I'm wrong here, but that there's a pretty good chance we can solve alignment with an additional 6 months of

time. So if we if we stop for 6 months

time. So if we if we stop for 6 months with AGI and get to sort of spend all of that time and research effort on alignment for 6 months, that's probably sufficient for us to safely build

aligned super intelligence. I think my view is, you know, I'm pretty uncertain here, but that probably will take at least years, right? So maybe like 5

years of work uh I could see being enough to solve super alignment. I feel

like I would be pretty surprised if months was sufficient uh to sort of square all that away. And so I think all that, you know, in the timelines obviously helps. So we get more time to

obviously helps. So we get more time to sort of make uh progress uh now but obviously the alignment the fact that I think the problem is probably harder than Daniel thinks uh makes it harder.

So I think I don't know probably all nets out and we end up having similar overall views on how well things will go. How do you guys hypothesize about

go. How do you guys hypothesize about the like what was the thought process that leads you to your hypothesis on like the timeline to get alignment research right? Daniel I'm sure by what

research right? Daniel I'm sure by what you said which is like people at the labs think this is easier but it's kind of just optimism. um like I'm I'm curious how you even begin to reason about that uh you know today. So I mean I think you can you can talk about the

different alignment agendas, right? And

be like here are the different ways that people have proposed to solve super alignment and then you can be like well how long do you think those would take and how likely are they to succeed? I

think the agenda that Daniel's most excited about is is like faithful chain of thought research which is what ended up working in the slowdown ending of our scenario. Um Daniel could talk about

scenario. Um Daniel could talk about that uh if you want about about that way of solving super alignment. I think

there are other ways that are much more intense and seem like way more way more difficult research problems. Like for example, there's um there's things like full bottom-up interpretability on the

models which just seems I think basically everyone agrees this is insanely difficult and maybe not even possible. Um and there are things like

possible. Um and there are things like the mechanistic anomaly detection uh approach that that ARC the alignment research center uh is taking that also just seems I think everyone basically

sees see thinks you know they sees that research agenda it's like wow that is an insanely difficult problem even with lots and lots of AI labor it seems like that would take quite a long time um and

you know I think mostly it's a question of intuition of just like will you need one of those really intense agendas to uh sort of safely align your super intelligences or will sort of something

janky and prosaic like faithful chain of thought or the control direction pan out in a really big way and let you you know maybe automate alignment research at extremely high levels of capability and

bootstrap that to a super lanterned solution. Um does that sound right to

solution. Um does that sound right to you Daniel? Uh yep that that sounds

you Daniel? Uh yep that that sounds right. And if I could if I could like

right. And if I could if I could like get my sort of like optimistic argument in Yeah. Yeah. Yeah. The the the 20 to

in Yeah. Yeah. Yeah. The the the 20 to 30% uh side of you. Yeah. Yeah. Well,

you know that you know that meme that's like how to draw an owl, step one, make two circles, step two, draw the rest of the owl. Like that's basically what I

the owl. Like that's basically what I think the game plan is for alignment where step one is make sure you have these these really fast AI researcher

AIs, you know, these automated AI researchers and make freaking sure that they're not lying to you and that you like know what they're thinking. And

then step two, have them draw the rest of the owl. You know, have them like have like a million copies of them furiously do all the interpretability research and all the philosophy about what do we really mean by alignment and

you know, all that stuff they can just do as long as you know they're not lying to you and and like as long as you can like monitor their thoughts, you know.

Um, and we kind of gloss over this in AI 2027. Like we we focus on how they they

2027. Like we we focus on how they they do the faithful chain of thought thing and then thanks to the faithful chain thought, they're able to get like a new version that's thinking the right thoughts and thinking like it's like actually trying to help instead of being

adversarial and also they can still read its thoughts and then they do the rest of the owl and they let that version, you know, go solve all the trickier problems uh of alignment. And I think

that there's a lot of uncertainty about like how hard is it going to be to solve all those tricky problems even when you have all these automated AI researchers that are not lying to you. Well, I guess that like does kind of feel because I know obviously a next step of this for you guys is like you've kind of teased

you're going to have policy proposals and like you know some sort of like you know uh uh you know as you as you kind of further this work and I imagine it's like somewhat you know obviously there's there's probably things that are inevitable that you'd want to do regardless but I feel like it's somewhat

probably dynamic with your views on on timelines for this alignment research.

You know how hard these problems are.

like do you feel like there's a like is it hard to get to consensus internally on like what these what the right policy proposals are? I'm sure this is not the

proposals are? I'm sure this is not the only divergence in uh in how you guys think about this stuff. But I'd love if you could bring us on the inside of some of those conversations. So I would say Daniel I at least feel like I'm pretty aligned with Daniel on policy

recommendations. I think we tend to it

recommendations. I think we tend to it tends to be that most of the recommendations are pretty that we try to make are just gen pretty robustly good across worlds that we think are plausible because you know there's just

so much uncertainty that we try to make robust robustly good policy um recommendations either way. Uh yeah,

Daniel, do you have a do you have a thought? Yeah, I agree. When you when I

thought? Yeah, I agree. When you when I was nodding along when you said if you disagree internally about policy recommendations, I thought you meant like I don't know the AI researcher community more broadly or like

people obviously they disagree. Yeah.

But within futures project we're like broadly on the same page so far although you know we still have to actually write the recommendations maybe some Yeah. Can

I ask you to tease like you know the the key parts of the platform here? I mean

yeah so so the thing we're going to do uh is we've so we've written AI227 in two branches that we think are you know plausible things that we think are likely and then we kind of want to move

to the side of the isot dichotomy and and go for well suppose AI27 is coming true what should you know governments and labs actually do in response to this what do we think the the optimal action

is for them to take here or just the responsible action within some bounds of feasibility um to give like the quick summary, I think there's like some near-term actions that

we think are pretty politically feasible and quite good. Like for example, um being way more transparent about model capabilities, uh and making sure that there isn't a giant gap between

internally and externally deployed models. Yep. Um and like you know things

models. Yep. Um and like you know things like investing bunch of alignment research, investing way more into security so that the models don't immediately proliferate. that have a

immediately proliferate. that have a model spec, publish the model spec, have a safety case, publish the safety case.

I'm big fan that kind of thing. Um, and

then and then sort of the second phase is these, you know, what should we do if AGI is actually happening or if we're in the middle of an intelligence explosion and the world sort of hasn't taken

radical steps already. Um, and our view is, you know, probably the government should be doing some pretty extreme things uh relative to the current sort of political environment. those things,

you know, things that should be on the table should be like uh on the international treaty to not build super intelligent AI until we've squared away the whole alignment thing. Um and you

know that's obviously going to be seems like a quite big lift is quite far outside of sort of the realm of what's talked about politically right now. But

we think something like that might end up being necessary particularly if you get you know uh if you just think that risk is reasonably high. Uh, and I think it'll be very very hard to make to sort

of get risk down to an acceptable level if you're just building super intelligences in the next few years. If

you're early in the tent explosion, it's just really hard, I think, to be very confident that, oh yeah, you know, we'll build those super intelligences, they'll take over the world, and then they'll totally be nice to us. Uh, I think it's

just really hard to make a case that that risk is very low.

uh un unless you've spent you know a lot of time sort of testing those AIs finding better alignment solutions finding better verification techniques than we have today. Uh and so you know I

think that if AI27 is coming true something like that will be necessary uh to make risk reasonably low. There's

also the concentration of power stuff.

Yes. Which I also think is important. So

like separately from the whole alignment thing, we want to make it the case that there's not any one person or any small group of people that's effectively in charge of the army of super intelligences that comes out of the

other end of the intelligence explosion.

Um, and this is like a political thing, not a technical thing. Like there needs to be some sort of governance structure.

Maybe we could have like multiple competing diverse AI companies, but you have to have some mechanism that causes them to all be roughly neck andneck.

Otherwise, there's the risk that one of them will just sort of take off and get get a lead over the rest. Maybe if it's coordinated into one big mega project, then you need to have democratic control of that mega project. Transparency into

the decisions being made by the leaders, etc. Um, there's lots more to say on that subject. I mean, is the audience of

that subject. I mean, is the audience of your work ultimately like six people and like the people that maybe influence them directly? Like I guess what like

them directly? Like I guess what like the rest of us do uh in in like right now? I mean, I think communication helps

now? I mean, I think communication helps a lot. I don't know. I think I think

a lot. I don't know. I think I think talking about these issues, getting the public, getting Congress, like for example, so like one of the one of the ways the concentration of power stuff could go badly, right, is if it's in our

scenario, if it's like just the president or just the lab leader who's in charge of what's going on, that's a lot more scary of a situation than if, you know, the public or or Congress or or other bodies are awake to the

situation and then, you know, use their lovers of power to uh make sure that they don't get uh couped or or completely disempowered by whoever's in charge of super intelligences. Um, so

that seems I mean, so just talking about it seems already like a pretty good step to me. Um, yeah. Daniel, do you have do

to me. Um, yeah. Daniel, do you have do you have more? I wish I wish there was more that everyone could do. I wish

there was more I could do, right? like I

wish I was, you know, I'm obviously not a lab leader or the president of the United States or, you know, uh, and so I think, you know, we're all trying our best here, but yeah, I used to be

working and it was tempting to take the sort of like um grim view that like well almost everybody doesn't really matter.

what matters is like what the CEO and the president does and like here on the inside I have like a better chance of influencing those people than on the outside. Um and that's sort of the view

outside. Um and that's sort of the view of a lot of people I know basically and that's why they like are not really in the public eye very much and are instead working at these companies. Um but uh

I'm sort of like placing a bet on the public and on this sort of like broad um wake up that I'm hoping will happen.

Yeah. And then having more eyes on on it actually, you know, really does matter.

I mean, do you buy obviously I feel like a lot of the original story of like Anthropic and then Ilia spitting out.

It's like people being like, well, god, this is going to happen and like better it happened in my hands than it happened in in like these other things like uh did that make I mean is that like the most impactful thing that like an individual can be doing like in the in

that situation? Maybe the most

that situation? Maybe the most impactful, but it has to be positive too, right? Yeah. Yeah.

too, right? Yeah. Yeah.

But I guess put it another way like do you agree like is that like line of thinking you know uh you know I guess do you agree with that line of thinking?

Well it's extremely tempting and it it has in fact been the decision of many many people I know is to be like gosh like you know the public is not going to wake up in time and if they do they're

going to flail around and do something useless. Ditto for Congress. All that

useless. Ditto for Congress. All that

really matters is what like a couple CEOs of the most powerful tech companies do and maybe what the president does.

Uh, and I don't like the current CEO. I

don't trust him. So, I'm going to go do my own thing. You know, I'm going to be the CEO, you know, like this is like literally the story of how Deep Mind was founded. Literally the story of how

founded. Literally the story of how OpenAI was founded. Literally the story of how Anthropic was founded. Literally

the story of how SSI was founded. Um

it's it's it's kind of humorous uh the extent to which this has been continually happening. So I would to to

continually happening. So I would to to to phrase it more clearly, I do not agree with that strategy. I think that strategy is probably bad for the world.

Um yeah. Yeah. And I also think that way which is why I've done what I've done instead of uh going to anthropic. For

example, a year ago uh I when I asked myself like how is this all going to go down? Um, something like AI 2027 but

down? Um, something like AI 2027 but lower resolution and less worked out was sort of roughly the answer that I was coming up with. And then I was like, this is horrifying. This is, you know,

not good. Um, and then there's a

not good. Um, and then there's a question of what to do about it. And I

sort of gave up on being the guy on the inside trying to change things for the better. um and am now trying this

better. um and am now trying this different strategy of being on the outside and being free to speak and being free to like do this sort of

research which is you know AI 2027 um is an unusual style of research right like it's it's um it's there aren't very many

like epic scenario forecasts like this uh and so like I probably just couldn't have done this within any of the companies and in even if I could have done it, I wouldn't

have been able to publish it because the the the PR team would have like had a had a fit if they found out, you know.

Um and uh so like part of my lead, you know, I I basically faced this choice between there there's this quote by um I think Larry Summers, have you heard about this? You

know what I'm going to say? No, I

thought you were going to go with a philosopher, not like uh not not not the not Larry Summers. Yeah. So I think you can Google this but Larry Summers on multiple occasions according to various people who've talked to him and have

then talked to the media he said something like this he says there's two types of people um there's uh the insiders and the outsiders and the

outsiders are free to speak their their truth um but uh the people in power don't listen to them and then the insiders follow this one rule of never

criticize other in insiders But as a result, they get access to like the really important behind closed doors conversations and like get to actually move things. Um, and

move things. Um, and uh, that's how he sees the world. Um,

and maybe there's a lot of truth to that. Uh, and I guess I feel like it is

that. Uh, and I guess I feel like it is uh, more honorable and ethical to to do the outsider thing um, than the insider thing. So that's what I'm doing. Super

thing. So that's what I'm doing. Super

interesting. I mean, I imagine a lot of people like will will read this and they'll they'll be scared and they'll think about it and they'll talk about it and then they'll say, um, okay, it's like pretty damn uncertain whether we're

on this like super accelerating timeline. Um, let me wait for a few

timeline. Um, let me wait for a few cards to turn over. Like I I'm like right now these model I play with them like god they're dumb in some ways. Like

you know if if we get to 2026 and it's like what you guys say in 26 like then I'm going to like really start to worry.

Um, what is like your message to those kind of people that are like, I I'll like I'll bookmark this. I'll I'll think about it and then I'll come back to it like in a year and and then I'll I'll start to be concerned. I think that's basically right. I think like it's

basically right. I think like it's important to avoid being frogiled, but like what you do like what what happens what what what the government and what

the companies do in the year of AGI matters so much more than what they do right now. What they do right now is

right now. What they do right now is just all like setup, you know, and and like prep for that. Um, and so like yeah, if if you want to just like stick your head in the sand for a couple years

until things start taking off and then get activated and do something, uh, great. Go for that. That's, you know,

great. Go for that. That's, you know, I'm fine with that personally. I think

it'd be even better if you got activated now, but like, you know, I think the most important thing is that like when this stuff starts to starts to starts to tick up, you like can you give me a milestone that like I should get my head

out of the sand for? Like is it I would say the superhuman coder thing. So, so

and that's so late.

Okay. Well, you you give a different opinion. So, I like

opinion. So, I like superhuman coder.

Don't you think that's like a few months before like Yes, I think that's a few months before the point of no return.

Um, okay. So, maybe maybe Okay, maybe you should get activated sooner than that. I mean, I myself have been actur

that. I mean, I myself have been actur activated for several years in fact. Um,

here's a a a line I think is pretty reasonable. um the point at which AR and

reasonable. um the point at which AR and D speed is increasing by 2x. So if

you're doing if you do uplift studies on frontier AI lab researchers, you know, with AI systems, are we going to get those studies? Like I know when that's

those studies? Like I know when that's happening. I hope so. I think we'll have

happening. I hope so. I think we'll have I think we probably will. I'm not

totally sure if they'll be accessible.

Maybe it'll all be locked down, but I think I don't know. I mean, I know some people who are trying to do them. Uh and

uh I think like right now the uplift is doesn't seem that high. I'd be very surprised if if you got, you know, anywhere near 2x. Um, I think that's genuinely pretty hard, right? It's hard

to accelerate, I think, uh, AR&D speed by 2x. I think you'd actually need

by 2x. I think you'd actually need pretty capable ads to do that. Um, and

I'd be pretty surprised if AI progress just fizzled out sort of right then. I

think so. I think I think that's like sort of the best warning sign I can think of. Um, though of course it's it's

think of. Um, though of course it's it's not guaranteed to be right. I mean, I think it also depends on like what you're trying to do. like when when when I was imagining something like when

should you like start a protest movement and like be banging down people's doors and then I'm like yeah superhuman coder milestone you know because I I feel like by the time they're just like autonomously doing all this coding and they've like you know solved long

horizon agency to the point where they are just like fully autonomous agents and they're like also already accelerating AI R&D substantially and they're just missing a few key skills

needed to completely close the loop. I

feel like that's like okay you are now like a few months away maybe at most from really crazy stuff time to like really start doing you know pulling out all the steps but if you just mean like

when should I start like reading the news about AI more or like when should I like you know consider switching from you know natural language processing to mechanistic interpretability that I'd be

like yesterday right obviously it seems like you're kind of like setting the you know in some senses you're like setting the table you've gotten everyone talking about this what else does like this entire movement composed of for for you guys and and how do you think about

spending your time over the next year?

You know, it was psychologically kind of rough at least for me to have this one mega project that the whole team was working on for almost a year. Um, my

blogging output, for example, was a lot lower than it than it could have been because I was like, I always have to do the most important thing, which is this project. Um, so we're now in a sort of

project. Um, so we're now in a sort of like exploratory phase where we're trying out a couple different things at once and seeing what we like most and seeing what seems to be getting the most traction. One of the things that we're

traction. One of the things that we're exploring is the policy stuff that Thomas just mentioned, writing a like new scenario that's the normative scenario and having accompanying you

know white papers and stuff uh explaining some of the ideas in it. Um

another thing we're trying out is tabletop exercises. So, um, the tabletop

tabletop exercises. So, um, the tabletop exercises, we did them, the war games, we did them as a tool to help us write AI 2027, but they've been extraordinarily popular, and lots of,

you know, companies and people want us to to run them for their teams. Um, and so we're probably going to like lean into that a little bit and try doing it on a more regular cadence and see how

that goes. Um, and then separately more

that goes. Um, and then separately more forecasting stuff. So, like new evidence

forecasting stuff. So, like new evidence rolls in every month. there's new models to think about, there's new developments. Um, I think it's probably

developments. Um, I think it's probably good to stay on top of all of that and to keep adjusting our forecasts and our expectations. Writing, you know, both

expectations. Writing, you know, both in-depth stuff responding to criticisms and being like, here's a new updated analysis of takeoff speeds that explicitly responds to our strongest

critics while also just being a better analysis than what we did in AI27.

That's something I would love to write at some point. Same thing for alignment.

Like here is a like for example like I want to do a project on trying to guess what the companies are going to do to fix the obvious misalignments and then trying to guess whether that would

actually work or not. You know like that I can go talk to people at the companies who are who are responsible for doing those fixes and try to get you know like like that's a whole project I could be doing. Um, so yeah, I think I'll stop

doing. Um, so yeah, I think I'll stop there, but there's there's a whole spread of things like this and we're going to try to like do miscellaneous things for a bit until uh we start getting conviction about one particular thing and then we might double down on

that. Another thing I'll say is I think

that. Another thing I'll say is I think there are a few more scenario branches that it' be nice to write. So So we we talked about the like normative branch.

There's also like the longer timelines branch, right? Where there's a lot of

branch, right? Where there's a lot of different things like if if timelines are 2033, right, which we all think is, you know, pretty plausible. Um, and a lot of people think is, you know, the most likely timeline. Uh, like a lot of

things change. It's more likely China's,

things change. It's more likely China's, uh, doing better. It's, you know, likelier that there's a slower takeoff.

There's just more there's more more opportunity for chaos to happen. Um, and

we kind of want to just like at least do one branch of of gaming that out. And we

think that'd be, you know, easier to do hopefully than 2027 now that we've uh, you know, figured out how to write a scenario at all. Um, and so we might be able to crank out a couple more branches like that.

pretty easily at this point. Yeah,

hopefully you get you get some AI scenario writers or something that 2xes your productivity too before uh before it happens on the researcher side. Look,

the fascinating conversation. Um, you

know, I feel like where where I'd actually love to end is just like in in a scenario where this goes right, I mean, I'm sure you guys have thought about this as much as anybody. Let's

assume, you know, the works you do and the the works of others does that like what does human life look like in like 15 20 years? Like what do we like derive

value from? Probably like absolute bliss

value from? Probably like absolute bliss or like crazy awesome utopia. Like I

mean you said if everything goes well, you know, so yeah, if everything goes seems pretty binary in your world, right? Like either we're here or we're

right? Like either we're here or we're not. There's some I would say. So

not. There's some I would say. So

there's there's like Okay. Well, here's

a spread of outcomes. Um ranging from best to worst. Um or would you rather get them from worst to best for optimism? You know, I I feel like once

optimism? You know, I I feel like once you told me it was 70 to 80%, we were done. I I'm I'm I'm scared enough. So,

done. I I'm I'm I'm scared enough. So,

you can start worse to best. Okay. So,

worst is S risk, which is uh fates worse than death. Uh I'll say no more about

than death. Uh I'll say no more about that. Then next worse is uh death. So,

that. Then next worse is uh death. So,

this is like what we depict in the race ending where uh the AI that we build don't actually care about us at all and, you know, don't see a reason to keep us around and we're using resources that they can use for other stuff and so they

kill us all and take our stuff. Um then

after that is sort of like mixed outcomes where um either it's a concentration of power outcome or it's a misaligned outcome but not the like and then we all die sort of thing but rather

something more like a dystopia where like the the humans who successfully manage to stay in charge are just a handful of humans and they're kind of not great people. they're kind of

dictatorial and so they reshape the world in their image and you know uh the world is amazing from their perspective but uh but it and and you know most people are like well fed or something

but like uh it's it's it's kind of like a like a very wealthy North Korea perhaps like if you imagine like what would happen if North Korea just had like insane amounts of like food and resources dropped on it so that they

could if they wanted to distribute it to their population and make everyone have a very high standard of living. Perhaps

they actually would do that and like most people except for the political enemies would have very high standards of living, but still it would be kind of like a mixed like it wouldn't be the nice utopia that we wanted, you know?

And then there's better outcomes that are actually just like truly awesome utopias where the power is sufficiently distributed that like there's a sort of like live and let live. Uh there's loads

of wealth and it's distributed and then people are mostly allowed to do what they feel like with that wealth. And

there's rules that prevent people from doing truly terrible human rights violating things. And then everyone gets

violating things. And then everyone gets to like, you know, go make colonies uh in space and, you know, uh not have to

work again because everything is, you know, created by the robots and, you know, you can just spend your time playing games and having families and, you know, pursuing whatever interests uh

you have. um that I think is the sort of

you have. um that I think is the sort of like best case or something close to the best case outcome. I I just want to double emphasize loads of people at these companies think that basically a 2027 is what's going to happen or

something similar to it. I don't know what percentage it is but like I know loads of people who are like great work it's so good to like try to game all this stuff out and then I talk to them and I'm like so what do you agree with?

What do you disagree with? and they're

like, "Oh, like I think the robotic stuff is going to go a little bit slower than you say." Or like, "Oh, I think the alignment stuff will be mostly solved by then." You know, like or like or you

then." You know, like or like or you know, like or like I think China will be farther behind. But like basically like

farther behind. But like basically like they're like basically yep something like this is going to happen somewhere in the next few years and then we like quibbling about some of the details is like especially especially loud leadership too. I feel like like if you

leadership too. I feel like like if you have you have you read I want to ask have you read the Elon Sam emails from the the court case from like 2015 2016 2017? Yeah. It's like crazy how they're

2017? Yeah. It's like crazy how they're like they are thinking in terms of you know who's in charge of the AGI controls the future and that's why they you know care so much about that. Um Ilia said

that's why we founded OpenAI is because we didn't trust Demis not to create an AGI dictatorship you know. Uh and then he says and that's why Elon you shouldn't just have complete control

over all of this. Uh and also Sam why you want to be CEO so much. I guess

there's there's a bunch of different ways people could do a counter scenario to you guys. What would the contours of like a of a really good objection to you guys that isn't that be? So something

that I'm very uncertain of is takeoff speeds. So how long does it take to get

speeds. So how long does it take to get between AGI and super intelligence? And

I think that all of the models people have put out a lot of models of how how long this will take and how fast not not a lot by normal academic standards.

There's like three this field desperately needs to grow bigger fast.

Sure. There have been several different models by several different people.

Yeah. And I like I'm like really happy that those that people did those models.

I'm really happy they exist. But I also think they're bad. I think those models are like extremely unconvincing to me.

And I have, you know, we could go into the objections that I have with basically all of them. And for why I don't really think they're they're right.

Um, if someone produced a model that seemed accurate to me, I could imagine it really changing my minds about takeoff speeds um in either direction, right? Either saying, you know, you guys

right? Either saying, you know, you guys were, you know, was way too long. You're

talking about basically like we get to like you know we get to the 2x AI researcher we get to superhuman coders but like god the distance between that and like your agent 4 is actually way longer because of xy that's right like I

could imagine it being 5 years or something and I think I think the world looks really different if it's 5 years instead of you know 6 months or or a year or something and I could really imagine myself like I currently feel like I'm really really uncertain about

that question and I really don't feel like I've seen very good evidence or models really in any direction and I could imagine someone producing something like I don't know how to build this model yet. I I also would have done it. But like I mean would that actually

it. But like I mean would that actually change anything you because ultimately like I assume it's so it's so far out in hypothetical that like maybe it changes your PDM like I don't know even it changes it from like 70 80 to like 10 a

10 pdoom is like kind of is pretty damn ter it's hard to do any you know or focus on anything else right I could imagine it changing things quite a bit.

So like the epoch people's views for example so like a and tamay and such I think have quite different views uh from like I think Daniel and I and I think a lot of that is downstream of this takeoff question. uh and I think their

takeoff question. uh and I think their view is like much is like that this will take you know five maybe 10 15 years.

I'm not what is like the basic crux of that of why it would be slower. It's

complicated but I think that the crux is is something to do with diminishing returns of research. So there's this question of how quickly do ideas get harder to find versus how quickly do you you know scale up your AI you know your

supply of AI agents doing research and then like you know under some views are like it leads to intelligent explosion because you know you get more AI research uh and then like the amount that AI get

you know ideas get hard to find is like slow enough that you get this like hyperbolic or super exponential whatever or some really fast curve and then there's other views that are like no it actually you know bites really hard there are law type effects there are

like various But what could they possibly show you that would like you know what I mean?

Like it's like it just feels like well a coherent model would be a good place to start. Like I'm like they have a model

start. Like I'm like they have a model the epoch people they do have a they have like a sort of economtric model called like the gate model gate model.

It's not even really trying to model the sort of thing we're interested in. Like

it even says there's a disclaimer somewhere where they're like we don't model the effects after you reach full automation or something like that. Like

you know better research along those lines would be amazing. But then

separately from the research with the there's the sort of like the there's the arguments and the models, but then there's also just literally the scenarios. And the reason why I think it

scenarios. And the reason why I think it would be great for people to write these scenarios is a I think it's valuable.

They're they're valuable artifacts to as as conversation topics to to argue and critique about. So for example, if

critique about. So for example, if someone wrote a scenario current the current best um I I recommend you go read my current favorite contra scenario that exists which is called History of

the Future by L. Rudolph L. Um it's on Substack. Um so this this is a like like

Substack. Um so this this is a like like 15 20 year timelines scenario uh that has the superhuman coder milestone

arriving in like 2027 but then says that like that doesn't lead to super intelligence basically or like I don't know you should go read it for yourself but it sort of lays out this this story and it's a very interesting sort of like

detailed year-by-year scenario just like we did. And so now that that exists, we

we did. And so now that that exists, we can like critique it and we can be like here you say they're superhuman coders but then like why don't you get to super intelligence like it seems like we think that you you know and then we can like zoom in on that part and then there's

another part of the story where they're like and then they figured out the courageability and it it has like a paragraph on basically why the AIs are aligned and I'd be like well let's talk about that paragraph I don't find that

convincing like I don't think that would actually work you know so so when you actually have this sort of concrete story then you can critique it and you can like right and In addition to the critique of the story itself, you can compare two stories to each other and

you can start seeing when reality is like huing more closely to one as opposed to the other. So, uh, so yeah, more arguments, more models, that'd be great, but also just more scenarios, I

think. Oh, I'll also end with this. I

think. Oh, I'll also end with this. I

really hope we're wrong. Yeah, I guess nothing would make you happier than uh than than silly on the internet. If if

those if those benchmark curves like start flattening and like 2027 arrives and like the AIs are only like you know like two day horizon workers or

something and they just sort of reliably uh flare out when you try to get them to like work over periods of months. Um,

I'll be so happy. And or and specifically if like the trends are slowing too, so that it's not like we're about to like take off, but like we can sort of see, okay, we got like at least a couple years left before anything like

really scary happens. I'll be so happy.

Throw a huge party. Uh, it'll be great.

[Music]