Co-Intelligence: Wharton Prof. Ethan Mollick on Collaboration Between Humans & AI | Technovation 860

the power of AI is in the edges the people using it for their job you need to figure a way to incentivize them and capture that you need to give them access to Frontier models and I know that's hard because it's not how it departments work but the damage is going

to happen one way or another everyone's already using AI everywhere in your organization they're do not tell you about it welcome to technovation I'm your host Peter High my guest today is Ethan mik

Ethan is an associate professor at the Wharton School of the University of Pennsylvania where he studies and teaches Innovation and Entrepreneurship and examines the effects of artificial intelligence on work and education in

addition to his research and teaching Ethan also leads Wharton interactive an effort to democratize education using games simulations and AI prior to his time in Academia Ethan co-founded a

startup company and is an adviser to entrepreneurs and other Executives as well Ethan's latest book is co-intelligence living and working with AI a book that I've had a chance to read

recently and recommend I first got to know his work through his blog which can be found at one useful thing. org which

has nearly 120,000 followers Ethan welcome to technovation it's great to speak with you today it's great to be here um thanks for having me oh it's it's a pleasure well let's begin I I

would love from a context setting perspective you teach entrepreneurship and you are a former entrepreneur I wonder if you could take a moment and talk a bit about your entrepreneurial

Journey uh and your journey back into Academia following that uh sure so I uh became an entrepreneur during the first internet boom in the late '90s early 2000s I had a brilliant college roommate

of mine who was a technological genius and knew in Industry really well and he got me to join him in a startup and we we created the world's first pay wall so I feel bad about that still I feel like that's what I'm trying to earn off by

being an Academia so sorry we created the pay wall um but um you know went and I was the sales marketing outward facing person tried to convince 500y old companies that literally had the

original Gutenberg Press to that they should go online and sell their stuff um and it was pretty successful but we made every mistake possible right so every hiring mistake you can imagine

every management mistake you could imagine Equity all kinds of like we successful to play ourselves so I thought I got to figure out how to do this right decid to go get an MBA and MIT realized nobody actually knew much about how to make entrepreneurship

successful and then decided to get a PhD and study this stuff so that's sort of where I am and um you know why I was interested and some as somebody who uh has been an entrepreneur but also helps

entrepreneur I wonder are there innate qualities that are important for successful entrepreneurs in from your perspective you teach it so this is the you know you take care of the you know teaching them the aspects that are not

already in them perhaps but are are there certain you know attributes that you tend to find among people who are successful entrepreneurs so one of the things I think that's most important about doing a study of Entrepreneurship

is almost everyone's instincts aren't that helpful in entrepreneurship like a lot of people's gut feel people tell stories about entrepreneurship that do aren't matched by the data we actually know a lot lot of things now about Founders and what makes them successful

and so for example entrepreneurial personality deeply studied it turns out that there are personality traits that predict you wanting to be an entrepreneur but there are no personality traits that predict

entrepreneurial success durably right outside of things that generally make you successful in life being uh you know less neotic more you know conscientious those things are very helpful but you know the number one personal TR for

predicting entry into entrepreneurship is overconfidence so and so you end up getting lot of people who look like they're going to see but they're just the people trying more but all those factors don't actually predict things so one thing that actually worries me a lot

is when people say oh you don't got the entrepreneurial type that's just not backed up by the data similarly like age is you know people assume you have to be a Young Person's game the average age for found in the United States is 42 the

average age for a Founder who raises venture capital is 42 the average age for a Founder who achieves hyper growth is between you 45 and 59 like so a lot of people's mental image of what an

entrepreneur is aren't matched by reality well one statistic that I believe is uh broadly known and thought about is the concentration of them in places like silic valy in the US more

generally speaking you and I were uh talking recently about how pen itself where you where you teach I guess it was two two years ago uh uh hen alums

received more Venture Capital than the combination of France and Germany combined um pretty remarkable uh what is it about the US that uh what are some of the secret ingredients of this this

country that have made uh entrepreneurship so so prominent so it's a hotly debated issue and and the answer

seems to be a mix of um of um you know there's always some kind of luck and continuing on Direction universities um I mean if you the brew that sort of

caused Silicon Valley right universities immigration of of Highly talented people right and then free market kind of capital that is pretty fairly

meritocratic um and a um and sort of a a medium level of Regulation seems to help but also tax incentives turn out to matter a lot in these kind of circumstances uh you know the way interest is treated and Returns on

investment are treated so there's lots of little things but it's it's tended to be a you know very good education system and you know University system um High

incentives from you know and returns from entrepreneurship and high skilled immigration yeah very interesting and I asked you the question earlier in the the balance between nature and nurture the nature aspects of this from a

nurture perspective at least in the context of a class like yours what are some of the forgive me this is I'm sure you could take the the entire hour that we have to talk about this but what are

some of the top items that that need to be taught uh or or most useful to learn in a setting like yours for for those who aspire to to start companies so if we just start from where the data

supports things there's three or four sort of key things that are definitely teachable so one key thing that's teach that makes a big difference is teaching people to do disciplined experimentation in entrepreneurship so that is

hypothesis you know you hypothesize something relevant about your business and you do testing and you either pivot or continue that turns out to be really important and people who disciplined hypothesis based testing have higher

revenues by like exponentially higher than people who don't engage in that process the second major thing is there turns out there's a lot of stuff that you that actually makes a difference

around um you know how you pitch how you explain how you raise financing there's complication there a lot of people view like raising VC as a prize and it's not it's a method of getting somewhere with advantages and disadvantages so that's

another set of stuff the third is management matters so being hiring is hugely important and a lot of people don't know how to hire or how to make hiring work building an organizational structure to so you could scale past 20

people where your individual level of like energy is no longer enough to kind of cover an entire organization that turns out to be very important and there's a whole bunch of stuff around uh mentoring networking that all seem to be

a big difference in teaching people skills and that helps a lot and and forgive me I should have asked this question perhaps a little bit earlier but how do you define an entrepreneur um is it as broad as anyone who starts an Enterprise is a is a you know a baker

who starts his or her own Bakery and entrepreneur in your mind are there further qualifications how do you define it so it's one of the you know you're asking an academic and academic question so we love that but um the and the

academics answered every question like that is it depends right so um I focus in my classes and my book and uh previous book not this AI book on uh high growth entrepreneurship so that is

people who are looking to start a business or an organization and grow and scale rapidly but yes there small business owners as entrepreneurship there Entre entrepreneurship inside companies is sometimes defined as part

of Entrepreneurship um launching a nonprofit that you know was intend to scale and grow is entrepreneurship so it's a very flexible term yeah and how many people who take your class end up starting companies I have not tracked

the exact number but um and the interesting thing is it's not the first class that does it right so we actually do track entrepreneurship rates overall when people graduate from Wharton maybe 1 to two% of people upon graduation

launch companies but by the time your three or four jobs in 20 to 30% have launched a company so a lot of it going back to that age thing a lot of this is It's hard to launch a like you can launch a consumer company aimed at college kids when you're in college but

you're not going to really understand the Enterprise needs of a Fortune 500 company but work at an Enterprise at Fortune 500 company for a couple years like there's 50 things I know this company needs and I keep telling them

they need it and they won't do it so I'm just going to leave and I by the way I know the best people in my companies so I'm going to take them with me and I actually have some resources now so I can raise funding and that's a much more

common form of Entrepreneurship I I also want to ask you the I'm sure you have many of your students approach you during your office hours or after class and ask you some version of what would

you recommend I go do uh and given the fact that you're professor of Entrepreneurship I imagine at least in some cases it's going start something but but uh I wonder across your your tenure uh as a professor how that's

evolved and certainly I would imagine artificial intelligence will get into a lot more specifics as this conversation goes on M must further qualify your response as well but you know how is how

has your advice changed across your time as a as a professor of Entrepreneurship I mean I think there's a divide bright dividing line in in about AI that changes everything in very large ways so

I don't think past advice versus current advice is a useful comparison because I don't think the world's the same shape as it was now uh before so um you know my advice has always been you um you can

wait to be an entrepreneur there's nothing wrong with it people have student loans they have things they they haven't found the perfect idea they don't need to force it there's not a limited window of opportunity which I guess is back to the age thing right um

at the same time you need to if you want to be an entrepreneur you need to be open to opportunity there's no angel of Entrepreneurship that will Des send and say my child now it is time launch a company so you have to be aware and open for opportunity you have to be making

sure you don't get golden handcuffs a of pen students go work in finance and it's very easy to have a lifestyle that makes it so it's very hard to launch a startup company so you have to be do watchful waiting preparing um and that's that's a

very different um mindset than a lot of people have I I was fascinated in in preparation and just referring to some of your your past writings uh that you've noted that one of the things that

has changed is AI in some ways can serve as a co-founder of a business um and I wonder if you could explain that there's been a lot at least historically written about the importance of co-founders and

having like the example you had that you had you were the salesperson to somebody who is much more technical than you were as you launched a firm uh soon after undergrad um talk a little bit about

your your vision of of at least in some cases AI being the co-founder of a business first of all let's just talk about co-founders for a moment because I have some research on this great another cult-like thing is there needs to be

co-founders and why commentator famously for a long time wouldn't allow you to join the company if uh if you didn't have a co-founder and so there's a famous about Dropbox uh the founder getting into why combinator and this

turns out to be true uh and being told upon joining like no no you need a co-founder you need to have one by this afternoon and he went to the MIT cafeteria and just found someone random and said want to co-found Dropbox with me and that's how he found his

co-founder um that is not a great way to go so there's a sort of belief that co-founders really matter I have some studies with my colleague Jason Greenberg and we find that actually single Founders outperform co-founders

in many cases because you don't have all the arguments like a large part of the reason why Ventures fall apart is co-founder conflict and you don't have that argument if you don't have a co-founder the danger is you don't have someone else to bounce ideas off of it

do work but turns out what you could do as a Founder is hire someone and give them a larger Equity stake and they're founding team member but they're not the founder of the company right so they might still have a 15% equity and

they're the CTO of the company but they don't have to be the founder of the company as well so that's an important part and I think that's incredibly important in uh when think about AI because AI can fill a lot of the functions you wanted a co-founder to do

so if you if you're a technical co-founder it does really writes really good sales emails it does projections it actually gives you pretty good advice there's a amazing study um by some uh some great researchers out of Kenya

where they found if you were a top entrepreneur in Kenya small business owner and you got advice from gp4 you had 20% higher profits afterwards and people who didn't get advice from gp4

like so it definitely fills a role there yeah it's really fascinating uh you've referred to yourself as historically AI adjacent uh and also as the business school representative at the AI table uh

now that AI allows non-technical people to become much more technically Savvy or at least AI takes over some of the fills some of the gaps of of those of us who are less technically Savvy uh does it

level the playing field more generally speaking between someone like yourself and maybe you're no longer a great example of this as you've gone from AI adjacent to somebody who's who can out write books about it uh but relative to

those people who are deeply technical AI focused as opposed to AI adjacent I think in some ways it's a mistake to even lump AI in with technical things

right um because the fact is large language models that they're very technical products they're using them doesn't require technical knowledge and in fact I would argue that technical

people are often are the worst prompters of AI so um I think part of this is It's a technological advancement but it's sort of like saying Excel is a technological advancement are you you know does Excel change how technically

you are sort of but also you know because you're not writing your own spreadsheets anymore but also who thinks of it that way so I think that it it's a democratizing force in a lot of ways I also noted that you've you said that

your syllabus has changed as a result of AI to ask students to do the impossible uh I wonder if you could explain that so I literally give people the assignment like if you uh like in my class you have

to do something impossible so if you can't code I need working software if you never did design work I need a working website um so I need you to be like actually fully operational in um

you know with with AI um so people who used to do I you do a prototype and now I want like that to work um and you don't have to be technical to do it so that by the way you also have to get critiques from three famous Founders

through history of each of your assignments you sh in because the number one predictor of entreprene Entry is overconfidence so if I have a you know one way to to vanage that is to have someone whispering in your ear you know

you're mortal and that's a useful thing to have the AI do so uh it really has changed how I work how we do this work fascinating I I I want to get into some of the details from the book

co-intelligence um you talk about four rules for co-intelligence and I wanted to cover each of the four with you uh the first of those is always invite artificial intelligence to the table

what do you mean by that AI has what we call a jagged Frontier that means it's good some tasks and bad at others so you can get a um so if you ask the AI to give you a 25-word summary of a page you

might get 22 words or 28 words or some other number because the AI doesn't see words the way we do it sees tokens which which are words or parts of words so like a space is part of a token um and

so the AI will might miscount the words if you ask to write a sonnet summarizing the work it'll do a great sonnet for you how do we deal with a system that could write an amazing sonnet but can't do 25

words um and so the idea is that if you use AI enough you understand what's good or bad at and that lets you navigate this Frontier it also lets you know what what difference it makes nobody knows

how well AI will be applied to you know to the author of worldclass it strategy like nobody knows that right and you can figure that out right what does it know that you don't know what do you know that it doesn't the only way to do it is

to take it to the podcast that you come to and see how it does summarizing our conversation and to also have it do prep work for the podcast and compare it to your prep work that you would do and then help it you write your next you know post or piece of information then

help with an next Consulting venture or your next speech that's how you figure out what this is good or bad for and that leads nicely then to the next uh next of your rules for co-intelligence Be the human in the loop um I think it

flows nicely what you described but do go into some detail as to how best to be the human in the loop so this is an idea for control systems that you want a person involved in working with AI um

and it's a problem because AI is pretty solid like in our studies at Boston Consultant Group we found it operating like the eth percenti of B Consultants in a lot of ways not in every Dimension but in main dimensions that's tough these are Elite Consultants right from

you know they come from place like won we were highly trained um and so you need to think about as a person what do you want to do and right now the good news is is the AI is at the eighth percentile of high performance but not

the 100th and whatever you're probably best de in the world you're probably in the top 1% 5% 10% and that's what you like to do and the AI is not going to be better than you at that at least not

right now so um what that gives you is this opportunity to do focus on what you do well and give away the stuff you don't want to do so being the human in the loop is also about how do you make AI part of your decision- making but how

do you focus on what you do best and you talked about how uh as you as you noted if an expert in a field you're likely better than than AI is today and so it is in in many ways best suited towards

taking over those things that you don't enjoy and don't do best perhaps um how do you see this continuing to advance such that it goes from 8 to to 99th um and more fully replicating the sorts of

things that we do so I mean that's the question and kind of the only question that matters I don't have answers I talk to people training these systems they don't have answers right there's dividing lines between whether we keep on an exponential Improvement curve

until we reach artificial intelligence and machine smarter than a human being or whether we don't succeed at that and you know it evens off in the near future and a lot depends on that and I don't have any easy answer but I think people

should be prepared for more upside case than they are right now yeah uh the next rule for co-intelligence is treat artificial intelligence like a person but tell it what kind of a person it is

explain that if you would you make AI do things by prompting it essentially by giving it a sentence and then it autocompletes everything else afterwards right um and people make this very hard there's there's all kinds of Tri tricks

and prompting I'm a very good prompter like you do all kinds of weird stuff but the easiest way to work with AI is just to talk to it like it's a human being treat it like a person and even though it's not a person um that's why managers

are often so good at working with AI give it instructions the way you would a person correct it the way you would a person but then also tell it what kind of person it is you're a manager at a you know you are a marketing manager at

an IT company you are a editor who have you know preference for Clear writing and you'll get better results that way you're the the fourth of the four rules for co-intelligence is assume this is

the worst AI you'll ever use um again describe that everything you're using right now is obsolete there's better things being trained so what one of the FAS things about AI is all these models are being released and they're all sort

of chat Bots they're all the interfaces all slightly broken they're not optimized for any particular job one way or another and people kind of are they think that the story might be oh maybe we need to launch a startup that makes this better for our business or

something like that the reason why that's happening is every AI lab is spending all their time building the next generation of AI and as soon as that evens off they'll go back and figure out how to commercialize it more

but they're all building new stuff so whatever you think the capability limits of AI are today that's not going to be the limits in the near future so everything you're using today is obsolete makes sense and how do you especially in light of that how do you

remain a breast or how do you advise others to remain a breast of the many tools that one could use I mean the the proliferation of them is such that you can have you know dozens and dozens of options to choose from do you personally

focus on on a handful of them and how do you add to that what are the criteria by which you you determin to add to the the list as it continues to grow so my general advice to people is is I

differentiate between tools there's lots of tools out there especially sort of image creation but lots of people creating tools and then Foundation models which most of the tools are built on Foundation models are large language

models there's a bunch of them out there anything from llama 2 to grock to chat GPT gbt 3.5 gbt 4 I would draw the line between those especially if you're an

entrepreneur or you're you know you're an executive and a um and what are called Frontier models so the frontier models are the largest models and right now there's a very strong relationship

which is the scaling law the larger your model is the smarter your model is and the effects are quite large so Bloomberg spent $10 million plus training Bloomberg GPT which was a specialized

large language model That was supposed to do Financial Pro projections they threw all the blomberg data in there they trained it up GPT 4 out of the box the same model that you have access to for free and Bing and everyone in mosan

Beek and Sri Lanka has access to for free beats Bloomberg GPT and stock predictions right gbd4 beats the best specialized models we have on um medical

advice and it beats doctors in in terms of giving advice and patients prefer it because it's more empathetic than doctor answers right so the larger models do more things and so when I talk about the

tools you should pick one of the three Frontier models right now and spend 10 hours with it and that's my number one piece of advice then later on you can decide if you want to buy do something specialized it does something cool great but like don't worry about the

proliferation of tools all that matters is the frontier models in the short term they're all kind of similar think of this they're all first year PhD students with different personalities so there's

three of them right now arguably maybe four that are Frontier models so as of the time we're talking that is gp4 which you either get access to through chat gbd Plus or through Microsoft's being

co-pilot office is kind of sweet there is uh Google's um Gemini Advanced or or their Gemini Pro 1.5 Advanced you can pay 20 bucks a month for what Pro 1.5 is

coming out soon and is you can get a preview access to and then there is Claude 3 which just came out this week um which is all from anthropic all three of those are Frontier models there

arguably a fourth called inflections uh Pi 2.5 but that is a model optimized for chat and it's very friendly and it's very compelling to speak to but it doesn't unless they do work for you and

and may I ask what how have you divided your time across the the models you've mentioned so I know gp4 are the best so I use that for most of my work but I also know the personalities so I know

Claude 3 is an excellent writer and I will often delegate writing tasks to to it um and Gemini is very good at compiling information to web search and

very good at creating plans so I'll use it for that and and speaking of writing uh how did you leverage it as part of co-intelligence your book there's 's almost no AI writing in the book that's not AI writing and you'll you know

having read it that I I have the the AI appears as kind of a character in the book um at times um but I used it extensively in writing the book because I approached it from a integrated work

we call I call a cyborg perspective so for example I've written books before you've written books and you know one of the most annoying things is like you get stuck on a sentence and then it just kind of kills your productivity for you end up going walking away you're like I

can't do it give me 20 versions of the sence of different styles ask the AI and that unlocks something uh give me five analogies to explain this concept uh read all these academic papers and summarize them for me to make sure my

summaries are correct read this in the style of a uh you know a 44y old dentist from Wisconsin and tell me what confuses you uh and I use the AI in all those different ways in the book you also

write about artificial intelligence as a person a creative co-worker tutor coach and our future these are also uh aligned with different sections of the book

itself I'd love to we've talked in some ways about several of those already but um as you think about AI as a cooworker we talked a little bit about AI as a co-founder and perhaps there are some similarities in your mind between those

but how do you think of artificial intelligence as a cooworker because the AIS work best like people and not like machines they slot very well into human

systems so you want help read if you show it a document it'll think about the document the way you think about the document roughly and so it works very well as an individual worker to use AI to get get things done because I can

experiment with it I can throw you know like I can throw stuff at it and see how well it does at it if it doesn't do well I won't delegate to it anymore so everybody basically got a uh you know an

infinite number of First Year PhD interns yeah I love that analogy uh in terms of the interns as well or the PHD student that you talked about before um how do you use AI as a tutor or how

would you suggest that others do so that is both one of the most promising issue areas and also when I give advice to at an individual level is different than what I give advice to at a policy level

so tutoring is the is the magical thing um that is kind of one of the most exciting things about AI because there's a famous paper called blooms 2 Sigma paper that's probably mostly um I don't

know if the math still works but the idea has been Pro proven before which is that oneon-one tutoring improves performance by up to two standard deviations so you from the 50th percentile 98 percentile performance

after receiving tutoring right um and tutoring is very expensive it's very hard to do and so the AI is showing real promise working as a one-on-one tutor we're not there yet now the that's

different from how You' use things individually like in a classroom I want a tutor because tutors teach you things and they teach you things by not telling you the answer if you Google something you won't remember it as well there's actually a bunch of studies on this but

and so if the AI explain something to you you won't necessarily get it better but we have gotten the AI working as a tutor where does what actually a tutor should do which is ask you questions so you know tell me how you think AI fits

into the world of you know education that's really good but maybe you want to think more about how the downside risk can you tell me what those might be great how would you apply that to what we learned before a tutor is soliciting

information correcting and like and that's what makes the AI so interesting is it can do that interactive stuff now personally if you want to learn the AI is very good at explaining things in your context so you just say you know my

name is X and I've worked in this industry and stuff before and could you explain this part of quantum mechanics to me in a way I understand and it does a very good job with that yeah very interesting and H how how

would your advice differ for somebody who's let's say in elementary school versus High School versus college versus grad school do you see I mean naturally of course there's the sophistication of

what would be entered in uh to get tutoring from uh and the the level of advancement that that the individual would have in those different settings do you see it first of all as something that's additive in each of those

settings that would encourage parents to encourage uh their their children especially uh to get involved with but can you talk a bit about how see some of those differences as somebody who studies the intersection of of U these

topics with education more generally speaking well we don't have good tools right now right so like the best tool out there in tutoring is Khan Academy's kigo um which while flawed is still the best approach out there right now to

doing tutoring with AI and like anyone can subscribe for 20 bucks a month or something I think it's 10 bucks a month like that's the tool like I subscribe my kids to that right um as a parent by the way it works really well if you try to

remember how an algebra problem worked you can have explain like no actually like explain this to me remind me why it works this way so it can get you back up the speed very well but it really is about interacting with the AI again

using as a tutor and asking questions it makes mistakes there's errors in it the real issue is like what do you not want to use it for like what do you need to learn and that's a bigger question and uh as you think about a tutor or an

educator how does that differ from another one of the areas that you focus on in the book which is um AI as a coach how do you suggest that as a potential use advisors are helpful and so there's

a lot of stuff we teach you an MBA program that basically is sort of self- coaching so for example let's take premortem so you increase the chance of

project success by 18 to 22% by having uh doing a premortem which means sitting down and saying how could our project fail right and you don't that you never

guess the real reason why it fails you're not actually doing real cause analysis but it gives the people in your team permission to talk about failure which we don't usually have and that lets you think about downside risk and

that's why it increases success right doesn't proof you against it but it let you have a conversation about what could go wrong we teach people how to do that there's some delicate things you have to do to make a premortem work well I can give you a paragraph and you could paste

that in gbd4 or Bor or whatever else you like or gemini or whatever else you like and you'll get a pretty good interactive coach that will take you through the process of doing a premortem uh you know it can read all of your transcripts from

our conversations happening right now and I could say where you know where was I funny like did I have any jokes Miss what should I I have said differently um was there a moment that was awkward how

do I deal with that help me improve myself yeah very all interesting uses and as you as you point out all of us need coaches to to some extent so the extent to which this is something that's

a an easy version of that for us to access so much the better one that's only getting better as time goes on I wanted to also ask you about again as a professor how you see all of this

changing the classroom setting there was a lot of promise and a lot of um a lot written and pontificated about as to how the mukes would change learning and in in the grand scheme of things it didn't

seem to at least so dramatically forgive me as you've been in a in a university setting more recently than I have but at least that's my sense that there's still a lot of classes that are taught as they

were taught 250 years ago today uh and I wonder what will be different this time around uh in terms of the the the changes that will be made specifically in the classroom and in the group

setting itself so this is a really good question so for those who don't know muks are you know large online CES it's uh it's massive online uh courses right

corer Udacity L learning I've got I've got a corsera courses that had a quarter million people take it like I've I've done this thing and so I've seen this from the inside in fact part of the reason why I launched W interactive

which is an interactive simulation experiences after recording some muks I'm like oh we need something very different than this because muks did something very important which was expanded who get access to education right I've had a quarter million people

take my entrepreneurship courses wonderful like you know they're not pay like they pay or not pay I mean I don't it's not an issue to me they can watch these videos for free and they have a chance to you know that like to get an

experience and for some set of Learners that is transformative if you're you know you're could be anywhere in the world now and get a decent experience of a college lecture like you know not interactive but like you know and that's

valuable and there's been millions of people who I think have had their lives improved by that but it's a very static thing right it's like buying a book or watching a videotape the mukes are not

Dynamic they're not built around the pedagogy of how we learn best and so they did do something amazing which was expanded democratized access but they haven't changed the life of my students

because lectures have always been one of the worst ways to teach right the way to teach is we know how to teach properly which is active learning you do stuff in the classroom now there's a bunch of problems with active learning one of the problems with active learning is that

it's really hard to do because you have to keep your students engaged in doing learning the second is people hate Active Learning like students do there was a great study at Harvard where um they took the intro physics class and half the class got lectures and the

other half had to do Active Learning problem sets and other stuff in class the people who did Active Learning reported learning less but they did much much better on tests why they learn less because as opposed to being a lecture

where I could talk and you can like on this podcast you could sit back and be like yeah yeah I got this right you don't have to do anything it's like the our tutoring argument the tutor forces you to realize what you don't know Active Learning forces you know what you

don't know no one likes feeling ignorant and so they hate the active learning experience right right um but that's the valuable way to teach we also had trouble implementing Active Learning because that means we have to teach

outside the classroom and that means watching like a Muk video which are not great ways to teach AI unlocks both those possibilities because it lets us do these flipped classrooms because I can have an AI tutor outside of class do

the content teaching and then it can help me create good engaging experiences inside of class so we're doing Active Learning when we're all together and doing the passive learning outside of class so I think that's a real chance

for transformation and are you seeing some of these changes already especially in a a class that's as Progressive on these topics as your yours is oh my class are 100% AI required now um so it's absolutely it's working out

amazingly and and when what do you I mean you you've explained some of your vision which presumably uh has been pulled forward given given your focus in this area I I I one thing that I I've heard you say is that people raise their

hands less um and the the way in which they're interacting with you naturally and interact with each other changes as a result of this can you get a little bit further into the details or your of your observations uh having implemented

this in in the classroom in your classroom yeah so let's talk about you know the the the downside risk is already happening right everyone who wants to cheat can cheat and there's no easy way to detect AI writing it's not a

great situation right the second thing is that um you know people are interacting a different way like you know there have already issues where people on social media and class have their phones out whatever else but now

like you can get caught up by the AI helping you catch up uh so that changes and that changes social construct class normally you pay attention you don't know something you raise your hand and then I know that if you don't know that

probably half the class doesn't know it and we could have a conversation right and it's a people can extract Knowledge from me and I can you figure out what people need to know that social contract breaks with AI on the other hand it

enables new kinds of teaching so we talked about doing a possible thing another example I have the students uh co-create cases with the AI where they have to kind of correct the ai's direction I have them build I had my

mbas do an assignment where they had to Place themselves so they had to uh for a job that they were applying for they had to automate part of that job so they could just go into the first interview

say here's my job is done with this AI um I need a raise and um you know they did it like everyone from Navy Pilots to private Equity Funds the hipop designers had figured out ways to make this work

it was very cool to watch that all happen and um and you know a few of them got jobs right afterwards as a result by the way so like it changes what you can do in classrooms and we we've been

mapping out um all kinds of ways you could do that and you you talked about how uh you know people have I I heard you speak about someone in your class who clearly was not paying attention you

were you were lecturing and they were building a business model in the class itself and by the next day they were getting funding for it and you're seeing more of these sorts of things happen as well as a as a result of some of these

changes yeah when I first introduced chat GPT to my students literally one person built a working app by the end of the class and had uh VC Scouts reach out to them the next day right so that was

like what other like so this this really enables all kinds of new changes for people it's very exciting that's really interesting I want to return to a point you made uh just a couple minutes ago that anyone can cheat clearly in a in a

setting like yours uh this is something that must be contemplated tremendously across a university as it is of course across high schools and all schools and maybe even more so in in in I don't know English classes or history classes where

um there's that much more with the written word perhaps um although correct me wrong but but anyway I what what is your perspective on this I I know there's been consternation with each Advan in technology I mean the the

calculator even as it was introduced was uh thought of as a a method of cheating and I know know there was at least some thought put to as whether or not this was an appropriate tool to have in classes this of course is much more

sophisticated as this conversation certainly reflects um now that anyone can cheat what's the what sorts of Governors if any would do you advise for academics to put on it or is it simply

look live with it let's think about new Norms associated with this as a result I mean so calculators absolutely disrupted math in the 1970s right it took a while to figure things out and what we decided

to do was you still need to learn how to do math by hand but then by sixth or seventh grade you could switch over to the calculator do more advanced math so you still have to do tests where you do math by hand without a calculator we've

solved the problem and so you're going to see mix of classes that way I would expect a lot of English composition classes to switch to in class you're going to be writing essays maybe getting feedback from a mix of AIS and humans

that outside of class you're going to learn how to write essays but there's no more take-home assignments right I expect to see that happening um you know just like we saw in math class I expect to see a lot more classes with testing

in class testing because that still works um so I think we're going to see a mix of going back to the days of blue books for some classes and others embracing AI tools the way I have and I think that's completely okay that's

interesting I I I wonder more generally speaking um what worries you as you think about advances in artificial intelligence I mean there's of course many people who are talking about I mean

even in its most dramatic form once AGI is is uh U accomplished artificial general intelligence that that there are some existential issues that may result from that in terms of competition

between us and and the AGI um but but you know as you contemplate the the realities and likelihoods of continued advancements especially at the the remarkable Pace we're currently going at what what what worries you as you look

to the say medium term we have a major technological change happening these happen rarely generation or so and it creates a lot of unpredictable effects so one way I like to think about in

scenarios right so I think a lot of people who are listening to this who haven't picked up you know haven't used AI a lot or have only used free chat gbt which is very much not a great system to

use right um those people uh don't have any idea of like they have a mental model that this is static that AI come out it's kind of like blockchain I'll get to it like there's a lot of hype it's obviously most ofly garbage I'm not

going to worry about it right now right that they're wrong this is advancing very quickly on the other hand we have the most advanced C which is Agi artificial general intelligence that doesn't always mean a super intelligent machine that competes with us but it

might mean right now it's at the you know it's doing well in test scores but you know what happens when it beats every human every test score and it's like a chess you know computer where except for your job whatever you could

do the AI will do better that's the explicit goal of open Ai and anthropic and many the AI companies to build that and they think they could do it so I think it's worth spending some time worrying about less so is the competitor

that you know species that will wipe us all out of the earth and more as like what happens when a you know a computer program can write a better book than you can right what happens if we can do better Consulting work than you can I

don't know if that's possible nobody does but they're building towards that the more likely scenario or at least the most likely scenario in the near- term is either continued linear exponential growth it's the 8% obesity Consultants

this year next year is at the 82nd 89th 98th 10 108th I have no idea and in that world right of then the future involves you focusing back to the human in Loop

focus on what you do best and the AI can be very liberating because it takes away the stuff you don't want to do for your job and I think that's where I would be focusing in the near term and what do you think about the the role the

government should play in all of this how much have you contemplated that and and uh reflected on that I mean it needs to play both positive and negative roles we obviously need to protection against the downside case we need thinking about

what happens if there's mass unemployment ultimate I don't think that's likely that's not but we don't know and even by the way if this is just changes in jobs which is likely right just like we had then J jobs of

accountant changed once spreadsheets came out right accountants didn't lose their jobs they switched to a higher-end job instead of doing the math all day but that those changes on mass if you live through the Industrial Revolution

it's pretty disruptive so there's likely to be some changes happening one way or another and and there's some downside risk using AI for you know malicious purposes the the outside risk of AGI all

things the government should be thinking about on the more but on the sort of day-to-day level we also need to think about where they're permitting stuff there's a lot of regulated Industries out there that are afraid to expert with AI how do we do that in a safe way that

protects privacy and protects you know data security and respects patients big questions but if the but if there's no regulation on it then people aren't going to use it and they'll just keep using it secretly so I think there's a

lot of open questions that they can help us address you we talked understandably about a lot of the entrepreneurial implications of this and therefore small but but but fast growing environments

for larger scaled organizations as people here represent in many cases what do you think about the sort of broader adoption of you know co-pilots and other tool sets and how would you suggest

organizations think about scaling those in their organization uh more broadly speaking so co-pilots are both a really great way to introduce yourself to

something and also a little bit of a um of a secret sabator inside your company um so I'll explain both those things as it it's like so let's first take the general principle that AI is helps build

boost productivity in almost every job we measure right and but to do it you need individuals using AI because it works best at the individual level not the group level so the thing I worry about talking to it strategy people is

that this is centralized in the IT department there's a whole bunch of problems with that right one of those is that this doesn't work like it technology coders are very bad at using this because it doesn't do what you're supposed to do it doesn't always give

you a well you know formatted Json file instead it might insult you or tell you it never could do Json file or say it's unethical what you're do like that's not how software is supposed to work differently every time and the second

thing is that because AI has meant many things over the years to it departments the other concern I worry about is that they tend to think about this as sort of amalgam machine learning style Ai and

they they're very concentrated their own data matters a lot and it's not actually clear that your own data helps you very much with a with a GPT 4 plus class AI because the P stands were pre-trained

already knows a lot of things and as we talked about before having all Bloomberg's data inside the Bloomberg GPT did not help it work better so there's this kind of obsession left over from the Big Data training days of AI

which is we need to use our data so the first solution everybody turns to is almost always a talk to our documents model that's what every it Department builds and that's a terrible model for AI because even if you have the perfect

just a little technical rag based system that's pulling back data in the right way the AI still lies and hallucinates and makes stuff up and that's not what anyone really wants from AI when you figure out how people want AI is

actually treating it like an HR problem or a training problem how does everybody the organization use Ai and then how do you get them to tell you what they've done to streamline their work and how do you incentivize them to not be afraid to share that with you and so I think it's

really important to consider it that way now co-pilots are a halfway step there but there're but the thing is you're being kind of shielded from the core of the AI Itself by the co-pilot but the co-pilot does all of the disruptive

stuff of AI without letting you do the advanced stuff so I use Microsoft Word co-pilot I can write 50 pages like in two minutes is that good or bad like many managers jobs inside organizations

producing words that's their only job and you judge by the quality of their words how how smart they are you judge by the quality of words how much effort they put in you judge by the lack of Errors of words about how much diligence

they have right their job might be ensuring the supply chain from Vietnam for monop or P you know you know um pigments but to make up a term but you know what is that person doing who's in charge of that they're writing a report

every week the state of the Vietnamese Market um and now they're just going to hit a button and create that and they're going to send that to a boss hit a button and say back good job what does that mean for organizations and that's

what I think I worry about the roll out without thinking what the meaning of this stuff is yeah yeah likewise actually do you see there's also been a lot written about you know the the number of books and movies and video

games and and and art that is that is being created will continue with greater levels of sophistication to be created do do you have a sense for uh the the

competition that that yields I mean will will will there be a premium put on those versions of those things the extent to which it can even be deciphered uh that are created by humans versus those that will be created by by

technology well we're back to the question we asked before which is how good how fast right now ai is not a compelling writer compared to a human being right I talk to Hollywood people it's not like they're the top 0.1% of

ability to write a story the AI is not close to that right now right it's funny when you know going back to the AI Sonet I actually had a Twitter one of a world expert on sonets and I had the a right of Sonet and she's like this is looks

like a good son but it's actually a bad Sonet because the whole idea of a Sonet is there's this emotional reversal in the final two lines of the Sonet that put into context and the AI doesn't do

that well right so like it right now it's producing art you know and sometimes it's great art by accident sometimes humans can guide like it so right now I think of AI like a synthesizer when synthesizers first came

out they caused all this controversy because you couldn't play piano now you could do this right so it helped boost creativity as an individual will it get good enough that it makes a compelling

movie and or video game so I don't want to watch a movie created by humans maybe if we if it keeps but like that's not the immediate threat right what you know I I know somebody in a game company who's as a single person is building a

very high quality game because the AI is making the art and help you write the code and helping write the storyline that's that's compelling yeah yeah very interesting do you back to the large

company setting uh the extent to which you continue to evaluate this what are you seeing you've started to talk a bit about like use of co-pilots and the the value or perhaps even in some cases inflated value derived from that are

there um what are you what are you seeing as some of some other um aspects of the value that larger organizations are are deriving from from use of generative AI so again every early study

we have finds performance boost whether that is analytical tasks writing tasks persuasion tasks ideation tasks programming tasks 20 to 80% performance improvements like that's very common to see also a huge leveling effect bottom

performers get leveled up these are compelling enough numbers that I think companies should be panicking even if you don't worry about any other aspect of AI if all your competitors are just getting 20 to 70% more employees and

you're not thinking about that that feels like a mistake and I think a lot of companies are kind of waiting on this like another technology when you have that large a gain if you do like an Enterprise installation of Salesforce or whatever right you know it's going to be

a four-year process you're going to spend x million dollars to make it happen you're going to pull off this many cons and do this much off the line to hopefully gain you know whatever it is 13% efficiency right we do a lot in

it for relatively small gains there is a chance for very very large gains from relatively small investment and I think that's an incredibly compelling story and I think the problem is is that there's no instruction manual there's no

one to help you you cannot hire a consulting company to do this because they don't know anything I'm sorry they just don't right apologist consults in the room I know this because I talk to open Ai and Microsoft and Google on a

regular basis and nobody knows anything they don't know what these systems could do you know open AI got 12 people you know build like it's a slight over but like they're not doing testing on the

use case for a large scale accounting company they're not doing a testing on the use case in a university classroom they're they don't know any of this stuff the fact that the AI is good as a doctor was a surprise they never

realized it would disrupt all of education so you need to be doing rapid experimentation the the biggest danger is centralizing Authority and a committee of people of seven people who

will report back to the CEO of uses for AI the CEO will set up RFP request that will go be filled in four months from a series of Consultants who again don't know anything who will then come out and do an analysis for the following eight

months to give you recommendations to build an internal facing rag system that talks to your documents don't do that right the power of AI is in the edges the people using it for their job you need to figure out a way to incentivize

them and capture that you need to give them access to Frontier models and like I know that's hard because it's not how it departments work but like the damage is going to happen way or another your student everyone's already using AI

everywhere in your organization they're do not tell you about it so Shadow it spend is going to be your biggest concern not actually internal security that way people already I could take a picture of a screen and the AI can read

it like security doesn't mean what people think it means anymore yeah great great insights all around well Ethan mik thank you so much for a very compelling uh conversation thank you also for your your great contribution through your new

book co-intelligence living and working with AI uh and and and thank you for your many ruminations through one useful thing. org something that I I personally

thing. org something that I I personally find very valuable to keep up to date on on your thinking um I really appreciate you spending time with me today thank you so

[Music] much