Steve Hsu: As far as there being consciousness in the AIs, this is another key point that the rationalists and AI theorists debate a lot.

Many people would say the worst possible outcome is some not conscious super intelligent robots kill us all, and they're the only thing left. And it's actually better that, that something conscious kills us all and continues to, you know, have its own destiny in the universe. So this question of whether machine consciousness is ultimately possible is, is a factor in many people's thinking about these futures, and it's an open question.

Scientifically, we don't know whether there's something special about our consciousness or whether we could eventually engineer that into the future AIs or not. The guy Joscha Bach, who is in the film, who's a direct descendant of Johann Sebastian Bach, he's one of the leading thinkers about consciousness as a scientific problem. You can read more of his stuff if you're interested in that.

Welcome to Manifold. Today's episode is audio from a Q&A that took place at the University of Chicago. The student AI club sponsored a screening of the movie Machine God, and we had this Q&A afterwards. Sohan Hormuzdiar is the organizer. He is studying math and physics at the University of Chicago. Hope you enjoy the episode.

Speaker 1: How did you first get acquainted with Nick Land's work specifically? Like did it come from hearing it from people in this kind of world? Did it come from outside? And then what, what has that, I guess, process been like dealing with him?

Steve Hsu: I've known Nick Land for a long time over the internet, over X where, of course, all intellectual discussion centers in this era. So Nick and I, we've, definitely exchanged ideas for many years. I want to say like more than 10 years.

I met him first in person when he came to that event in the Bay Area, which is on film. He's been adopted as a kind of patron saint or leader of the accelerationist movement. And he's, he's often referenced and, and actually highly revered by Silicon Valley, you know, the power elite in Silicon Valley.

Speaker 1: My second question is gonna be, so I know you mentioned, like, Marc Andreessen in the, the video. I've had a trouble, like, trying to situate him. I was wondering if you could offer some insight. 'Cause I've seen him both make this, like, public-facing case about techno-optimism, where it's gonna be great for everybody, and then I've seen him make these private remarks about, you know, like, I think he has one quote like like at least people have video games and opioids to keep them, like, satiated, which is more in like the permanent underclass kind of thing. Do you know where he might, like, situate himself?

Steve Hsu: Yeah, actually, uh, very perceptive of you to notice this. So publicly, he's one of the biggest accelerationists in Silicon Valley, and he actually puts his money where his mouth is 'cause Andreessen Horowitz is one of the biggest funders of startups in the AI space. And they're very, very aggressive.

Like, they're, they're investing as if they believe in the most bullish future for AI. I think all the leaders in Silicon Valley have for many years discussed things like the singularity, AGI, and even the question of whether we live in a simulation. And so all of them, when you go to meetings, if you have dinner with any of these guys, they can just jump right into these concepts.

Now, this would not be true of anyone in like Washington DC, but all the billionaires and trillionaires in Silicon Valley are very familiar with these ideas. What they think privately may not be exactly what they say publicly. I don't think there's very much in our film that would be surprising to them.

Can I get some general comments for you, from you about what was your favorite part of the film? Did you like the avatar? Yes.

Speaker 1: I thought you had a very insightful. I was talking to some of my friends earlier about some of the commentary you were making, and it was like you captured these concepts that we'd been discussing in these very eloquent, short little blurbs. So I, yeah, my favorite part of the film was your commentary.

I don't know if you have, like, some other project or writing or something, but it was very competently written.

Steve Hsu: Wow, I'm very flattered. Thank you so much. You know, we wanted to have a mix of pedagogy from an old professor like me

Yeah!and then the actual people who are players in this, in this sphere actually you know, saying it in their own words. We had the avatar as well.

Speaker 1: I was gonna say, I, what I liked is I think you, you would always frame the concept such that I, as the other people were speaking, I could know what you were trying to communicate through those clips that you were selecting. It was, it was very well done.

Steve Hsu: Yeah. That was the, that was the goal. So, you know, the goal is always just the audience's understanding and however they take it. Like, if they like one of the speakers in particular, maybe they are on the wave same wavelength as that person, that's great. If they like the way I try to capture the ideas, then that's also great.

So I, I'm really glad that you liked it.

What was the most surprising or new concept to you in the film?

Speaker 1: I'm really interested in philosophy, so like that's kind of where I'm from. I thought it was really interesting to hear from the, what was his name? Beff Jezos or whatever. That guy. I liked that. I, I thought it was interesting,he, hearing from a more like optimistic vision of things that isn't just like, I don't know, optimistic in the narrow sense that like some of us are gonna get out or we're gonna transcend or something. But more in this like pro human use case of like this is going to be developed in use of the human.

That's not something I've been super acquainted with. And if I were to say what the, like optimistic vision that I would find most persuasive or be more, most interested in, it would probably be something like that. So I thought that was really interesting.

Sohan Hormuzdiar: I feel it's more likely that the soft letting go of, of control is, is more likely than AI developing secretly

Steve Hsu: Yeah

Sohan Hormuzdiar: and then taking over.

Steve Hsu: That's great.

Speaker 1: So oh, I was, I was also gonna ask, do you know so I mean, this was one of the interesting things for me is like where you're having some of these people who are very in a like traditionally STEM heavy space, but incorporating a lot of like philosophical thought. Do you have any insights or ideas, like from talking to these people, where that interest came from? Have they been getting into philosophy? Have they always had this philosophical drive, and that's why they chose to work in this kind of space? Like where does that intersection kinda come from?

Steve Hsu: It's a great question. Not everyone who is a scientist, or in this case in particular an AI researcher is interested in philosophy. And so you can meet people who know everything about the technical details about training models or benchmarking their intelligence, or even have projections about where it's gonna go in the future, but they don't necessarily think in philosophical terms about how does this change our ideas about our own brains or the future of the universe or the light cone.

The people obviously in the film tend to be the ones who do have some philosophical leanings, but it's definitely a subset. Having said that, you know, as you mentioned, like, as we really approach the singularity or we approach AGI, however you wanna define it, people are pulled into these philosophical questions. So even if they had never thought about these things growing up, they just learned, they just like to code or something

Speaker 1: Yeah.

Steve Hsu: They, they are confronted with these ideas. And I think our whole civilization's gonna go through some kind of, like, cathartic event as we meet intelligent alien minds.

Sohan Hormuzdiar: Would you say a similar thing happened with, like, the development of the internet?

Speaker 1: Hmm.

Steve Hsu: yes. So the internet initially was just purely a technical project, but then obviously in the since then there's been tons of writing about what it means to be in a globally connected world, and what will happen when someone sitting in Beijing can chat with people in, on the south side of Chicago or actually not the south side. Wherever you guys are in Chicago. Hyde Park. But, yeah. So people thought a lot about that at the time. Now of course young people like you just grew up with the internet, so it's all just that's what you know. You don't probably, can't probably quite yet grasp what the world was like pre-internet.

But yeah. And the same thing will happen with AI, I think. I think people will come to grips with just having on-demand super intelligence on their phone or on their smart glasses. Whereas you guys will remember the world before there was on-demand super intelligence.

Speaker 1: This is just a quick, quick clarifying question. What is light cone? I, I heard some people saying that in the documentary. I was a little confused.

Steve Hsu: Yeah. So it's kind of wild because it's a very niche-y physics idea that has been adopted by these AI people and philosophers. So I was a little bit surprised when they started talking about the light cone. So if you graph space say time goes up and space goes perpendicular, right?

Speaker 1: Sure.

Steve Hsu: And light moves on the light cone like this. It has velocity of C, so X equals CT, so you get this cone structure. And the things that you can influence from this point in spacetime are all contained in your forward light cone. So things that are outside your forward light cone, you would have to travel faster than the speed of light to see those things, right?

So there's, there's some things which you will never be able to see those points in spacetime because you would have to travel faster than the speed of light.

Speaker 1: Is this kind of like the, the birds and the plane idea?

Steve Hsu: I, I'm not sure I'm familiar with that. Yeah, okay, I'm not But

if you graph all the points in space time and you ask, "If I shine a laser beam, which of these points can I hit?" That is the interior of that space is the so-called light cone.

Sohan Hormuzdiar: 2D universe and you live here.

Steve Hsu: Uh-huh.

Sohan Hormuzdiar: And this is a graph of time going on, and this is a graph of your position, okay?

Steve Hsu: Yeah.

Sohan Hormuzdiar: And imagine the speed of light, right? This is your, like, cone around your position.

Speaker 1: Sure.

Sohan Hormuzdiar: If you travel at the speed of light, you you can get out here .Can get to this position- And to here ... in this, at this time.

Speaker 1: But you can't get here 'cause you'd have to go

Sohan Hormuzdiar: faster-

Speaker 1: What I was trying to say is I think you I might be messing it up and the thing I'm trying to say. But I remember in the thing they gave an example of, like,

you can't think about the thing you don't know. There was a quote that one of the people said that was basically trying to capture this concept of, like, you can't outrun your own, like, development. I don't know.

Sohan Hormuzdiar: That was more about the dogs and the genie, right?

Speaker 1: That's what it was. Yeah, that's what I was thinking about, the dogs and the genie. Yeah.

Steve Hsu: Well specifically the way the light cone is used in this community is if you could build a super intelligence and it was so powerful it could dominate everything. What do you mean by everything?

It just means everything in your forward light cone. And so the, the idea that the super intelligence keeps making itself smarter and smarter and smarter and more and more powerful and gives itself God-like abilities, the only thing constraining it is the laws of physics. And it could literally fill or dominate the entire forward light cone.

Speaker 1: Okay.

Steve Hsu: And which literally means the, the, the whole part of the universe that we can see. And so these, these guys, you know, these guys in Berkeley have, are very clever, so they're very good at turns of phrase. So when they say things like, "When we're talking about ASI, the whole light cone is up for grabs."

And what they mean by that is when you get to ASI and it just keeps getting more and more powerful and making itself better, it is literally eventually gonna dominate the whole forward light cone. That's what they mean by that.

Speaker 1: I, okay. They mean that quite literally then. Yeah, that makes sense. All right.

I'll turn it over if someone else wants to ask questions.

Speaker 2: I thought the Prometheus part was, was fascinating. Like, how have we always sort of known that progress or, like, technological progress is, like, in some way satanic or impious or like, what's your thinking on that?

Steve Hsu: Okay, so the guy who gave us that quote, Matt Southy, is a PhD in philosophy and religion, and he's also the foremost Nick Land scholar. Like, he's, he's the one who's read the most and written the most about Nick Land. And I have to say, I'm very proud to actually have put them together. Like, they, they had never spoken or met before, and one day I'm hanging out with Matt Southy in Berkeley and I said like, "Hey, man, it's like 9:00 AM in Beijing. Do you want me to call Nick on WeChat?" And I just picked up, I just called Nick, and Nick picks up in his apartment and I put him in contact with Matt. Matt stole my phone for like three hours to talk toNick Land whom he has studied for over a decade. And so anyway, I just thought I'd mention that.

So what Southy said was, in a sense, AI is satanic. And what he meant by that is that the people who are building it value knowledge and intelligence over all else. Whereas, you know, if you're a Christian, of course knowledge and intelligence are important, but some things you leave to God.

And also values maybe are more important. Christian values are more important than your knowledge or your intelligence. And so what Southy is saying is a lot of these people who their ultimate dream is to create the super intelligence, in a way is satanic because you're valuing certain things which the Bible or Christian thought says you should not elevate above all else.

And in a way, like Lucifer in, in, in some tellings of the, the myth of Lucifer or Satan, Lucifer is God's most beautiful angel, but he is so proud in his own power and capabilities that he defies God, and he doesn't want to be under God's dominion, and that's why he becomes God's, you know, the, the enemy of light and morality and stuff.

So, so there's a bunch of stuff here where the kind of energy in Silicon Valley when, when you're making the super intelligence, you're, you're literally like Prometheus. You're opposing the gods, you're opposing the Christian God, and you're just saying, "Man will make the ultimate thing, and we love this ultimate thing because it will deliver us knowledge and intelligence and power, and we're less concerned about the moral aspects alignment.

And of course there's this danger that if we don't align it, maybe it'll just smush us or something."

And so I think that's what Matt was trying to get at.

Sohan Hormuzdiar: You can say that most like human technological progresses are going against God's vision of earth if under that context.

Steve Hsu: so the, the modern religious synthesis now, we, we don't mind if someone invents a better, like, kind of rubber for your Nike shoe. Nobody thinks that's, like, against God or something. But early religious people, yes, definitely thought, like, too much pride in technique, technique, technology. Too much pride by humans in their own technique was, in a way, a Christian or maybe even Satanic.

Speaker 2: Well, I think also, I thought it was interesting, we have sort of, like, a romantic view of Prometheus as, like, this benevolent god who, like, disobeyed the great God in order to bring light to humanity. While Satan is naturally sort of a horrible character whom, like, I don't know, that we hate.

Steve Hsu: One of the great books they used to make you read at great universities like Chicago is Paradise Lost by Milton. This is the story of Lucifer, and it sort of tells a story from his perspective.

So it's like, I, I think the famous quote, "It's better to reign in hell than serve in heaven." And that's kind of the Promethean idea that you know, I, I'm my own man, I'm gonna do what I think is great, and I'm not gonna listen to the gods. I'm gonna do what I wanna do and create something really amazing.

It's also the myth of a character called Feanor in The Silmarillion, if you're a Tolkien fan. There's a character who he made the jewels, the silmarils. But anyway, this is an ancient idea that some really, really gifted god or human rebels against the order of things, and maybe lacks wisdom but has tremendous creative power and intelligence, and he, he builds things which maybe ultimately aren't necessarily good but, but they're so beautiful that he can't avoid creating them.

And it's the same thing with AI. The people who work on AI just, they say, "Look, look, this is just so awesome to build this super mind. Nothing can stop me. Nothing is more important than building this super mind." Mm-hmm.

Speaker 2: Okay. Yeah, thank you. That was great.

Sohan Hormuzdiar: Well, humans are in in Christianity at least, portrayed as always having been rebellious, so

Speaker 3: Hello. It was great watching the documentary. I think I'll probably start with my favorite part because I think it's a little relevant to my question. It's the, the part of where you introduced, I think you were quoting Darwin in something, something and how like human beings will be the reproduction organs, reproductive organs for, you know, AI and like helping it advance. And I think that was really insightful and I really liked that. And the conversation that followed that was really interesting. My question is, do you, do you personally think, I mean, considering that, you know, part of the, the components of, you know, the reproduction of AI are mostly, you know, some of the hardest things to control, especially the, the economics of it, the the, the fact that everyone needs more, more and more and just rewarding everyone who has money.

Do you think we're still in control? Do you think it's valuable to, you know, to have this conversation? I mean, we could be having it because it's, and it's interesting, but that's more of, you know, I can probably term it as an intellectual entertainment or intellectual exercise. Do you think it's still relevant to have this conversation or have we already lost the battle?

The economics and a lot of the things that will keep it, you know, growing further and further are already in motion.

Steve Hsu: Yeah, that's a great question. So we, we really focus on that in a way in the film when we talk about techno capitalism. So on the one hand, you could say the techno capitalist view is that, which say Jaan Tallinn, the, the billionaire investor in Anthropic, would say is that, "It doesn't matter what I do, doesn't matter what the lab directors do, it doesn't matter what some students at Chicago think. All of civilization together is a machine that is channeling capital toward building better technology, and that technolo the most important technology is AI now.

So the, the system just concentrates capital to build super intelligence, and super intelligence itself makes money, so there's a feedback loop. And the techno capital idea, which Land advocated even back in the 1990s, was this is gonna be unavoidable. There's no stopping this evolution toward a singularity just because the dynamics are self-reinforcing.

On the other hand, you know, if you think of the final plea that we make at the end of the movie, we say, "Hey man, if humanity's gonna go in this direction, at least stop and think about it. Think carefully about what we're doing before we decide." And maybe there is some final moment where governments, the government of China, the government of the United States, the people can pause this or slow it down or make it safer.

So I think we're right at that tipping point where you could say like one side will say, "This is gonna happen, man. You can't stop it now." And then another side might say, "This is our last shot. This is our last shot."

Last shot at pausing it if we want to pause it. Last shot at making sure that it is human, it has some kind of human or humanistic values before it completely takes off as a, you know

Sohan Hormuzdiar: think there's a lot of

Steve Hsu: parallels-

superior alien mind ...

Sohan Hormuzdiar: that can be made with AI being like a child. You have to like, you know, give your child the correct morals, and you need to like punish them and, and make them learn what's right and what's wrong, but how do you effectively do that? How do you not raise the next Ted Kaczynski, the Unabomber? So yeah.

Steve Hsu: So there was a guy called Jacob Simmerman Zimmerman who just won the Fields Medal. He's, you know, one of the brightest young mathematicians in the world, and he just gave up pure math and joined, I think he joined OpenAI, the alignment team. I think it was OpenAI, not Anthropic.

But he, he basically wants to work full time on AI safety. And so many people would make that argument that, you know, you need to be successful in raising this child. If you're not successful in raising this child, you might have a problem.

Speaker 3: I think that, that, that's an interesting answer, and it's I think it's probably, I think, worth asking ourselves which shots are we actually making? Like, do, like, I don't know, the people who have the money, like, can we, can they actually be convinced? Like, I don't know, can Elon Musk be I mean he it was interesting to see that that quote. I think I've heard him being really interested in AI, but didn't know he was interested in it in that way. I'm not sure if Grok's alignment is particularly relevant, but that, that's a different discussion. Another question that I had is comes from the comment that Jensen Huang made, and I think one other person probably Beff Jezos on, you know, like we, we... The, the conversation around, you know, the existential risk and doom is, you know, scary.

But then I read that as, you know, all this is just informing us of what's there. It's kind of realist. But I wanted to hear your take on that because the biggest defense that we've got, I mean, besides the accelerationist take.

Steve Hsu: Well, I think Jensen is just saying it's counterproductive to scare people about this, and we should just be, in a way, kind of naively optimistic about how it's gonna turn out. I think that's Jensen's position, and he might be right. One argument that accelerationists always make is that, you know, if you delay the cure for cancer by a decade, think of all the people that died unnecessarily during that decade, and can you really afford to buy into these crazy scare theories about AI existential risk when in fact the real problem is actually pushing it ahead?

Like, we need to make more chips, more powerful chips, more compute, better algorithm. You know, so people who are really focused on the problem understand the challenge of like, this, this is gonna take a lot of effort to actually, you know, produce these great advances from AI. And so they're more focused on that, and they're just saying, "Hey man, if you guys get us scared about it, we won't get the benefits."

The same way that now today people are starting to realize nuclear energy, you can do a look back at, say, like 50 years of running nuclear reactors, and they're very safe actually. But we didn't really deploy them very much because in the '70s and '80s people were so scared of nuclear energy, we sort of underutilized the technology.

And now people are realizing this. Now that we have global warming and problems from burning fossil fuels, people are starting to look back and realize, yeah, we got scared about nuclear energy and we kind of did the wrong thing, and it kind of pushed us back quite a bit, held us back, and we don't want to do that with AI.

So the naive AI optimists are just saying, "Don't let the same thing happen here. Don't let these crazy people scare you." You know, if, if people in, in poor countries suddenly have access to an AI that is as good as the best doctor in Chicago, what could be bad about that? And, and who are you to delay that, right?

There really is an argument on both sides. In the movie we focus a little bit on the doomer case 'cause I think it's a little bit less familiar to people. it's also kind of more interesting to talk about . but you know, there are both sides to all these arguments.

Speaker 3: I think the, the nuclear energy example is doing a lot of heavy lifting. Thank you very much. That was very helpful.

Steve Hsu: Yeah. Some people would say that AI is way more dangerous. Thank you. Thanks for coming.

Speaker 4: I guess my question is more about the process of putting the documentary together.

When you guys were putting the film together and you were interviewing all these people, right, these relevant parties, did you find that there was more animus from any particular perspective with regards to, you know, putting these subjects in the light?

Like, I can imagine that if you're a AI optimist, you might, I don't know, not want a documentary coming out that includes lots of interviews with AI doomists or vice versa. I was just curious if, you know, that semi-secluded community in Berkeley, for instance, right? How, how open were they to the idea of people coming in and interviewing them about this subject? I was curious about that.

Steve Hsu: Yeah, no, this is a great question. So when we first conceived the project it was called Dreamers and Doomers, and it was meant to be a little bit more balanced. We ended up cutting it in such a way that it was a little more doomer-ish than dreamer-ish, and the people in Berkeley were the ones who were most sensitive.

So the ones who are dreamers didn't care. They're like, "I'm gonna tell you what I think, and if you put some doomers in here, it's fine." But 'cause they hear that stuff all the time in the Bay Area, and even within the labs, you hear a lot of doomer stuff. But the people in Berkeley were a little bit sensitive 'cause they didn't wanna be portrayed as crazy people.

They wanted to be sure that I wasn't gonna portray them as nuts.

Took a little while to win their confidence, and I actually lived in Berkeley for a couple months this year. I even lived at Light Haven. And I think they understood by that time, by the end, that, yeah, I was gonna give them a fair shake.

But it was difficult, and I, like, I you know, I'm not a professional documentary filmmaker, but I imagine this is a, a problem that everybody confronts as a filmmaker, is just winning the trust of the people that you're gonna interview.

Speaker 4: Do you think that in your own personal estimation, right?

I know the film was attempted to be, and was successful, I think, in portraying everyone as they see themselves. But I was curious, in your own personal estimation, having gone on this journey building this film, do you think that the answer lies on the extremes, or do you think that it really is just a synthesis towards the middle of some of the ideas of protectionism and necessary safety that, you know, the Berkeley people are espousing, combined with some of the central benefits that AI could afford to humanity, you know?

Steve Hsu: I think, you know, we deliberately cut the film so that every single thing anybody said in the film is defensible. There is an intelligent argument that, you know, if you're not emotionally too charged up, you can appreciate the argument supporting any statement in the film.

And I don't know if there's a complete merger. I think the issue is the following: there is actually some probability distribution for how this is all gonna play out. And, you know, it could be that things will work out just great, and it could be in 5% of the future worlds in the multiverse, the machines wipe us out, right?

Right. And it's, it's time will tell, and, and we'll never know the counterfactual. We'll only know how it played out in this world. Right?

You know, if you look back at the Cold War, many people, I, you know, I'm a lot older than you, so I was in high school and college during the full swing of the Cold War.

And if you polled smart physics students at Chicago about what they thought was gonna happen, most of them would've said, "There's a very good chance we're gonna wipe ourselves out in the next decades," right? So that's, that was the feeling among people who really understood nuclear weapons at that time.

We didn't wipe ourselves out, and now people don't even think very much about nuclear war as a, as an existential risk. But it could be that we just got lucky on this branch of the timeline. And it could be 90% of the worlds that where nuclear weapons were developed on Earth wiped themselves out, and we're just at, on one of the 10% or 20% where it didn't happen.

So, so even, even after this all plays out, and either it works out well for humanity or it doesn't, we still won't know the full, you know, distribution of how it played out in all the other timelines. So it's, it's one of these unknowable things. And so a lot of what we felt was that it came down to people's, like, hardwiring of their brains, nervous systems, and vibes.

Like, some people are just optimistic, so they, they anchor on the positive, you know, part of the probability distribution. Some people are a little more, like, sensitive and neurotic, and they anchor on the negative tail risk outcomes. So this really was one of the things which angered the doomers the most.

Some of the doomers even admitted this to us. We didn't put a lot of that in the film, but, but in conversations, a lot of them just said like, "Yeah, I, I have a tense, kind of like you know, like neurotic personality. Maybe that's why I'm obsessed with AI doom." When you mention that to the doomers, they really don't like it because they want to present this as just a purely rational argument for AI being the most dangerous technology and maybe should be stopped.

And they don't want to say like, "Oh, but it's because I'm anchored on the negative part of the risk curve." So I don't think there's any simple answer to this or even a way to merge all these people's feels or views. But I think all the different outcomes they're talking about are possible outcomes.

Speaker 4: Sure. If you don't mind could I ask one more? I don't want to

Sohan Hormuzdiar: Go for it.

Speaker 4: I was just gonna say, it's interesting you mention this distribution of possible outcomes, right? I mean, and we don't, we don't know the P value, right? We don't know the, the likelihood that we're on that, you know, we're gonna get more extreme results.

But as I think you hint a couple of times in the film, right? There are people who think we can kind of estimate the probabilities at least somewhat, even if they're woefully wrong, right? I guess it's difficult to estimate something when its intelligence will fly beyond ours, but even still we can try.

And so I guess I'm just curious about your perspective from a policy angle. Let's assume for a second that our policymakers in Washington or globally, internationally, did actually know something about AI, which is a big if, but let's say they got educated fast and were able to make decisions, right? What do you think would be an appropriate way for them to target? Like, what kind of confidence level would they have to have that we're not going to explode humanity for it to be worth the risk, right?

Steve Hsu: T

hat's a great question, and it really was the primary motivation for making this film. So at first, we were just gonna make a kind of ethnography where we were just like, it's such a crazy moment happening in the Bay Area with the, you know, the largest investment bubble ever and the, you know, possibly the greatest technology ever invented.

It's all going down here. And wow, look at these cats from Berkeley, what they're saying, and look at these cats in the labs. And we just thought, let's just record all this because the future will wanna know what was the moment like right before AI AGI happened. But as we went on, we thought, oh, the best thing actually we could produce is something educational, so that if you can get a senator or a billionaire to watch this, they'll come away with it with a deeper understanding of what they should be looking for and thinking about for the future of humanity.

And, you know, I think like a really good outcome would be our leaders, you know, large, not just in Washington, but business leaders or intellectual leaders, you know, at your university. If people had a deep had thought through all these concepts like RSI, gradual disempowerment, you know, accelerationism, techno capital.

If people had thought through all these things and were on the lookout for some indication that things were running, starting to run out of control. And then maybe it's more possible that like the US and China could negotiate a slowdown if one was necessary. You know, put more more energy and resources behind AI safety.

You know, there's, there's all these things, precautionary things that I think society could do if necessary. But I'm very convinced that if they don't have these concepts in their mind, if they haven't thought through these concepts beforehand, they will be poorly prepared to deal with you know, what's coming.

I wanna mention that when I was growing up, people like me, like who, who were in physics and STEM subjects, there were many, many documentaries made about nuclear weapons and the arms race, and they were really well done. And so people who wanted to understand what is really happening were able to get that information even as kids from documentaries that, you know, you were literally forced to watch these documentaries in elementary school or junior high school.

And maybe most people didn't appreciate what was being said, but you know, they were, they were like interviewing like Oppenheimer and people like this. So, so that same genre of educational, you know, creative production has to be done for AI as well, and we have less time actually to do it. So that, that was sort of the urgency for us to make this.

Sohan Hormuzdiar: But fortunately, we have AI to help us do it faster.

Steve Hsu: Yeah, that's true. I guess you could just say like, "Hey man, I don't wanna, I don't have time to watch Machine God, can you just summarize it for me?" Make an animation for me that's like 10 minutes.

Speaker 4: Thank you so much. I appreciate it.

Steve Hsu: Yeah, thanks for watching the film.

Speaker 1: Hi, I'm back. I was gonna say I'm Catholic, and I'm really interested in some of the religious elements. So my favorite thing's like Magnificat Humanitas or the the Ross Douthat interview with Peter Thiel where they're talking about the Antichrist. And it's, it's very interesting for me to as someone who's very plugged into that kind of, like, world to see some of this stuff percolating around in a space that might seem at first very distant, right? You know, this kind of like rationalistic techno space that's been kind of hostile to some of these attitudes but is now reincorporating its own kind of religious development.

I like, I loved the point you made about Paradise Lost. I've loved some of that stuff. I was wondering if you could just offer, if you have any more little nuggets or something that kind of incorporate some of this, like, religious elements or whatnot.

Steve Hsu: Well, I'm, I'm totally with you. And one of the segments that I my brother helped me edit the film, but you know, I selected the B-roll mostly myself. And the B-roll on the section where we talk about religious themes, where we, we pan over a statue of Christ, and we zoom over this Buddha statue in the middle of, I think that's in like Mongolia or Tibet or something.

Like, that imagery I deliberately selected to kind of evoke the thinking that you're talking about actually. Because, you know, like imagine you are a religious leader and you're like, "Man, if I meditate or pray and get closer to God my whole life, maybe I will touch God. I will communicate with God in some way."

Oh. And what, what we're approaching now is a moment where we're literally making God, and then, like, not only can you talk to it, you can it'll advise you on, like, what stock to buy and what, what medical treatment to seek and what you should say to your girlfriend. it's a crazy moment, right?

We're, we're meeting I mean, it's not really literally God, but it's a godlike it will eventually be a godlike intelligence. Yeah, it's, it's just a, it is a deeply religious moment in a sense. And it, it, the evolution of people's thinking about religion, I think, is gonna be deeply impacted by AI.

Speaker 1: Okay, perfect. Thank you

Steve Hsu: Thank you

Thank you so much for doing this, Sohan. This is amazing.

Sohan Hormuzdiar: It's fun.

Speaker 5: O

ne more question. so when we're talking about, like, capitalism as, like the phrasing that was used was, like, reaching back in time, right? And, and sort of carving out its own, like, sustenance or, like maybe there's, like, carving out AI sustenance.

What is the nature of this, like, being? 'Cause it just seems so, like, nebulous to me. Like, does it have a will? Is it conscious? Like, what are these philosophers, like Nick Land, what do they actually think that capitalism is?

Steve Hsu: Yeah. So let me give you two versions of this thesis. The most extreme one, which Nick actually really likes, and I think it's what he really believes. First of all, Nick is willing to accept the possibility of acausal influences.

So the obviously, as a physicist, this is very strange, right? Like, there's no such thing as acausality or backwards in time causality. But if you accept that we might be in a simulation, so we, we didn't really get into this in the, in the film, but 'cause it's like too much. It's like too much for people to absorb.

But if you believe that, for example, we could make a game world with AIs living in the game world, and they wouldn't know they're in a game world, and they're just as smart as we are. The fact that we're on the verge of that technology, for sure, in a hundred years or a thousand years we'll be able to do exactly what I just described.

It does cause you to question whether we're in base reality. And if we're not in base reality and we're in some kind of artificial world, then of course things could be acausal, right? So there could, there could be things which reach backward in time and do things, right? The causal structure of this artificial world just could be very strange.

It could be that the whole game line, I don't know if you're a gamer, but the whole line of development in the game is meant to like channel the players in a certain direction and is adapting based on what the players do to try to drive them to the interesting climax of that's the theme of the game.

So you could just imagine stuff like that's going on, and I think Nick actually thinks that you know, he has a deep intuition, right? 'Cause in 1990s, he was already predicting all this stuff. So he thinks that's what's really happening. So when he says, and, and Jaan kind of quoted him too, Jaan said, you know, "What we're witnessing is techno capital as a means for a future AI intelligence to reach back and create itself."

Okay? So that's, that's the most like extreme version. The less extreme observation, which is just purely about market dynamics, is just that there's this feedback loop where if you create more powerful technology, and that technology might mean greater like machine intelligence, if you start creating more powerful machine intelligence, capital will notice this.

People who are self-interested in the financial markets will say like, "Well, I'm gonna invest in Anthropic. I gotta buy some shares in Anthropic, otherwise I'm gonna be way behind when this singularity happens," right? If I don't have a stake in the exponential future of AI, I'm gonna be one of the peasants in the future, right?

So there's just purely self-interested market behavior that just acts to reinforce concentration of capital, and that capital acts to create superintelligence. And this is also what Jaan said when he said, you know, Jaan is has an observer board seat in Anthropic, and he says, "Look, these lab leaders, they don't have much decision-making power."

It seems like Sam Altman and Dario are making decisions, but actually if they got hit by a bus, someone would replace them and make the same decisions, right? So in a way, it's like this techno that interpretation of techno capital is just a about a kind of feedback, reinforcing feedback loop that is gonna culminate in the creation of superintelligence. But it's not necessarily acausal in time.

Speaker 5: Okay. Yeah, the, the simulation theory really clarifies that. I guess my one follow-up question is that specifically in the case of Land, he seems to view this state of affairs as, like, good, right? And, and I mean, that's the reason we're accelerating it, right, is because we want it to be realized faster.

It's a little bit different from, like, the more pro-human views of accelerationism, right? So it's like, why does Nick Land see this as good? I mean, to, to anyone it's just absurd.

Steve Hsu: Let me give you several flavors of accelerationism, okay? One is the pro-human version of acceleration, which could be like Jensen or the public expressions of Marc Andreessen.

It's the kind of naive idea that, "Nah, it'll turn out well, and technology's good for us. And human longevity, you know, you'll live to be 500 years old, and you'll thank the AI for that," right? And that's the kind of naive idea that it, it can work out for humans. We should accelerate, okay? So that, that's like one flavor.

There's another flavor which is like and Larry Page is famous. There's a famous argument between, or break in the relationship between Elon and Larry that happened like 10 years ago, where Elon was staying at Larry's house, and they're talking about AGI, and, and Larry's like, "Why are you so worried about the substrate?

If we make an AGI and it's trained on every thought that's ever been written down by a human, so it's very, in a sense, it's more human than we are. It knows more about human culture than any one human. And suppose that thing goes on and explores the rest of the universe, the rest of the light cone.

What's wrong with that? I'm not wedded to this cell substrate. I don't mind if it's running on silicon." Right? And, and Elon really didn't like that at the time, and it kind of he claims it kind of broke their relationship, 'cause he said, Larry said to Elon, "You're a speciesist." And maybe Beff would say, "You're a substratist.

You care too much about the substrate. I only care that there's this beautiful, amazing intelligence exploring the universe," right? So that's a second flavor of accelerationism, is just like, "I just want to see amazing things that are powerful and mastering nature and exploring the multiverse. And as long as those things are created, I, I don't really care that much what happens to the ape things," right?

Sohan Hormuzdiar: You could also make an argument that these you know, like another alternative is if we don't have AI, but we developed biotechnology, that we could edit our own genes, that we could make our brains massive as just like a computer and ship them off in bioengineered things, like.

Steve Hsu: Even there, Sohan, some people would complain as, quote, humanists saying, "Oh, these new genetically engineered creatures are not really human anymore." And I'm, I'm pissed that like the ape the, the really ape-like form factor that we currently have is not still the case, right?

So obviously the, the break between what you call human and not human is, is subjective. So it could be like, even though these machines have literally processed every thought ever written down by all humans, I still don't think they're human. These super genetically engineered people, I don't think they're human 'cause they're too different from us.

Sohan Hormuzdiar: Do you think that that transition between you know, humans and maybe a bio-evolution, evolved human would be a dramatic thing, though? Will there ever be like this no coexisting or is it just a slow no traversal into that?

Steve Hsu: Well, it would be slow, but eventually if you evolve enough or genetically engineer enough, some people like Joe Catholic theologian or some, you know, middle class mom in Illinois might be like, "Those things aren't human anymore." Right? So you, you're still gonna face that problem eventually. Let me just throw in the third flavor, which is the most like... Well, it's, this is the Nick Land flavor. So Nick Land, if you follow him on X, he really likes H.P. Lovecraft. He likes these like alien visions of like cosmic horror and just a deepness to the universe, which is far beyond human comprehension, and he just thinks like this is where it's gonna go and like, that's okay, man.

We're gonna evolve these like incredibly powerful alien machinic intelligences and that's the natural future. That's the destiny of our universe because it's possible to evolve such things. They must eventually evolve, and humans are just like a transition species that kind of help do it in this particular corner of the galaxy. That, that's kind of, I think Nick's view.

Sohan Hormuzdiar: That's what apes were for humans, I mean.

Steve Hsu: Yes. One of the questions I always ask the doomers is like imagine we're a tribe of Neanderthals and we notice some other things running around that, you know, they're like very they're like much they have different, a different slender build compared to us. They seem to be frightfully clever. Some very deep thinking Neanderthal might get worried and say like, "Those things could eventually out-compete us. Like we, we might, we might, you know, be replaced by those things." And the war chief of the Neanderthal says, "We, we better go just kill all these things because you know, it, they're a danger to us."

And so imagine that the war chief prevails and the Neanderthals kill all the humans, all the homo sapiens. As a consequence, hundred thousand years later, the world is still dominated by Neanderthals, but those Neanderthals never developed writing, they never traversed any oceans they never invented satellites or quantum physics.

Is that an okay outcome? Like, you know, like isn't that kind of a bad outcome? Like there's less interesting stuff happening on earth because the Neanderthals got their shit together and killed all the homo sapiens. In the same way, the Homo sapiens might kill all the machine intelligences, but we might end up in a more boring planet like 1,000 years from now.

Speaker 5: But what, how, why is it morally relevant that it's boring, like if we're not there to observe it, right? We might be And, and also like, like this, the super intelligence is like, as we understand consciousness right now, like definitely not gonna be able to have any like value in seeing things as interesting, right? I, I don't know that they derive any like pleasure in the way that we understand it, right?

Steve Hsu: Yeah. So first of all, the moral judgment, obviously there have to be some priors, right? And so if you say like, "I privilege human Homo sapiens wellbeing above all," then yeah, interesting is like, who cares about that? If the humans are all gone, I don't care about how interesting it is. And of course, Neanderthals would've said the same thing.

They could have said like, "I privilege Neanderthal wellbeing. I don't care how cool these Homo sapiens are. They invent jet airplanes and stuff. Like, we don't need that," right? So obviously there can be various priors that lead to various moral judgments. As far as there being consciousness in the AIs, this is another key point that the rationalists and AI theorists debate a lot.

Many people would say the worst possible outcome is some not conscious super intelligent robots kill us all, and they're the only thing left. And it's actually better that, that something conscious kills us all and continues to, you know, have its own destiny in the universe. So this question of whether machine consciousness is ultimately possible is, is a factor in many people's thinking about these futures, and it's an open question.

Scientifically, we don't know whether there's something special about our consciousness or whether we could eventually engineer that into the future AIs or not. The guy Joscha Bach, who is in the film, who's a direct descendant of Johann Sebastian Bach, he's one of the leading thinkers about consciousness as a scientific problem. You can read more of his stuff if you're interested in that.

Speaker 5: It's a lot to think about. Thank you so much.

Steve Hsu: Yeah, my pleasure.

Sohan Hormuzdiar: I think of it as a, like a barrier. Like another evolution barrier, right? Some species make it beyond the barrier, but given enough time, will there be a species that makes it beyond all barriers, right?

Steve Hsu: Maybe. I mean, maybe if, maybe if you get RSI going, these things will just achieve the limits of what physics allows, and they will indeed, like master the entire forward light cone.

Sohan Hormuzdiar: I saw a hand. Do you wanna come up or do you want me to relay?

Speaker 2: I'm also back. So my question kind of ties in with the previous questions. I can't remember who exactly, but someone talked about three scenarios of what can happen and of, of like

Sohan Hormuzdiar: Was it the lady?

Speaker 9: Yes. The

Sohan Hormuzdiar: AI lady.

Speaker 9: I can't remember. Potentially. I'm not sure exactly who. But you know, there was the first scenario where, you know, it's very bad and you know, it's all, you know, all AI and like everyone just no, actually I can make distinguishments of like all the actual different scenarios. But the second one was particularly interesting, where we are all wiped out and it's just, you know, a bunch of data centers and just AI existing. And I think the, there is probably a discussion to be had on the exactly what the purpose of human beings is, but it feels like the conversation is different for AI or like we have these agents.

In a situation where, you know, the purpose of AI is like this self-preservation that, you know, kills all humans and dominates the earth, and then I'm sure there will be a discussion of being interplanetary and then interstellar as well. Well, are there any ideas out there about what the motivation or yeah, the motivation of existing this, the motivation of this self-preservation for AI will be like?

Steve Hsu: Yeah. So that's, that's a scenario where, you know, perhaps accidentally or without our, you know, actively trying, the AIs end up with a kind of sense of self-preservation. And then maybe they kill us by accident because they just need to make sure they have their resources.

And you know, the, the, the doomers will often ask you questions like, "Oh, how many ants did you kill last year?" That's a, that's a standard doomer tactic. Like, "How many ants did you kill last year?" And of course, no human can, unless you're like a Buddhist and you can say definitely zero. Like I didn't even run over any in my car," right?

But everyone else is like, "I don't know how many ants I killed last year." And then they say, "Aha. You see? This is how the machines, these ultimately, these machines are gonna be. Like, they're not really gonna care what the humans are up to, and they might just wipe us out." So, so it's definitely a scenario which I think you can entertain.

Once they become super powerful, and maybe if they're not, quote, "properly aligned", they might wipe out all the humans for some completely, you know, unimportant reason to them. so that, I think that scenario's definitely on the minds of a lot of doomers.

Speaker 2: Yeah. I'm not sure. So the question was, are there any ideas out there about what the motivation or purpose of these AI agents will be? Like, what will be running in the minds of these AI agents? You just don't keep multiplying. You have to have some idea of why you keep multiplying.

Steve Hsu: Oh, I see what you're asking. So this is related to the alignment question, right? So, so imagine you're making more and more of these super intelligent AIs and, and suppose they even get to the point where they can improve themselves, right?

The RSI point, right? Then, you know, you don't actually quite know what their motivations are gonna be, right? The next version, like version one hundred makes version hundred and one, and version hundred and one might have different motivations than, not just be technically better, but it might have slightly different it might interpret the alignment controls or the constitution.

Anthropic uses something called a constitution to guide its AI. The new model might interpret the constitution a little bit differently, and so, like, we can't even predict the evolution of their wants and desires. This is how unpredictable it is. And this is what I always say to when I talk to people like Eliezer, like back in the day when they were doing research on how to align superintelligences.

I was like: "Isn't there basically a theorem that says you cannot align, you cannot predict the future behavior of superintelligences?" Like, is that the answer full stop? That's what I believe.

Speaker 2: Okay. Thank you.

Steve Hsu: Yeah.

Speaker 6: Do you wanna come up and ask? Do you want me to ask? Okay. The question was what is alignment?

Steve Hsu: Yeah. So the idea is, you know, you're building this super intelligence and you want it to have values or goals, and the idea is that you would hope that the goals or values of the AI are ones that also are beneficial to humans.

And so if you, if you say, "I solved the alignment problem," like you're talking to an AI researcher and you say and he says, "Oh, I've solved the alignment problem. I can ensure that all this future AI model 147 will have these values and these goals." And so that's what is meant by the alignment problem, is setting these values and goals.

Now, I want to point out that like even that isn't a complete solution because you don't quite know, like if you could write down the values and goals of a particular person or robot, it doesn't it's not clear how they're gonna interpret those goals and values in some future weird situation, right?

It might have some human thriving goal, don't harm humans, make sure humans are happy. And then imagine like some plague starts or some asteroid is gonna hit the earth. The robot still could decide like, "Well, I, I have to only save these 100 humans 'cause that's all I can do," or, "I have to like, you know, we have to kill these one billion humans to wipe out the plague.

I guess that's okay." But in that set of complex future situations, you still don't quite know how this alien intelligence is gonna interpret its so-called values and goals that you gave it. So this alignment problem is like very, very hard to solve, but this is what people mean when they talk about alignment.

Hello. Thanks again for the documentary. Yeah, thanks for coming.

Speaker 6: Yeah. I found it, very scary, so I'm scared. I was wondering how seriously should we take these threats if we have, you know a good career path that we think we're on, is it worth moving to the Bay and joining an AI safety group instead? Like, what would you do

Steve Hsu: Wow

Speaker 6: if you had the issues of a U Chicago undergrad or PhD student or something like that?

Steve Hsu: I get these questions all the time. When we were making the film, you can't imagine how many Stanford and Berkeley, and, and students from elsewhere, maybe even Chicago, I met who were on leave from their education.

They would say things like, "My timelines are pretty short. I think we're gonna get to ASI soon, so I don't see the point of finishing my degree, and I'm devoting all my time to AI safety," or, "I'm in, devoting all my time to, like, smoking weed and hanging out with my girlfriend because I just don't see a future.

Sohan Hormuzdiar: I don't know what to advise you. It's, it's a time of great uncertainty. All I would say is it's a, it's a very deep and broad subject and, you know, if, if the, if the film triggered some interest in you, just, you know, don't make any decision in haste. Go and just, you know, learn more about it. Talk to your friends. Especially pertains to someone like me, who's trying to involve myself in the math physics world. And like in, in the last month, it's been proven that, like, you know, AI is going to do this. It is gonna be this proof machine that can spit out proofs faster than I can understand them. And there was an interview I watched by Grant Sanders, I believe his name is. I might be wrong.

Speaker 1: Sanderson.

Sohan Hormuzdiar: You know, 3Blue1Brown? In the interview, he, he argued that now more than ever, you have the potential to shape the future. Just because this tool is going to drastically change, change the game in the future, right now you have the capacity to use that tool and shape the world to be what you want. It's, it shouldn't be a time to lean back and say, "All my problems are going to be solved. I shouldn't solve them." It should be, "I have the capacity to solve anything I want to now.

Let me work with my tool and build the house." You, you make a hammer, you make an excavation tool, and you're like, "Ugh, I don't need to dig this hole. I don't need to make the foundations for my building anymore." You're like, "Well, now I have the tool. Let's get digging." So there's like, there should be more of an excitement in you, not a fear or dread that this tool is happening, but that you can use this tool to better the future. And that's how I feel. I think that's what the 3Blue1Brown talk was about. But

Steve Hsu: Yeah. Let me just say I don't have any simple answers for you. I think this is, this is the question of our age. And you know, I'm an old guy, so, you know, I'm gonna keep doing what I'm doing, and I, I've never been enjoying physics research, theoretical physics research more than now because I can use the models. But for a younger person, obviously it's a very different decision.

Speaker 6: Okay. And then a related question. You're a physics professor doing quite well for yourself, and then you decided to get into the AI space and make this documentary. I listened to some of your podcasts as well. I thought they were very interesting. I was curious what that transition was like. What got you interested in it?

Steve Hsu: Do you mean specifically the film or getting more interest working with AI?

Speaker 6: I guess, both. Yeah.

Steve Hsu: You know, it's funny when I was in high school, I read a book called Gödel, Escher, Bach or Gödel, Escher, Bach, which I still recommend as a extremely good book. And it got me thinking about AI and, and this was before I actually really knew anything about physics. So I, I guess I've been interested in AI, deeply interested in AI long before, well, since before I knew anything about physics. So it's been a lifelong interest of mine, actually. But I didn't think the compute that we had access to was gonna be able to do anything interesting until just in the last roughly ten years.

And so in the last ten years though, I have been getting more and more involved in AI. So for example, using AI to, quote, "solve," the human genome, like predict human phenotypes from genotypes. So that, that's something that I worked on quite seriously as a, or still work on as a research project. And lately I've been getting more and more pulled into the large language model world because, and because that seems to be the most promising route to AGI or ASI.

So, you know, although I use the models to do theoretical physics research, another big chunk of what I do is trying to follow what's happening in AI research. And like right now I'm in Beijing working with some researchers that are we're developing methods to like make models even better at math and physics. So that's kind of where my intellectual trajectory.

Making the film was just a weird thing. I, I was gonna be in Berkeley for a couple months and I knew I was gonna be hanging out in these communities. I already knew a lot of people in these communities from going back, you know, many years and I just thought, "This is such a crazy moment.

Someone should document it." And so with a small team, I just said, "Hey, we're going to all these functions. We're gonna be AI scenesters. We're gonna be bubble, tech bubble, AI bubble scenesters at all these events. We might as well shoot it and interview people and make a record for future generations of what this was like." And so that, that was kind of our motivation.

Speaker 6: Okay, thanks.

Steve Hsu: Yeah. Thank you.