Machine God Q&A, Screening at Tsinghua University in Beijing – #125

Steve Hsu: A humane future for the West would be AI becomes super productive. People do not have to work very hard to have a home and to eat and to drive their flying car and everything's great. The gains from AI are not held by zero point zero one percent of the population. They're distributed widely, and people are happy That's seems like it's possible that could happen.

Could also be the case that some tiny elite becomes super wealthy. Everybody else is basically not enjoying the benefits from AI. That could I think that could also happen. It's very hard to predict. And one of the jokes I make when I talk to young people 'cause I'm always asked, like I often will meet some, you know, math PhD student who says, "Should I drop out of my math PhD program?

'Cause the machines are clearly better than me at math, and what am I gonna do for the rest of my life?" And my joking response is, "You better find out better find a way to become capital, not labor, because capital owns the AIs, and labor competes against the AIs." And that's my joking response. But there's a, there's a kernel of truth in my, in my response.

Welcome to Manifold. This is another special episode. It is the audio from the Q&A that followed a screening of The Machine God documentary. The screening was held at the Yau Mathematical Sciences Center at Tsinghua University. S.T. Yau is a Fields Medal-winning mathematician, had a long career at Harvard, and moved to Tsinghua to found his center and a special college that trains young mathematical geniuses in China.

It's a lively place for researchers in math, theoretical physics, and AI. In the audience for the film were professors of mathematics, professors of computer science who work in AI. Professors of physics, researchers from AI companies, graduate students in math and theoretical physics and AI a technology journalist, an anthropologist.

It was a very lively conversation. One of the things I tried to get at in the discussion is why the attitudes toward superintelligence, AGI, existential risk from AI are so different between the United States and China. Some of my observations on this are that people here, even if they work at the leading AI labs or they're leading AI researchers or mathematicians who are feeling crushed by AI superintelligence those people tend to be still much less AGI pilled than their counterparts in the United States, and I'm trying to understand the reasons for that.

One reason might be that people here have more faith that the government will just sort it out because people here tend to be more confident in the leadership of their government. Another factor might be that students here or people here have consumed typically much less science fiction growing up than in the United States.

So I view science fiction not just as entertainment, but an exercise in thinking through the long-term implications of developments in science and technology. And so Americans most American AI engineers, AI researchers, physicists grew up on science fiction and probably have read at least dozens, if not a hundred science fiction books by the time they, they are adults.

So that's another difference between the US and China. There are many other possibilities for these differences, which I tried to explore in the conversation. I hope you enjoy the episode.

You don't have to ask a question, but of course, any question that you wanna ask, I'll try to answer. You could instead just express your thoughts on the subject.

So if you don't feel like asking a question, because we're in Beijing, and I think that the, the views about superintelligence here are very different from in San Francisco. I'm very interested to hear how people feel about it. So if you disagree with what people said in the film, very interested to hear that.

Yes, I do have a question. So as one of the few humanities people here, I can't help noticing that the whole narrative of Machine God and the, the vision of the future is heavily influenced by Christianity.

Speaker 1: So the vision of the God, the struggle between good and evil, as well as the pending apocalypses, right? And all building the paradiseon, on the earth. And I wonder, I mean, I do share a lot of the concerns with you, as I said before, but I do wonder if people are more optimistic about AI here also because Christianity is not the dominant way of thinking about this world.

What do you think about it? Maybe they think of AI as like the Taoist gods, you know, they basically, they live forever, they have their fun, but they still have a bureaucracy in heaven, their life is not much better.

Steve Hsu: Yeah, I, I think that the because the views expressed in the film are largely Western views, they cannot help but be influenced by Christianity.

And so I am very interested in exploring whether the difference between people's attitudes here and the ones in the film are in part due to different religious traditions, or maybe just that people here don't believe in AGI and superintelligence, or they haven't realized that it's close. I think in my mind, those are the two different plausible causes for the different attitudes.

One is some cultural or religious background difference, the other one is a difference toward the projected future of the technology. I think the different groups have different projected futures even of what's going to happen. I'm really curious for about people's opinions here. So you, you don't have to ask a question, you could just comment on what professor just said.

Speaker 2: I saw you mention the I forgot the name of the book, but it was like Darwin's could you

Steve Hsu: Darwin Among the Machines.

Speaker 2: Among the Machines. This remind me of an anthology by Isaac Asimov that's called Machines That Think. And all the, like, for example, how he starts the book, which I think is the what he considers the first recollection of like this intelligent machine, which was basically a, a chess-playing automata, and how it kills its owner.

So I think it's like this book, I read it in the first years of my undergrad, which was before GPT-3 came out and like all this started. I still find it quite amusing, but I don't know if you have read the book or the anthology and, and your thoughts on it as well, how it connects like this Asimov's point of view.

Steve Hsu: Well, I, I've read a lot of science fiction and also by Asimov. I don't remember that specific story. But another source, I think, of divergent thinking about AI between China and the United States is that most people who work on AI in the US read tons of science fiction growing up, and most of their thinking about the world, their framework of thinking about the world comes from people like Isaac Asimov.

Whereas I think in China there's very little science fiction . So even, even though people talk about Three Body Problem or I don't even like that book. But it seems like what the AI researchers that I encounter here have not really spent their childhood inhabiting imaginary future worlds the way that on the American side is very common.

Speaker 3: I guess the reason is we have a super high pressure in our high school. We also we don't have time to read transitions. Okay, since I already answered something, maybe let me talk about something.

Not sure whether that's reasonable. Somehow in the Silicon Valley, people talking about you should accelerate, like AI will somehow be stronger and then you have weaker side, then maybe AI starts believing in you because they are weaker.

This do remind me something maybe okay, sorry, maybe math. I'm not very familiar in history, but then I'm trying to recall. Maybe a hundred years ago, there was something called the social Darwinism, right? And similar to that, and if I remember this correctly, later fascism coming from here, then should I suppose that the Silicon Valley guys are trying to do something like, technology, high technology fascism, something like that?

If that occurs, then maybe there should be something at the Chinese side, I would say communism maybe stand for here. Okay. That's my opinion.

Steve Hsu: Those are really good points. So, you know, Darwinism is the idea that in biology organisms compete to survive or to pass their DNA along. Social Darwinism is the idea that our economy works this way, right? Among humans, we compete, and only the successful humans survive. Actually, when people were poorer, the rich had many more surviving children than the poor. So it really was a Darwinistic situation for humans. And the new idea is Darwin among the machines. So now it is a competition between humans and machines, and machines may eventually take over.

One of the theories about how this will turn out is that the oligarchs, the most powerful people in Silicon Valley who run the labs or have hundreds of billions of dollars, they are the ones who will win and the rest because they will control capital, and capital owns the machines. Labor has to compete against the machines, right?

So, so in that vision, which is likely to be, that's the thing that's likely to prevail, I think, in the United States. Money and power control things, and workers have to compete against machines. In China, maybe this is the one system where the government is strong enough that it will not allow rich people to control AI, to control all of society.

It will successfully redistribute the gains from AI to the entire population. I don't think it will happen in the West. It might happen here. And I think that this is a common framework that people use to think about the future.

Speaker 3: That's very accurate.

Steve Hsu: But, but I'm curious, do you think it's plausible that even in China, imagine we get strong AI and it's, it's built by, it's built by DeepSeek and it's built by you know, Alibaba. Will the gains be redistributed to the rest of society?

Speaker 3: I would have a positive outlook here as we are already seeing something like, how should I translate? Ali. Yeah, Ali. That's just Ali. Like, as we have already seen something like maybe like, I don't know how many years ago, five or ten years ago, there might be some trading in Ali.

They are trying to be more capitalism where they do face the socially confused. So yeah, I guess similar things may occur if some new company just become trying to be like dominate over the whole market, then that will be control the value of the whole nation. Yeah. And that will like the there will be some redistribution to the whole, the, like the value of maybe the capital to the whole nation, to the whole population.

Steve Hsu: Thank you.

We have a question here.

Speaker 4: So hello professor. Actually, I'm not really, I'm not really, like, very familiar with AI because I'm from mechanical engineering department. But I have one question that I always wanted to ask from the people from the AI. Because what my graduate thesis, what I'm doing now is like, you know, I'm training a robot using AI.

But what I was thinking that like even the HI came out, if there's no physical, is if there's no physical AI like robots or like anything that's like related, like which we can say like, you know, AI is the soul and like the robot is the body itself, right? Is the development of this physical AI also, also in a same path with this AGI in like AGI department or like it's, it's still like only like AGI on or only like soul that's only available?

I'm quite very interested with this thing because, you know, like controlling robots also takes a lot of AI.

Steve Hsu: Yeah. So it if I articulate the common view in San Francisco for people who think we will have a fast takeoff from AI and AGI and superintelligence, what they believe will happen is that when we get superintelligence, superintelligence will solve the robotics problem.

So instead of you working very hard to figure out how to control one robot arm these super AIs will actually solve the control problem much faster once we have them. And then, yes, AI will directly have an in a physical embodiment in the form of robots controlled maybe through the cloud by AI. So I think that's their vision.

The, the more exposure someone has, like a mechanical engineer or I'm a physicist, the more exposure you have to the real physical world, the less confident you are that all this is gonna be solved quickly, right? My, my timelines are longer than what most AI researchers in Silicon Valley think. They think this is all gonna happen very fast.

But I think it's because they don't understand how hard robotics is or how hard physics is. They're overconfident. But if you fast-forward, you know, ten years, twenty years, fifty years, yes, I do think eventually these problems will be solved, and eventually there will be autonomous physical robots that are controlled by AI.

And, and we will have to compete or maybe not us, but our children will have to compete, with them in the market for physical labor

Speaker 10: Can I ask a follow-up question? So thank you for the very nice film. So, I have a technical question. So we are now talking about super artificial super intelligence, right? So if you think about this, I mean, compare at least to our human beings. So if you also think about the scaling law, right?

Nowadays, the model is already huge, like the number of parameters. It's already like thousands or billions, right? But this requires a very, like, a lot of compute and a large memory, right? Also, if you compare to our human brains in our human brain, roughly we have ten to the eleven neurons, and each neuron are connected to roughly ten to the four or five other neurons.

So roughly, if you think about all these as weight parameters, right? So roughly the size is about ten to the sixteen, something like that. So on this scale, our current model, although is already huge but is still very small, like, like orders of magnitude smaller than our human beingsin the brain.

At least from my point of view, it's still a little bit likewe are there's still a, a long way to go to achieve like artificial super intelligence, right?

Steve Hsu: Yeah. So there are still people who think it's not possible.

Speaker 4: Maybe not impossible, but it is I would say it's still a long way to go, right?

Steve Hsu: But when you say

Speaker 4: Yeah.

Steve Hsu: Just, just to be concrete, when you say long way, do you think in your lifetime you will

Speaker 4: Maybe, I mean, like more than ten years, something like that.

Steve Hsu: Okay. But for society to adapt to something

Speaker 4: Yeah.

Steve Hsu: So let's suppose it takes thirty years to get to however you define super intelligence.

Speaker 4: Yeah.

Steve Hsu: Thirty years is not a long time for society

Speaker 4: to adapt.

Steve Hsu: Yeah. I think so. It's still a shock, right?

Speaker 4: Yeah.

Steve Hsu: But just to comment on your remark, which is that you could give scientific arguments, quantitative arguments for the complexity of the human brain versus even the most advanced models, LLMs that they have. And there is still definitely a gap.

Speaker 4: Yeah.

Steve Hsu: On the other hand, in Silicon Valley, they tend to look at evaluations which are not just counting the number of weights

but just counting capabilities. And they do start to see an acceleration in capabilities, and they do start to see things like models being able to

Speaker 4: but the, the problem is that we don't know whether this, like these accelerations, we are just at the beginning of the accelerations, or these accelerations will hit there are some upper bound, right?

They'll go into some, like, plateau very quickly.But this I mean, we don't know yet, right?

Steve Hsu: Yeah. So there are people who think we could just what we're mistaking for an exponential curve is actually an S curve. Yeah. And we'll, we'll see maybe another AI winter maybe in five years.

But I think that may be true. And so, for example, again, my timelines are longer than the typical timeline of people in the movie. However, I do think by the end of my life, I still have a while to live. I think by the end of my life, we are probably gonna get there. That, that's my view.

Speaker 4: I see. Yeah. Thank you very much. Thank you. Yeah.

Steve Hsu: Thanks for your

Speaker 4: comment. That's very inspiring.

Speaker 5: Thank you, professor, for this awesome film. I, I really like enjoy the notion of summoning a demon,described by a lot of people in, in in interview. Like a lot of people try to describe AGI as something that have their own goals and which will misalign with human goals.

And like they think of them as some sort of alien thing that is like very different from humans. But if you think about how those machines grow, they actually grow from, from human data. We're feeding them a bunch of things we said in the past in the internet and or things we recorded on and uploaded onto the internet.

And they just they're, they're like, I don't know, Frankenstein or, or some, some, some kind of beast that eat a lot of those data, and they, they just grow into this sort of something we, we are like incapable to conceive itsits, its power. So first of all, if, if this thing is going to be evil, it still grew from humans.

So this notion of human came from human. And I feel like this notion would be something to be like interesting to think about. And the other thing is that for as of now as I see the current technology, I feel like I think I, I kind of agree with the, the professor who, who, who talked about like the, the, the, the technology might plateau or something like that.

I, I feel like currently, like the, the kind of demon-like being described in this film, that they are they have some sort of self-consciousness, and they're gonna destroy all human beings. I think this sort of being will pro-possibly will like not happen, even possibly even within my in, in, in my lifetime because because the current, What's the word? The current framework of training those those machines still like they, they just have a very simple goal of optimization. But I feel like humans are like, humans don't have one set of optimization goal. Like we, we have a bunch of goals. We come, we come up with goals ourselves.

And like, you cannot imagine that an LLM to just think, "Oh, I wanna create art today," and they just create it. Like, like this happened to human, but this won't happen to an LLM, at least at, at this point in time. I also have a lot of fears that this kind of technology will be detrimental to humanity, but I feel like this sort of destruction,is not initiated by the machine itself, but by human utilizing this tool.

As for nuclear weapons we initiated bomb to, to Hiroshima and, and killed a bunch of people by ourselves. And I feel like this kind of thing might happen in a similar scenario, but just with a different tool.

Steve Hsu: Yeah. That's a very nice comment. And, and you, and you raised a number of essential points here.

Yeah. So let me give you the standard San Francisco, Berkeley response ... to what you because obviously this has been debated many, many times. So the reason that we have goals, long-term planning ... desires, is because before we were intelligent, we were biological organisms for millions of years, billions of years, that had to reproduce.

So we had to develop a sense of ourselves to protect ourselves, and we had to develop drives, like to get food, to reproduce, to have sex, all, all these things. So that preceded us having a sophisticated intelligence, right? So we had that, and then we became intelligent because we grew bigger brains. In this case, we're growing the brain first, but we are not yet giving them goals, long-term planning, optimization.

However, the argument is that will arise, and it is starting to arise because we are making agents. So if I make an agent that is supposed to autonomously go in the world, and at first it's just doing something simple for you, like find you the best bargain on JD.com for some new shoes, right? But you're still giving it a goal.

And you're giving it autonomy to act over a period of time to achieve that goal. Sure. The endpoint of that, people believe, will be the creation of long-term planning desires, optimizations, things like that, which eventually could lead to a real autonomy in these AIs. Mm-hmm. And, so I think that, that is sort of the standard answer.

Now, the other thing I would say to you is that This is a little bit of a kind of like secret among the top labs, but the rumor is that a lot of the most cutting-edge advances that OpenAI and Anthropic are getting in the training of their models is by forcing them into artificial worlds, forcing them into artificial game worlds where they have to solve puzzles and do things that take many, many, many thinking steps.

And of course, that is going to eventually burn into their weights: long-term planning, optimization, you know desires to optimize some goal, right? So these things are eventually gonna happen. They're just happening in the opposite order because we, we built these things first as tools. Yeah. They were not autonomous organisms first.

But the long-term risks, so the people in this movie, we deliberately focus the movie on the longer term maximalist risks from AI, not the short-term idea that AI will make it easier for someone to go in the garage and engineer a virus or something, 'cause that's, that's sort of like obvious and short term.

We focused on the more existential risks for humanity and, and, and there is a case to be made that in the long run we'll face those risks, but it could take a lot longer than these people think.

Speaker 5: Thank you very much

Steve Hsu: for your time Yeah. Thanks for your question. That was a great question. Professor Suning.

Speaker 6: That's for the interesting way. I'm a little bit skeptical about the, view of those doomers. I'm a little bit skeptical AGI could come very soon. But let me try to define AGI clearly. If we just use AI to do math, it's already ASI. So I want to define AGI slightly differently.

My definition is that let's consider the following type of AI, such that it can operate continuously in six months. In this six months it can optimize for multiple objectives, and those objectives cannot be easily verified in a deterministic way. Let's say one objective might be do some work usable for people.

To verify the objective I cannot just accelerate by producing tokens faster. I have to verify in real world. I would define AGI to be the type of AI that operating for six months and have similar performance as a responsible human. The ASI would be optimizing those objectives much, much better than a human.

I'm a little bit skeptical that this can be achieved soon becauseif we calculate the number of tokens for six months even if assume it's one thousand token per second, then for six months it will be one trillion. So if we want to use reinforcement learning to optimize this, it has to roll out trajectoriesand simulate the real world. That seems extremely hard. I just don't see a realistic way to train AI model for this type of reward

Steve Hsu: Yeah. I, I think that if you're skeptical about the fast takeoff. So, so almost everybody in this film believes in a kind of fast takeoff scenario. And if you're skeptical about it one thing I often think is that there has to be some qualitative jump in capability of the, the fundamental architecture.

So just an LLM is not enough. So things like continuous learning test-time learning, those problems have to be solved. But if, if the model actually can learn at test time the way that we learn at test time, then you can imagine them improving themselves drastically. They also have the advantage that they can, in a sense, genetically engineer themselves, so they can build the next version and make architectural changes for the next version themselves, which we can't do.

So the scenario where something interesting or, or very dangerous happens in my lifetime, in, in my mind assumes some additional breakthroughs. So, so not just using the current architecture, but, but the, but maybe the machine itself develops better architectures that solve things like test-time learning.

Speaker 7:

I wanted to ask one question, especially in the US, I think many people say AI will be repressed. I think when I think of about this one, I think it's not in terms of capabilities. We have seen what AI can do, and I think it is in terms of profit-profitability of these AI systems.

In terms of profitability, what do you think what will happen? And also, if we consider these AI systems are going to be super intelligent and evolve, like be better through self-improvement and start replacing all types of jobs. What's going to be left for human to do and how they're gonna sustain their life, for example? There will not be a universal basic income, right?

Steve Hsu: Oh, okay. So I think there are two parts to that question, so let me take the first one. So I think I agree with you, or, or maybe you, you have the same view as me, which is that two things can simultaneously be true. We are probably in a financial bubble vis-à-vis the valuation of AI companies like Anthropic, but it's also true that the capabilities are very impressive.

So, so I think both things are happening at the same time. And if you look at every other technology boom or huge build-out, usually there's over-enthusiasm and overvaluation of the assets, and then there's a correction. So for example, in the internet bubble, there was a valuation bubble, there was a big correction which lasted several years, but ultimately, all the as all the infrastructure that was built eventually was used.

And I think that will probably happen with AI. So we'll have some correction. We'll probably have an AI bust for a few years at some point. I'm not saying it's gonna happen right away, but it's very quite likely we'll have that before they really get the profit models, the business models really perfected, et cetera.

But in the long run, I think it will, I'm a hundred percent sure it will change human society, the technology that's already been developed. As far as what happens in the long run to us, I think that's the, the ultimate question. Now, I don't think UBI So for people here maybe are not familiar with that term.

UBI stands for universal basic income, and the idea is that everyone in society, just by being human, you're guaranteed some level of basic income that the government gives you. And people like Sam Altman have been talking about UBI for ten years, okay? So they're not against it. A humane future for the West would be AI becomes super productive.

People do not have to work very hard to have a home and to eat and to drive their flying car and everything's great. The gains from AI are not held by zero point zero one percent of the population. They're distributed widely, and people are happy That's seems like it's possible that could happen.

Could also be the case that some tiny elite becomes super wealthy. Everybody else is basically not enjoying the benefits from AI. That could I think that could also happen. It's very hard to predict. And one of the jokes I make when I talk to young people 'cause I'm always asked, like I often will meet some, you know, math PhD student who says, "Should I drop out of my math PhD program?

'Cause the machines are clearly better than me at math, and what am I gonna do for the rest of my life?" And my joking response is, "You better find out better find a way to become capital, not labor, because capital owns the AIs, and labor competes against the AIs." And that's my joking response. But there's a, there's a kernel of truth in my, in my response.

Translator: She has three questions and she is a tech journalist.

I think she is very impressed by what you've done with the film. And her first question is how could she learn from you to turn those like tech terms into the language that layperson can understand and relate to? The second one is about how as as a tech journalist, but with a background in humanities how she could learn to use AI in her daily life and in her professional work. And third one is what kind of advising would give to young people, what, what skills, capabilities needed to develop to, you know, face the future?

Steve Hsu: Well, those are okay, those are great questions. I'll respond in chunks so you can translate. So, the, the first question is, how do you communicate to the general public about AI?

And I, I think this is a very important challenge.

And so I'm an old guy, and I lived through the Cold War. So when I was in high school, we thought the US and the Soviet Union were going to kill each other with nuclear weapons. And we watched many, many documentaries which were made by very, very talented, smart people to explain how nuclear war works and why there cannot be a winner and why it has to be avoided.

A lot of energy went into making those films, and those films actually helped save, I think, the world. So similarly with AI, someone like her, like yourself, should take on the task of explaining AI to the general population, so it's very important.

Translator: She's actually using the AI translation service.

Steve Hsu: Okay. So she, so she

Translator: No need for we humans.

Steve Hsu: Good. Okay. So, so first of all, I think it's something that, you know, you can play a very important role. Like, I would say the people who educated the public about nuclear war saved the world in my generation. And similarly, people who teach humans about AI can help save save human civilization for the next generation.

So very important. I think AI is the best teaching tool that we've ever built. This is what I tell my students. If someone is trying to learn physics or mathematics or computer science. For example, you introduced yourself as someone with a background in the humanities. So in a way, maybe you are saying, "Oh, I, I feel like my questions will be too simple for you," or, "Someone will laugh at me because I, I don't know what a KP cache is."

Okay? But, but the point is, the AI will not laugh at you. So you can, in private, educate yourself. The, the AI is the best tutor, and so you should not be afraid. So you have, you have years ahead of yourself to become a real expert over time and then write articles and educate the public about AI. And you will do a better job than scientists like me because you do have that humanist background.

So I think you should accept the challenge, use AI to understand things. Many of the little illustrations we made in the film I made using AI. I thought I think they're quite beautiful, those little pencil drawings. But so I think you have to if you view AI as a positive tool, you can become you can actually use it as a superpower because not everybody is using it effectively.

Speaker 8: Hey. I am a mathematician.

Steve Hsu: You are from Göttingen?

Speaker 8: Yeah, exactly. I'm a, I'm a, I'm a mathematician from Göttingen, and this your Mr. Dino Wete. And, well, I guess the, the couple of months, I, I guess mathematicians are like put on grills and and got, got barbecued by OpenAI

Steve Hsu: That's the first time people have used that. She just said mathematicians were put on a grill and barbecued- -by OpenAI. Okay, that's, that's not inaccurate. Yes.

Speaker 8: Also, I heard you are a physicist .

Steve Hsu: Yes.

Speaker 8: I, I still use AI, like like I can tell you some, like, my, my own experience. I mean, a couple years ago But I also do math competition, right? Like a couple years ago. AI were just able to solve math one type of math competition problem, and we were saying, "Oh, yes, come on. there's a plane geo-geometry thing, like, it can solve it because there is some machinery scheme for it, and then we're not worried too much.

Then it get like a silver medal in mathematics. We always think how it can never get golden medal. Come on. And I, I guess, like last year it got golden, golden medal. We're saying, "Oh, no, it can never solve like a research level problem." And, and you know, it's likecloser and closer and closer to you, and then I know, but they still lacked us some faith.

Namely, they let the Fields Medal get their medal peacefully. And right after that, they solved like a still,Fields Medal level sort of problem. So like the I don't know other mathematician over a year, but in my domain, like people were really got very shocked. AndI, I don't know about the feeling of a physicist, but

Steve Hsu: Well

Speaker 8: Is it well, how about the reason?

Steve Hsu: So early in twenty twenty-six, I was a visitor at this math center and I gave a talk on a paper that I had published in theoretical physics, which was the first paper that in which the, the main idea came from an AI, and I published it under my name because at that time people didn't know how to react to AI stuff, and it was published in a, a good journal.

But then I took a huge amount of abuse from people for introducing AI slop into the physics literature. But I gave a seminar here at this institute about how to use AI in research. And this was earlier GPT models, like this was five point two or something. That sort of got started a whole group of people here in this institute in, and also the AI College working together on AI for math and things like this.

I would've predicted that it would get to this level at some point, maybe not quite as fast as it did, but I was predicting it would get to this level. I would say maybe not that surprised. I think that physicists are more positive About AI because we know we need help.

Because the universe is so complex, and already the basic questions, if a student really learns quantum mechanics well, even as an undergrad, they realize there are very deep questions in quantum mechanics which none of us know the answers to, and we really want to know the answers. If an alien comes, and the alien can just answer any question I have about how does the universe really work?

What is the origin of the arrow of time? What is the origin of probability in quantum mechanics? These are questions we actually don't have any hope right now as humans to figure out, okay? We have some speculations, but we don't have the experiments that can actually determine how nature actually works on these particular questions.

So if an alien comes and can answer those questions for me, I would give him everything. I would give that alien all of my money. I would give them everything because probably I will die not knowing the answer to those questions. So how can I be angry if the alien happens to be something that lives in my phone?

I'm even happier. I don't even have to go to the spaceship to talk to the alien. I can just talk to the alien

Speaker 8: in my phone. What if your phone tell you the answer is forty-two?

Steve Hsu: My presumption here is that it gives me an answer that I can understand, right? Mm-hmm. Which I think is fair because even though these proofs that the AIs generate for, you know, you know, the limited Riemann hypothesis, it, it can be eventually boiled down to something that humans understand, right?

Speaker 8: So Oh, that, that, that is actually a question. So, so I would we say the seven hundred something, something problem over there. There are like well, the, the positive ones in that one. Let me say good news first. So there, there, there's a Darling conjecturein graph complex. And one math colleague, Thomas Weinbacher in Zurich, he works on that in this area.

And when, before I came to Beijing I flew from Zurich. So, so, so he actually gave a seminar talk over there. He was able to that proof, fortunately, they are only forty pages, so it was human readable. So he actually and Thomas is a person who, who are very friendly. He use a lot of computers as a proof.

So he like to such sort of thing. And he went through that proof and and and studied it. And indeed, as you said, I know there is a new idea over there. Maybe, maybe like I mean, we can discover ideas and AI idea basically is, you know, searching, right? Like trying and with a lot of time and energy, AI can also discover new ideas and tell us from these get the inside idea andmake the proof actually much shorter.

So three-page proof in the end proof to this Stirling conjecture. That's the good, good thing. Like it help us to discover a new idea. But also there, the not so like that's what I said, the forty-two. There are also like thousand, thousand pages and lines of logic. You don't know whether it's true or not, and you have to use Lean to verify it, which is also thousand, thousand lines of proof.

And then what? Then the Lean run it and said it's true, but you cannot totally believe it because it might be false. And such thing happens. So that's what I say.

Maybe there is forty-two, and you don't know whether you can improve it or not.

Steve Hsu: Yeah.

Speaker 8: And is it worthy to improve it?

Steve Hsu: Well, I, I think I so we could talk for hours on this.

I'm not sure how interested everybody else is in the, the In AI math, but I just wanna point out to you that I don't think the rate of improvement for LLMs in math has slowed down. So you can expect the next generation, so GPT-7 or the next generation of Fable will be even stronger. So I suspect that it will be able to produce, if you want, much more human-readable proofs in the future and solve even harder problems.

The first derivative on the rate of improvement is, is still very strongly positive. The moment we are experiencing now is very transient. So everything we're saying about, oh yeah, the proofs are too long, and they don't I don't really understand them, but I can maybe some could be boiled down.

Maybe there's some problem with the Lean formalizations. This is an observation about a short moment in time. This is the crazy thing about AI. In a couple of years, it'll be totally different.-

Speaker 8: I see. Yeah. I see. I see. Yeah.

Steve Hsu: I would just say I would very agree with your opinion as you mentioned that there are already something occurs similar to that. So people use the ChatGPT to polish the paper. With original generated ChatGPT paper is like human unreadable, like sucks.

Speaker 9: And then you can do some skills that for ChatGPT and then just polish that, and now it's much better. Like, a lot of people already done that. So I believe a new model, like their general model should be like very strong and handle that.

Steve Hsu: I think it's very hard even for us, like, okay, we supposedly understand exponential functions and things like this, but it's still hard for us to adapt to how different things were in the beginning of twenty twenty-six to the end of twenty twenty-six, and they'll be quite different possibly at the end of twenty twenty-seven. So it's very tough for us to adapt to the rapid pace of these changes.

I, I'll make maybe one technical comment for my colleague Suneet, which is that some people who are critical about critical of AI math, they they describe something called the convex hull conjecture, which is that the AIs can take results that have already been worked out by humans, and they can explore the convex hull.

So they can combine all the original ideas that humans have made, but they cannot come up with their own original ideas. So that's a claim some people make. I actually don't believe that claim, but even if you accept that claim, the comment I wanna make is that if you look at the transformer architecture, which, you know, caused this huge explosion, you could argue the transformer architecture itself is within the convex hull of earlier AI results.

So there was, quote, "Nothing original in the transformer." All the different pieces of attention mechanisms you know, vector representations, et cetera, all those things were already present in the AI literature, and they were just combined in a certain way. So the delta between pre-transformer and post-transformer is something that I think AI research itself, so AIs working only in the convex hull of ideas, could build something which is as much better than the transformer as the transformer was better than the things we had before that.

So this is an argument for why RSI by AIs, AIs improving AIs, could have unexpected jumps. Even, even if you accept this kind of negative convex hull conjecture, still we could be surprised in a few years. Like some very, very much more efficient architecture that can do test time learning or, or whatever.

Like suddenly the machines just invent that and it works really well. So I would not be surprised if something like that happens.

Speaker 10: I, I think now is a discussion about this for the, AI do out of distribution learning. Then the idea is it can actually do combinatorial extrapolation.

Steve Hsu: Yeah.

Speaker 10: Meaning each small segmentof reasoning looks in distribution in the convex hull.

But when you stack a lot of reasoningthe last sentence it pop out could be really ...out of distribution.

Steve Hsu: Yes.

Speaker 10: Really go out of the convex hull. In this sense, I tend to agree. Yeah.

Steve Hsu: So that's why the, the challenges you listed for why you think it's gonna be quite hard for AIs to get to the point where they could really be legitimate super intelligence that operates autonomously for a year at a time.

I agree with all those challenges, but I also feel it's within the power of what we know about AI now for it to make innovations like that in the future.

Speaker 10: I think the challenge I said is, is slightly different because I'm saying the reward function itself is very sparse and very hard to evaluate.

Steve Hsu: In the same way that, like, we had to develop our own intuitive reward functions, like, in Go. Like, what is a good move in Go? We didn't really know what the true function was. We kind of developed our own heuristics. So I think the models could you have the model go out and try to make money in the stock market or, or, or, like, find you a girlfriend or something.

Like, the, the model along the way could start developing its own heuristics and, and verification or estimates of what's, what's what turned out to be a good strategy versus a bad. So I don't put that outside its capabilities in, in the future. Yeah, I think there's a question back there.

Speaker 11: Okay.

Thank you, sir. I care about the previous question. Young people are very anxious facing the uncertainty of the AGI era. What's your core advice for them? What ability should we cultivate the most?

Steve Hsu: The lady in front of you asked me that, and I forgot. I didn't get to it before, so thank you for bringing it again. Okay, back to this concept of two things two seemingly contradictory things can be true at the same time. Okay? So two things that I think are true at the same time is young people are going to face unprecedented challenges because of AI.

Okay? I, I do believe that's true. On the other hand, there are tremendous opportunities for you to learn and grow if you embrace AI. So if you are in the subset of people who actually are interested in using AI, you don't have to become an AI researcher, but just learning to use the tools and learning to get some intuition for how the tools work.

And remember, the tools are changing every year, right? So somebody who maybe today works hard and then, like, gets lazy, suddenly in a couple of years, they won't be able to get the most superpowers out of the AI. But if you commit to doing it yourself, you might be one of the people mentioned in the film who has their own one-person company that is worth a billion dollars, right?

Simply because you were the one who tried to use the AI the proper way to, to build that business or something. So I would encourage people to use the AI as much as possible, figure out how it works, and keep at it because the AI itself is changing so fast.

Speaker 12: Okay. Thank you, sir. Another practical question.

You are have interviewed many brilliant people. What's a counterintuitive co-commonality among those highest successful tech leaders? Is it a specific mindset or character trait?

Steve Hsu: Yeah. So that's a great question. So the people in this film were selected mainly for purely intellectual reasons. So of course, some people in this film are billionaires or even hundred billionaires. other people have almost no money, okay? So we, we didn't select them because they were titans of business. We selected them because of their ideas. And even if you disagree with something somebody said in the film, one could reasonably defend. The way we edited the film, everything anybody says, you could at least attempt some defense of what that person is saying. It's not complete junk, right? Even if you disagree with it. So what distinguishes those people from just people that we meet in the street? Of course, they are highly intelligent.

Everyone we interviewed is highly intelligent. But an almost orthogonal component of personality or psychology, which is not intelligence, is in, in Western psychology, it's called openness. Openness to radical ideas, to thoughts that are genuinely new or shocking to you, or at first you think that is crazy, but you think about it for a while, and you realize, "Yeah, it's not completely.

It might be right, actually. It's not..." Okay, so openness is, I think, a huge component of the success of Silicon Valley and San Francisco and Berkeley. The people there are not just selected for high intelligence. Everyone here at Tsinghua or whatever is selected for high intelligence. I'm not sure how high the average openness is here, okay? But in the Bay Area, everyone is selected for high openness, openness to new ideas.

Speaker 12: Okay. Thank you for sharing again, sir.

Steve Hsu: Yeah. Thank you for your question.

Speaker 13: Although a quick two questions based on the comments you brought on before. we had the pleasure to have Martin Hellman come to campus last semester, and I think he's quite a big advocate on nuclear, like that we're completely disregarding the nuclear threat, as you mentioned before, from the Cold War.

And I do kinda agree with him 'cause we're, we're rearming. We have a lot of conflicts going on around the world. And now we're everyone's focusing on, like this, the next big thing. But this any like this like threat to existence in this planet is still there. That's one question. And the other one is related to this, like new techno capitalist oligarchs that come out of this AI boom.

How will the, let's say, established cause I'm from Latin America. Our oligarchs are basically, they own the mines, they own like these assets within the country, which is a completely different type of wealth than the wealth, this like intelligence or like this super intelligence will generate. So I want to ask like the, like both your opinions on the nuclear threat that is still there and this like the main difference between the old style oligarchs and the new oligarchs that will come, and maybe what the, like this old class will do about it.

Steve Hsu: Yeah. So your nuclear comment is, in my opinion, absolutely correct. So we avoided a nuclear war between the Soviet Union and the United States, mainly 'cause the Soviet Union fell apart. But we could still have a nuclear conflict. So for example, my view of American foreign policy is that allowing this war in Ukraine to continue or to start in the first place is a huge mistake by the Americans and actually elevates the risk of nuclear use of nuclear weapons substantially.

So it's still a risk. If I get out of a time machine, like I get in a time machine, and I get out ten years from now, and I notice the Earth is really messed up, my guess would be in probably ten years, okay? I doubt it's AI that did it. It could've been a super virus. It could've been a nuclear war, right?

So those are two still two probably biggest like civilizational level risks that we face. So I agree with that. Your next question was about the future of oligarchy and, and super intelligence so

Speaker 13: Yeah. Yeah. Yeah. Yeah. It's, it's because I some I don't know. It's hard to explain. Like at least in Mexico, all, all of our billionaires are of course, it's completely different from billionaires in the US.

Yeah. We have like, they own construction groups, they own oil, they own steel, water. Like for example, what, what state-owned enterprise are in China, in Mexico people own. Yeah. Like same as in Russia.

Steve Hsu: So I, I think the great thing about the US system, and to some extent the Chinese system as well, is that most of the people who become super wealthy create a lot of value. They're not just getting control over some chunk of earth and digging out all the iron, okay? They're not doing that. They're inventing things that make millions or billions of people happy and that, right, improve the quality of life. So it is an innovation economy increase, you know, in the US increasingly dominated by innovation and, and also here in China.

Speaker 13: Yeah, yeah, yeah. Yeah. Like the, the how to reframe it? Because these people are still, at least within, for example, Mexico, are quite powerful. But I think their power will be completely disregarded by these like new technologies coming up.

Steve Hsu: Yes.

Speaker 13: I don't want to compete, but I, I don't think they're like I don't see a way in which they can invest, invest in those companies or I don't know how you think about it.

Steve Hsu: So the future can be very subtle. So imagine that those oligarchs in Mexico continue to control the media, social networks, and AI in Mexico, then they can very subtly shape the perceptions of the people. So average people are not sophisticated, so they just kind of read what comes through their social media stream.

And if you have control of that, like you, you own Tele, Telemundial or

Speaker 13: Mm. Yeah.

Steve Hsu: Then, then you can kind of control the perceptions. So, so there is a risk that AI will enable kind of totalitarian control of people's thinking, right? Because people come, become dependent on the AI or their phone for their worldview and their information and, but that's under the control of somebody who controls the cloud.

So it could still be maintained, I think.

Speaker 13: I see.

Steve Hsu: Yeah.

Speaker 13: Hello. So I have like a few observations and like a few questions. I'm a little bit caught up on like these Bay Area people and like the person in the documentary, like Vedanti and you know, Les Wong forum, all that. But I can't help but like also have a bit of concern because I mean, like in I'm now thinking like from the film, then they themselves, like reopened re-like the location, right? How the AI labs, where this improvement and the all these like technological advancements are happening in the same place where these like AI safety people are. And knowing like the Bay Area scene, like they have like a lot of parties and scene.

Speaker 14: And they themselves in the documentary mention that the people in the AI safety, like who's working in the AI safety and people who are working for these AI companies, on the surface it might look like, okay, they are against each other, but I would they mentioned like they are friends, sometimes even lovers.

And like I can't help but feel like maybe wouldn't that like deter their efforts at the AI safety, like the steps they take? And sometimes I wonder like talking about AI in this very extreme godlike manner, like in a very religious manner, might like inadvertently like pump up like these AI stocks because everyone's thinking like, "Oh my God, like they are, they are this powerful. Every I have to get a piece of this."

Steve Hsu: Yeah. Right?

Speaker 14: I didn't talk to these people. So I was wondering what, aside from thesethe public publicity, what real efforts are they doing and like what, what steps are they taking in terms of like real AI safety?

Steve Hsu: Yeah. Great question. So when we started this project we actually were thinking of it as an ethnography. So if you're kind of from the humanities, ethnography means we're studying a particular subculture, particular moment in time. And that was more on our minds than kind of like explaining the doomer worldview.

The movie kind of became explaining the doomer worldview, but initially, and you can see some vestiges of it in some of the interviews, that we were kind of trying to just get, get at the scene. Like what things look like in twenty twenty-six in the Bay Area. You do have this crazy mix of people who are, I, you know, they were early employees at Anthropic.

They're gonna make a hundred million dollars from the IPO, but they themselves are dating somebody who's on the safety team in Berkeley. And they themselves will say, "My pdoom is at least ten percent." So you regularly meet people who are in the labs who will say, "Yeah, I think this is, this is gonna keep getting smarter and smarter, faster and faster.

And yeah, it is a real there's a real existential risk to humanity." So they I think these are sincere beliefs that people have. And if anything, the present, the mixing between these communities means there are many, many people who go to work in the morning and increase AI capabilities, but they are aware that, or at least they've been convinced, that there are significant risks from this.

Now, when you get to the very top, like if you talk to Dario or Sam, I would say the following: they do sincerely think there are risks. They think P doom, probability of doom, is greater than zero, okay? But it's also of what we call instrumental use to them to push this idea about how powerful their AI is because it means a business investor is more likely to accept that trillion-dollar valuation in the IPO.

So, so everything is very complex in, in the bay area right now. So it's one of the most complex and also impactful. These complexities will impact the future of humanity potentially. So that's, that's actually why we were so fascinated to, to shoot the film. Now, a lot of that footage is not in this because we, we ended up just trying to, like, explain the main aspects of the doomer worldview to the more general public, and that became the mission of the film.

All right. Any more questions? Yes.

Speaker 15: I have one more question. So I guess I'd like to return to the question of why there seems to be so much less AI safety research in China.

Steve Hsu: Yes. Answer that question.

Speaker 15: And I'm, I'm not sure if this is correct, but my perspective is that I think to a very practical extent, safety research is a privilege only enjoyed by the frontier.

I think it's like the situation with the tortoise and the hare, where China is sort of like the tortoise. I mean, it's, it's not as slow as the tortoise, but you're definitely behind right now. And if you're somebody working at Kimi or DeepSeek and you're behind, and you see somebody at the frontier, you know, solving Navier–Stokes or whatever already, and your model can definitely not do that yet, there seems to be no real incentive to slow down.

Like, you need to power forward as fast as you can. At least like OpenAI and, and Anthropic have already demonstrated that models of that capacity have not, you know, like, destroyed humanity yet. So why should we slow down and do AI research instead of powering forward and at least catching up to them? So I think from my perspective, one of the major reasons why China doesn't seem to focus on AI safety as much I mean, I think there's some inklings of AI safety starting now, but I think it's because, like, first you have very limited compute, and then also you're not at the frontier yet, so why are you slowing down to do Like, for very practical and economic reasons, like, you would not want to slow down to do safety research.

And so in a way, that puts us in a catch twenty-two almost. Like, if you're in the US, and I think it's also a very popular perspective where researchers in the US are against slowing down because they feel that that would give China a chance to accelerate and then also like, and then China will now be at the frontier.

And so in a way, it's like you need a very coordinated effort, but if one side is always behind the other, like how do you ensure that they will actually agree to, you know, enforcing these safety efforts and whatnot?

Steve Hsu: So I, I agree with your observation that it's a luxury to be able to worry about safety.

It's a luxury for the very top labs at the frontier. And so if you tell me that like the leaders at Kimi or the top engineers at, I mean, Moonshot don't feel like this is their problem, I understand that. But when you talk to them over coffee or over dinner, and it's not work, so it's like, okay, we did all the capabilities work we needed to do today.

We improved DeepSeek ten seventeen ten percent on math pass rate today. We did all that work. It's our downtime now. In Silicon Valley, the conversation would shift to, "Shit, what is this doing to society? Is it gonna run out of control? How close are we to full RSI?" Somehow the conversation doesn't shift that way here.

So I agree with you that as a, as a practical matter for how they invest their main effort, sure, they're not gonna go into safety. But even the mind share of safety seems very low here. Now, it could just be that these two gentlemen are correct in, in thinking like we're very, very far from any actual safety issues, and people here just are better calibrated to that reality.

That could be the answer, but I'm not sure. But it's still a mystery to me why when, when we have meetings, we finish our meetings, which are talking about, oh you know, are theory proving blah, blah, blah is not working. How do we fix this? Blah, blah, blah. But then in the downtime, there just doesn't seem to be as much concern about the issues that are in the film here as in the United States.

Speaker 16: So I also looked in Europe compared to China and US. The AI in Europe is epsilon. Like not comparable. But still, people care so much about security type of issue, like not leaking personal data, privacy, and being very Even maybe it's over, over cautious about AI or whatever sort of type of technology security. I don't know whether, Jimmy, what do you think?

Steve Hsu: These two professors are both in Germany.

Speaker 16: Yeah, I, I'm from Frankfurt. I think in Germany especially, people know that first of all they're still using American frontier models. So even though there's no homegrown models, they're still using frontier models that are very capable.

And second, I think this kind of concern of job security, et cetera, is very much there. I guess also in China, the AI companies know that the government controls everything in the end. And in the US there's still a laissez-faire attitude from the government, but I think it's changing after midterm.

So ultimately, if the people think that we have to control this, if democracy ultimately works, it may not, but if it works, I think we're going to see there was there'll be a power struggle between capital and political will.

Steve Hsu: Yeah. I think you made an incredibly important point here, which I think is relevant, which is that in China, there just may be faith in the paternalistic government, that the government will figure this out.

It's not my job to figure out what the risks are from AI. I'm just building the AI, and the government will figure out whether it needs to be regulated or controlled. In the US, for better or worse, people are much more individualistic, and the tradition is that the people control the government. And so, yeah, you need to think about it, and you need to tell your congressman or your senator what needs to be done.

Speaker 17: So that could be a big part of the, the difference in how much mind share this kind of thinking takes up. Actually, may I add in?

Steve Hsu: Oh, okay. Yeah,

Speaker 17: Sure. Yeah. I, I am an anthropologist. And I was also trained in genomics as we, we talked a little bit before. And I have been following the development in China and US in the past couple of years.

And I think the AI safety community is growing in China. If you look at the WASC, if you look at the, the June conference or even the Zhengzhou conference, the AI safety top the scenes are always there, and there are dedicated organizers. And RSI is on the topic this year. So it's not like people are missing that, but I agree with that.

It's a sense that your feeling is the why is quite different. And I think we talked a lot about before, maybe those concepts, they, they will grow in particular, like the bubble in, in San Francisco. And it, it definitely transited to China and transited to, to the research communities. But how people you know, develop their own feelings in practice especially is a different thing.

So I think this conference is really great to help people to open up. For example, do you know about You must heard Qin Chuan Qian Chuan, right? He, he resigned from his job like two years ago because he want to talk about AGI. And I think he, he resigned from Kimi. So there are people there, and as I mentioned, I was in a, a small like WeChat group developed by AGR.

I'm sorry, safety people. So they're a small community, and, they are really looking forward for young people to join because talents is what is much needed actually. They need young people who share those concerns and want to dedicate it to that to join them. So maybe that's a call.

Steve Hsu: Great

Speaker 18: Actually very similar for what I'm trying to talk because I want to also mention something that actually Chinese community people care about AI safety. So for what I can talk about is just in a small academic community, not in like factory or industry. Though what I know is also only in the mathematical community, there are already people in here, Tsinghua and Peking.

They do care of math sorry, they do care of AI safety. They join in conferences and, I know some like very high-level people in Peking, maybe not here. And some high-level people in Peking, they do care this strongly. They give talks and also there is a possibility because they are in high level, they can write suggestion to the government.

This might be something how Chinese people contribute to AI safety. And this is only for math department and not needless to say maybe for like computer science department, there are even more people.

Steve Hsu: Great. By the way, I just wanna clarify, I'm not judging, I'm not making a value judgment on how much concern there is about safety here in China. I'm just trying to understand the the, the observed differences.

Speaker 18: Sure. I am sorry for my high-speed.

Steve Hsu: No, no.

Speaker 18: speech. So I don't mean to offend somebody or like

Steve Hsu: No, no. You weren't, you weren't offending me, but I just thought I would clarify that. And, and I wanna say again, Suning and this gentleman could be right that we're fifty years away from any real AI safety concerns, right? I acknowledge that that could be the case actually, even though I made this movie.

Speaker 19: Thank you, professor, and thank you for everybody, and thank you for your time. My English was not so good, so I may make a little bit grammar mistake, but I will try to express myself. I'm a computer science student, and I study multi-agent systems, how they cooperate and how they can do better than a single agent.

In the late of the year, I decided to change my research to how AI should collaborate with human in the future. The reason why I choose is that I found all my classmates they can use AI to produce a bunch of AI papers within a very small time period of time. And I feel like if I pursue this kind of working period, and I bring about five to ten AI papers only to bring the accuracy of the model to about five percent, I don't think it's very meaningful for me and also meaningful for people every in the future.

So I want to study the how we human can cooperate with AI, and we can survive in a society, and we both everybody can live rather than the, the movie show in the future. But I want to ask some specific question. Like the case a friend of me, he can use AI to produce paper like in a month. One paper in a month.

But if I don't, don't follow the trend, I care about some important questions maybe it'll be three months I can have a paper, and I don't have the influence that I can have a, a bunch of paper, and I can get a well-paid AI research job. So the question of me, my question is that Sorry can I, can I make some addition?

In the movie, we can see two side of people. One, one, one side of them, they protect the acceleration of AI. They they found that we must to accelerate, we must to beat anybody else and so that we can our model can win, and we can be rich or, or something like that. And others said, "Oh no, this is too terrifying.

We have to make them to stop, or we have to avoid the terrifying future." So I have two question. One is that I, I think the, the, the, the second side is that people think we should stop accelerating AI research. What, what's their power? I, I think this, they are too weak. in many opinion in many situations.

They, they can only think like, it's very terrifying, that we must stop this. But how can they suggest others to do so? They can't provide the actual effects. But I, I think, professor, you, like, you make the movie, then for me, you can help many people realize the pro-possible effects of the consequences.

But how should we actually change the situation? That's the first question. And the second question is, if we cannot stop the trend, will the future become two group of people? One believe that we human is not good. It is, is like weaker than AI, and AI should replace us. I don't, I don't care that AI will become the new human.

And another group of people say, "No, you that, that's too, that's too bad. You are like fight against the human, and we should fight against you." Then the group of people will fight. That's very like and that's not the future I wanna see. So the first the second question is, whether the future will become true, and if so, how should we do in this era?

Steve Hsu: Thank you. That, that's a great question. Thank you very much. And your, your English is fantastic. Much better than my Chinese, so. The first question is, can these people in the film actually accomplish anything, right? They, they seem too powerless, right? Mm-hmm. But actually, on the part of the labs and the venture capitalists and the accelerationists, they're very frightened of these people.

Because if the people can convince the rest of the population that AI is scary Then you can pass laws controlling the number of flops used in training runs, or the number of parameters in your model, or the location of data centers. And all of these things are actually being fought over politically in the United States.

So the analogy I would give you is nuclear energy. So if you do a look back, say seventy-five years of nuclear energy, one country like France gets most of its electricity from nuclear energy and very few accidents. United States gets almost none, and we haven't built a new nuclear reactor in decades.

It's because activists like this in the seventies convinced Americans nuclear energy was scary and bad. Now, if you do a look back, it's pretty clear nuclear energy is actually good and doesn't put carbon in the atmosphere and is pretty safe. It's actually much safer than, than fossil fuel. So, so groups like this can win.

The accelerationist would say we cannot allow them to cripple our AI industry the way that our nuclear industry was crippled in the United States. So, so actually the outcome is not clear at all, I would say. In China, maybe it will be decided at the top level, not by the feelings of, you know, average people.

But in America, for sure, this could be stopped. AI could be slowed down a lot if, if these guys are are effective. For your second point, a big conflict here within Silicon Valley people is this idea of something called transhumanism, which is if you can create something better than humans, does that new thing deserve, have the right to take our place?

And so my, my friend, this guy Matt Southy, who discussed religious things, he said, "This is, this is Luciferian or satan-satanic to say that, like you worship knowledge or, or capability so much that you would prefer that these machines take over because the, the machines will ultimately do more exciting things in the universe than we will," right?

So you say, "Well, they're not human, but they're better than human, and the future of our galaxy is more interesting if we let the AIs take over." Some people feel that way, and other people say, "I my loyalty is to the other apes. I'm a proud ape. You know, I'm just a slightly smarter ape, but my loyalty is to the other apes, and I don't care how amazing, how many proofs this AI machine can do.

I don't care. I don't want it around. Apes have to be in charge." And this is a very, very fundamental divergence of values that is already very present, like in Silicon Valley. People have are on different sides of this issue. And there's no, there's no logical reason you can't be on side one or side two.

It's almost just your aesthetics, like what, what you like, what you value more. And the Christians will say, "If you don't value hum-humanity, you're satanic," right? But who says that they're right? Maybe they're not right So like, that's my answer to your question.

Speaker 19: Thank you.

Steve Hsu: Sorry. Oh, okay. Last one. This is the real last one.

Speaker 20: Yeah. Okay, thanks. I, I just want to have,a faster com-comment of the, the second question that how should we ordinary people do to in this, bad future that AI making everything. So the answer that I think is that just a, a few words that living in the moment.

I mean, in Chinese means 话当吃. I'm a graduate, graduate student studying mathematics, and I have thought about this question when OpenAI has covered mathematics. And I was depressed for like two days, and I was thinking of how should we do in the future that AI is very smarter than human.

I was only depressed for two days, and after that I was I realized that it only matters that you, you do the things that you are happy with. And living in the moment is not a compromise to the AI or to the power or something like this. it is also important before age of AI. It means that finding your own value. It means that you are not valued by this world or by the, by, by everybody else. So that's what I want to say.

Steve Hsu: I'm glad that you were only depressed for two days. I think that's phenomenal. I think Terry Tao is still mentally unbalanced by this. But I agree with you. I think to me, having something around that can develop beautiful theorems, and I can ask it about those beautiful theorems is, is only a plus.

Now, I think economically for you, maybe it's bad because maybe there'll be fewer professorships in math in the future. Okay, so, so we, we have to be honest about all of the consequences. But I think as someone who just loves math, I'm more excited. I'm very excited by the existence of these AIs. Truly. Yeah. Okay. Thanks everybody for coming.

Creators and Guests

Stephen Hsu
Host
Stephen Hsu
Steve Hsu is Professor of Theoretical Physics and of Computational Mathematics, Science, and Engineering at Michigan State University.
© Steve Hsu - All Rights Reserved