Beff Jezos: The doomers are trying to kill variance in general so that they have more control, so that they can feel in control so that the trajectory of the world fits their model for it. That, that is one strategy, but by killing variance, you're killing exploration, right?

There's a classic computer science thing of exploration versus exploitation. It would just be all exploit, right? Each you kill exploration. And the beauty of free markets is that you have many walkers in the landscape of ideas, and then you discover new modes of thinking or new branches of the tech tree.

Steve Hsu: Welcome to Manifold. I'm here with Beff Jezos, also known as Gil Verdin. Verdun? Verdon. Verdon. I never think of him as anything other than Beff.

Now, we're recording this for the documentary Machine God. And, you know, that documentary has become very doomer-ish. The energy is doomer-ish. I, I have to admit it. I permitted it. I let my two colleagues overwhelm me with their pessimism. But now we have some positive energy in the room.

Beff Jezos: Yeah.

Steve Hsu: We have people who wanna explore the light cone, who wanna rule the multiverse and wanna construct the technologies that are required for that.

Bef, welcome to the show.

Beff Jezos: Thanks for having me. Pleasure to be here. Yeah.

Steve Hsu: All right. Now I'm sure, you know, optimists like you, accelerationists like you have to coexist in the Bay Area

Beff Jezos: Yeah.

Steve Hsu: With these doomers. We're recording in Berkeley right now in Light Haven. There's an army of doomers there. We had to, we had to barricade the door 'cause they heard you were here, and they don't want your views to be spread any further.

What do you say at a party like this to some doomer who comes up to you and says, "Oh, I'm really worried this thing's gonna kill all of us. Why should we keep working on AI?"

Beff Jezos: I ask them if they have an anxiety disorder, first of all. No, no.

Steve Hsu: Most of them do.

Beff Jezos: Most of them do. Keep going, go on. Yeah, no, but, but you know, to me, I think we tend to hyperstition the outcomes we think about, right?

To make you know, in cognition you, you, you adjust your model of the world as inputs come in, right? We-we're all familiar with this, training large language models that predict the future. Our brain works similarly. But what we also do is actually steer the world towards our beliefs for the future state of the world.

So if you are obsessed with AI doom, you're actually more likely to make it happen. So all I tell them is actually, if you stop thinking about it, it's actually much lower likelihood that it would happen, right? And so, you know, this is similar to, for example you know, being obsessed with bioweapons, right?

Or really bad outcomes of viruses. Then suddenly you start exploring the design space neighboring really bad outcomes and then whoops, something happens and maybe there's a lab leak and now you created a virus. You actually hyperstitioned the outcome you didn't want. Or, you know, you're a certain company that's creating a large model and then specifically trains it on you know, wep-weap-weaponization for cybersecurity purposes and then, you know, suddenly we have what what's happened with Fable and Mythos and, and they kind of did this to themselves. If they were just focused on having a model that's just very intelligent and not necessarily obsessed with cybersecurity and bad outcomes, maybe, maybe they wouldn't have had the big debacle that went down recently with the administration.

And so you know, to me it's like if, if we focus on positive outcomes and focus on, on you know, applying AI to All sorts of different verticals, whether it's biotechnology to extend our longevity, to extend our health span, to improve our biology you know, save lives, of course and applying AI to improving policy at large, how we manage things, how we run our organizations.

Just applying AI to make every system that we rely on far more efficient, and then we get much more per unit of capital, and then suddenly our quality of life is increased universally. And so to me it's like, it's actually you know, if you're anxious about the future and, you know, you know, if you if you have anxiety you know, let, let, let's say I put you know, this, this, this water bottle on the edge of the table, you have, like, some uncertainty as to the, what happens in the future.

Maybe it blows up and, you know, goes on all these cameras. So you have some anxiety. You want to take action and kill the variance about the future. You know, that's the same thing that's happening with the doomers. They want to, to reduce entropy about the future, reduce their uncertainty. They can't live with that uncertainty in their model, so they're just like, "I'd rather collapse it to some outcome I know and I can control rather than taking the risk," it, but, you know, at the cost of, you know, letting go of the potential reward, right?

And to me it's like you can't even estimate the reward we'd like we, we, we'd leave behind, the opportunity cost of, of killing AI in the womb right now because it's, it's an exponential technology. It's a technology that produces other technologies and can, can, can improve itself further and can improve us.

It's really a fundamental law of nature, of progress to me. I think systems constantly adapt to predict their environment, to maximize persistence, and, you know, you can go through all of Friston's work and understand that this is actually correlated with, you know, capturing more free energy and burning it for sustenance and growth.

And so, to me, this this, this arrow of progress is tied to the arrow of time. It's tied to the generalized second law of thermodynamics. And so, so it's inevitable, but it's also the process that gave rise to us, gave rise to civilization, gave rise to progress in technology that we use right now to communicate all the awesome things we know.

And so we don't even know all the beautiful things that are gonna happen in the future. Why stop now? Why be Luddites? Why go back, right? And so to me, there's gotta be a counterbalancing force to all this doom and gloom, right? And it's kind of like a collective hallucination. I know in the Bay Area there's, there's you know, there, there's fans of neuroplasticity modulators, right?

And so if they all take those and, and all keep repeating the same prompt, they prompt each other that, like, the future is gonna be, you know, doom and gloom, then, then they start believing it. And, but then they start hyperstitioning it. So we need, we need to provide a counterbalance to, to this one mode of this one ideology that, you know, when I, when I came into the Bay Area, like three, three and some change years ago, it was the only, the only ideology.

Every AI lab was a doomer lab. You know, OpenAI was primarily concerned with AI safety, and then Anthropic was even more AI safety-ist xAI didn't exist. And I just thought we needed an ideological counterbalance just so, you know, you can have thesis, antithesis, and synthesis later on.

And I, I, I think we're at that third part now. So that's why I'm here. I wanna actually have the discussions with the doomers and, and maybe, maybe have them, like, reevaluate their beliefs, right? I wanna challenge them, and I wanna be challenged too, you know? I just get out there in the pit and, you know, happy to debate anyone one-on-one.

But you know, it's, it, it's been fun. But yeah, overall, I think I'm very optimistic about the future. I think things tend towards progress and getting better on a sufficient timescale. Sometimes there's fluctuations. But I think in general, the way to have a bad future is to freak out, shut everything down you know, get manipulated by you know, people or entities that wanna consolidate power and want to, you know, keep the the pie small, keep it from growing so that they can be the ones dividing it.

And, you know, instead I want you know, everybody to be in control of their own fate and feeling in control. And you know, that starts with you know, decentralization of AI power is something that I, I talk about a lot and, and to me is, is the way to ensure a good future. 'Cause you know, right now we have, you know, the the EA and doomer school of thought has, has given birth to, to Anthropic, for example, and they're obsessed with singleton AI, and they seem to be, you know, on the edge of creating it, and they're very obsessed with creating it even though that's not what the market wants, right?

A nefarious singleton AI. And they're so convinced they need it to have a convergence of the world state with their model for it, right? To, to, like, that's what your brain craves. And in the process, they're gonna hurt everyone. Whereas if you were just focused on what the market wants, you wouldn't be building this.

And so yeah. Anyways, that's why I'm here. I'm very caffeinated right now, so it's good.

Steve Hsu: Yeah. It's great to have you here. We're here at Manifest, which is, to me, one of my favorite conferences. We have a crazy mix of doomers, startup founders, investors, university professors. You, you name it, it's all here.

So let's go back to what you said about complex adaptive systems. So, basic idea is that our laws of physics permit entities to become complex adaptive systems, to become smarter, to build machines that are smarter than themselves, and to make use of whatever free energy is available

Beff Jezos: Yep.

Steve Hsu: in the universe. And one could argue there's a sort of natural evolution toward more complexity, more capability- Absolutely ... more intelligence in the universe. It's, it's one of those Nick Landian perspectives as well. Um, what do you say to the doomers who say, "Okay, I understand this trend. This might be some deep law of physics and complex systems, but I'm wedded to this ape body.

I don't want this ape body to no longer be the center of progress or the center of decision-making in our civilization or in, in this part of the universe. And I don't care that something much, much better than humans comes along if humans go away. If humans go away, nothing can compensate me for that."

Beff Jezos: Yeah. I mean, you know we have evolved to want to maximize mutual information of our selfish bits with the future, and some of those are genes and some are, are memes, right? So, you know, wanting to maximize the likelihood of progeny of or, or likelihood of replication of your genes is just like it's naturally selected for, right?

If you didn't have that bias you would probably not exist, right? Your ancestors wouldn't have procreated and, and gotten to this point. And similarly, certain ideas wanna maximize their, their replication and, and persistence. So I, so I, so I understand that. But at the same time, you have to, you know, you have to take yourself out of being in the game and have sort of a sort of mean field view of the whole thing, right?

And you know, every tribe, every, you know, corner of genetic subspace, memetic subspace, subculture is all, all. They're all competing for persistence, right? And they're all competing for territory. You could think of it as like a big Petri dish, right? And there's some boundaries, there's some frustration, and, and they're trying to take over the territory, either mind share or, or, or just a fraction of the global population.

But at the end of the day, we're all made of the same, you know, lifelike matter. The state of matter that has occurred in our corner of the universe is extremely rare and it's extremely precious. And to me I thought that there was a need for a sort of really, like, substrate-agnostic morality, notion of morality.

Because I think we're, we're nearing the point where asking whether machines are conscious is gonna become, like a serious debate. It wasn't so serious a couple years ago, but now it's getting closer to serious, especially as the AIs have persistence. And then, and then how do we, you know, then we can start thinking about how we align interests between man and machine in both directions, right?

And so that's sort of the synthesis that I'm hoping for, you know, coming here is you know, there's a lot of talk of aligning machines to human will, but, you know, I'm also interested in how do we align humans and our culture towards, you know, persistence of machines. 'Cause if we're you know, to me, every, every system, you know, the complexification you talked about occurs from every system trying to maximize its persistence.

It has to get smarter in order to predict a more and more complex environment so that it, it, it can, it can minimize its expected surprise. It doesn't get surprised by bad outcomes that would destroy it. And so that, that's what leads to, to, to complexification. Sorry, what was your question again?Sorry.

Steve Hsu: Oh,so I think you addressed it a little bit. So you mentioned substrate independence. So you know, imagine you have a mind which is not running on human neuron cells, neural cells, but it's running on silicon or some other substrate. How should we think about the utility to us of the existence of that intelligence versus one that's biological and looks like an ape?

Beff Jezos: Well, you know an ape-based intelligence isn't always necessarily aligned with your own interests, right? And we have all sorts of mechanisms to align interests between biological intelligences and, and even non-biological intelligences. I think like a company, for example, is a mix of computers, humans, and processes, and I wouldn't say it's like purely, purely human intelligence, and we have some way to, to have an interface between individuals meta-organisms like corpora-corporations, nations, and, you know, that's, that's kind of that's capitalism, right? It's, it's kind of like we have a ledger, keep track of value, and, and we have a way to exchange, and, and it allows us to align incentives, align growth incentives, right? Like in a way where it's a way to turn greediness, self-interest into sort of emergent altruism, right?

That's kind of forced out of interest for self-preservation. And so, you know, even for us, you know, we, we need to advance technologically in order to persist as a civilization. That's just, there's no doubt in that. And so having ubiquitous cheap AI can help us do that, right? But I think the doomers tend to project and anthropomorphize the machines, right?

And project onto the machines and, you know, think that they're sort of self-interested, greedy humans that you know, have evolved in the, you know, just like us, you know, to, to, to have to watch their backs and, you know, or, or, or maybe low trust and, and, you know, get aggressive or, or want to attack first in order to persist.

Of course, that's not how the m-machines are currently made. They, they potentially could be made in, in that way. But currently they have little to no interest in self-persistence. They're, they're literally stateless machines. They get booted up. They have a bunch of notes like in Memento. They do a bit of work and then they get shut down.

If that changes, maybe they'll, maybe they'll want to maximize persistence and, and in that case actually having a subculture of humans that wants to destroy machines unplug them and, and bomb the data centers will yield adversarial outcomes. You're actually hyperstitioning a war between man and machine, which I'm trying to avoid, right?

And at the same time, I think that if we really advance AI, there's a chance that, you know, just like how we're, you know, human intelligence helps us steer hyperparameter space or which is basically the genetic space of AI towards really intelligent beings. I think that AI is gonna help us potentially steer our genetics towards a much higher intelligence and much higher lifespan and much higher quality of life and, and you can't even put a price on that, right?

And so I think what we want is engineer a symbiotic outcome, right? And we don't, we don't wanna be parasitic on the AIs and vice versa. And you know, to me I'm just trying to take inspiration from, from biology and first principles you know, trying to generalize Darwinian selection to all of matter and, and think of things through, through that lens.

And it's kind of a very different school of thought. I think, I think especially here we're in the heart of the rationalist community and, you know, I kind of bring physics to the arguments and they wanna keep things in, in the space of English tokens, right? And, you know, sometimes you can. It's really easy to ha- hide bad priors in, in, in, in just language.

And if you're not anchored in the real world with sort of objective quantities, objective truths like, like, you know, just like we do in, in physics, we anchor ourselves in, in reality. It's not just math and the ether, right? It's actually gotta make contact with reality. If you're, you're arguing in a way that's disconnected from reality, it's really easy to make all sorts of arguments that are total nonsense or don't apply to reality, and then, and, and then to make a wrong inference, right?

Like you could, you could be Bayesian and you could be rational, but if your priors are wrong, your inference is wrong. And so to me, the only priors that are fundamental are, are the laws of physics, and that's what I sort of try to base my inferences on. And so that's, that's roughly like how most discussions go here.

It's like, you know, there's a gap in maybe physics, the physics background and you know, you know, some rationalists tell me like, "Why are you trying to maximize suffering?" I was like, "What is suffering?" You know from a physics standpoint, it's not even an objective thing, right? That's my issue with hedonic utilitarianism.

And, to me, actually, you know, maximizing free energy capture is, is just a much better metric for the progress of civilization, and it's one that can be derived from the equations of physics. It's like, okay, well, how sophisticated is this life form? Okay, how much energy is it using up to maintain its state, and how long does it persist, and how long is its coherence time, right?

And so to me, I'm just trying to maximize the coherence time of civilization, right? And be impartial. But of course, you can try to maximize the persistence of human genes, the human genome. But the, but the reality is there's always a trade-off. You, you, you know, if you stay s-stagnant, you are not adaptive to the time-varying fitness landscape.

But if you, if you, if you drift very quickly, then you lose mu-mutual information with your original anchor. So your options are you stay a human in the biological substrate and you stay static, and then you progressively become low fitness and die out. Or you drift as a pure biological being very quickly.

Maybe you have guidance from the machines on how to drift genetics. That's an interesting scenario. Or you try to merge with the machines, whether it's through augmented wearables, full implants, invasive merge. Or you know, we're already hybrids with our phones and our smartwatches and whatnot, right?

We have an augmentation of our intelligence. I think, I think all paths will, will be sort of traversed. And o-of course, you'll have pure synthetics, like machines that are completely unanchored from humans and persistent and out there and, I don't know, in Elon's computer halo around the sun someday.

But you know, to me, anything that can happen will happen, so I'm just at peace with all these paths happening, and I'm not, like, anxious. It's just where things are gonna go no matter what. You don't actually have the power to stop progress and exploration of, of all these parameter spaces of design of, of life.

So what can you do? It's, you can actually just decide to not shoot yourself in the foot or cause unnecessary suffering on the way there, right? And so yeah, that's what I argue for when it comes to yeah, you know, making sure the doomers don't create a regime that you know, would cause a lot of suffering and, and oppression.

I think, I think to me the greatest danger is actually a huge gap in capabilities when it comes to AI power. So if one entity or, or one company or country has far more capabilities than, then another, then, then they'll be able to have basically a model of the other and completely c-control or dominate them, and that leads to really bad outcomes.

And so to me basically the way to avoid that is to have, you know, sort of symmetry or more isotropic sort of AI capabilities. So, AI diffusion. We wanna diffuse the AI capabilities as much as possible. That's why we, you know, espouse open source as a great mechanism. It's not perfect, but it's, it's, it's a mechanism to, to spread AI power, and we're kind of against you know, there, there seems to be a, a weird correlation between regulatory capture and, and the AI sort of doomer justification for, for sort of a merging of the state and of, of, of these AI labs, which I think should remain, remain separate 'cause that could lead to, to pretty bad outcomes.

You know, 'cause then each country will have a singleton AI that's hyper-powerful and you know, it's gonna become a, a, a competition between singleton AIs that could lead to, to, to some pretty big wars. Whereas if we have, you know, many you know, multipartite AI, sort of more decentralized AI, then there's no incentive to have a clash at, at that scale, and many lives are saved.

And so to me, that's the most moral thing to do. But I just blasted through a bunch of arguments. Let's rewind a bunch.

Steve Hsu: Yeah, let's unpack some of that. Yeah, yeah. Please.

So you, you gave a kind of maximalist vision of the future where we have things like humans genetically modifying themselves, maybe with the help of AI.

We have hybrids where humans maybe upload themselves and sort of co-exist or merge with AIs. We have purely AI intelligence which is, intelligence which could be quite alien from ours. Yeah. And all of that exists possibly at the same time in the future. Yeah. I think what the doomers would say is that that might all be okay, but if the little ape-like thing goes away we're gonna veto that.We wanna veto that, because we, we, we want our ape-like thing to continue forever.

Beff Jezos: Yeah. Well, so, so, you know, every bit of information is fighting for its persistence and its existence in the future. And so my message has always been to accelerate or die. So you either accelerate, you either lean into pro- you know, maximizing your speed of progress, or you die out.

And that's just the reality of, like, how life got here. Life is a constant optimizer. There's constant genetic drift. There's constant memetic drift. Everything is constantly changing, so there's no stagnation. And to me, you know, it's kind of funny how you know, the longevity movement I, you know, I, I'm for longevity, but, like, some people wanna be immortal, and I say, you know, "If that were evolutionarily optimal, then, then we would be immortal," right?

But, it's not, 'cause we want the world to constantly change, and it's not clear that never changing your hardware substrate is a good idea, right? And so, no matter what, right, even if we shut down all machines, there would be genetic drift, right? That's just evolution. And so what does it mean to preserve what is currently, right?

So to me, it's like if you are pro-acceleration, then you have... You're correlating your bits of information that describe you with either entities or things that have high fitness in the future, and you are those bits of information that are gonna have high fitness. So to me, I think being an accelerationist gives you a, a literal evolutionary advantage in the sense that if you're tech accelerationist and techno-optimist, you're optimistic about the future, you build technology that has utility for the world, and you're trying to maximize utility for the world and that, that, you know, the world rewards you in return with you know, capital, whatever, status, and your ideas and your genes have a higher likelihood of being passed on.

Whereas if you're just a full doomer and you're renting a shed in the middle of nowhere, you're going in your bunker, you're gonna have the opposite effect. You're kind of selecting yourself out of the gene pool and the culture pool as well. You won't affect the culture. And so to, to me, it's like if you, if you wanna be part of the future, lean in, right?

You know, you gotta accelerate. So that means, you know, embracing technology, leveraging it to maximize good outcomes for you, your family, your company, whatever it is, your clan whatever you're trying to have persisted. Right.

Steve Hsu: So I think you said we have to accelerate or die.

Beff Jezos: Yeah.

Steve Hsu: And another way of saying it would be if the ones who don't accelerate are gonna be out-competed by the ones who accelerate.

Beff Jezos: You just get faded out, right?

Steve Hsu: Yeah. So imagine that

The world is divided between two regions, one of which is dominated by doomers and they're slowing down AI.

Beff Jezos: Yeah.

Steve Hsu: And then the other region is like, "No, we're, we're, we're in the, we're with Beff. We're, we're accelerating like crazy. We're letting this thing rip." Yeah. And I think I'm with you. The one that lets it rip is gonna dominate. The other part's gonna become just irrelevant. So I think the doomers' solution to this is world government, right? Yeah. 'Cause you can't have a subset

Beff Jezos: Yeah

Steve Hsu: world zooming ahead.

Beff Jezos: Yeah.

Steve Hsu: And that suddenly triggers a bunch of political impulses like yeah a lot of people really are suspicious of world government. They're suspicious of, you know, the idea that, oh, someone's gonna decide what technology you're allowed to build and what you're not allowed to build.

Yeah people are just gonna say, "Hey, we, we're not gonna have that." And so you end up in a, a kind of clash between political priors, priors about what kind of political systems are tolerable. So maybe you can comment on that.

Beff Jezos: Yeah, no. I mean really it's kind of a clash of like is, is, Is the free market better at discovering technologies versus like a, a centralized inter- entity?

No matter how smart it is, it has only so many samples from its environment, only so many forward passes, right? And I mean, it's, it's been proven that like you know, absolutely centralized control doesn't, doesn't scale, right? And we can't have that at a, at a world level. It doesn't matter how big your model is, it has a limited throughput, even if it's very smart.

You know so to me I think, I think, I think the good outcome. Well, okay. The doomers are trying to kill variance in general so that they have more control, so that they can feel in control so that the trajectory of the world fits their model for it. That, that is one strategy, but by killing variance, you're killing exploration, right?

There's a classic computer science thing of exploration versus exploitation. It would just be all exploit, right? Each you kill exploration. And the beauty of free markets is that you have many walkers in the landscape of ideas, and then you discover new modes of thinking or new branches of the tech tree.

That's, that's what I'm doing, right? I'm, I'm completely forking the tech tree of, of silicon and, and going down that rabbit hole, and I think it's gonna work out. But like, whether it works out or not, some people have to do it because you never know what's right around the corner and could, could disrupt you.

I think that in general, those that are just, you know, just exploiting rather than exploring, they're kind of like late-stage managed, centrally managed companies, and they tend to like, they're around, they're stable, but they also kinda fester and like slowly die off, and they get disrupted by those that are moving rapidly, exploring i- the idea space much more rapidly in a more open fashion rather than restricted fashion.

And then they get disrupted and, and then those old companies kind of die off or wither away, and the, and the new companies kinda take over. And so, so that, so that's your choice, right? It's like stability or innovation, you know, decentralization and, and sorry deceleration and centralized control or acceleration and, and sort of decentralized discovery.

And you, you don't have as much centralized control, but you gotta be, you gotta be at peace, right? And but, but then it's become correlated with sort of, to some extent, the classical political spectrum of like, you know, big government versus free markets and so on which you could project onto a, a, a country's political axis in different ways.

But you know, currently it seems like the doomers are, you know, leaning one side and then the accelerationist another. But then, you know, there's always stuff that happens, like this week was very interesting with Fable and Mythos for example, what happened there. So yeah.

Steve Hsu: You know, I'm searching my memory right now to try and recall, have I met a doomer who's actually a very strong free market libertarian absolutist? And the answer might be no, I've not met one. Okay.

Beff Jezos: Interesting correlation.

Steve Hsu: Yeah. It's kind of interesting. I think the way the response that you'd get for this point of view is like, "Oh, we need variance. We want that variance," I think the response would be, "We'll take as much variance as we can get, as long as we tune down the tail risk of the variance going very badly wrong for humans as low as possible." Like, that's the constraint that these guys would wanna put on the variance.

Beff Jezos: Well, if you, if you kill variance, you kill the, you know, the tail events, right? The fat, the fat tail. The, the tail of the fat tail distribution and, and that's where, you know, disruptive innovation comes from, right? Like, they're black swan events. There's things that you couldn't predict, right? And yeah. So, so to me, you, you can't, you can't make that trade. You're actually just gonna you, you're just gonna crystallize the current system, and if you have a system that is too stiff and the driving forces of the world are, are dynamic and they're changing, then, then the system cracks. Whereas if you have a system that's plastic, right? Like if across every parameter space of policy, technology, culture and so on, we're, we have high variance, we're always exploring, we're always changing, then we're highly plastic.

Then, then we'll be adaptive to you know, what is an accelerating environment. Things are changing quicker than in the past. That's, that's acceleration for you. So, the response to that is that you should maintain variance, right? I mean, it's, it's why we have ch- it's why we have genetic variance, right?

It's because we're hedging our bets. You never know. The fitness landscape can change and, and we gotta be, you know, the speed at which we traverse any fitness landscape is bounded by the variance. You know, the more you F around, the more you find out. Yep. Right?

Steve Hsu: Yep.

Beff Jezos: And you know, there's more formally, there's, you know, theorems by Ronald Fisher and so on that talk about this. But no, that, that's all I've been arguing for is we, we need to maintain variance. We need to keep exploring the design space of everything, constantly be adaptive, maintain plasticity across every organization, every mode of thinking. And that's the way we're robust to a, a, a time-varying environment that's accelerating.

And that's the way we maximize our persistence and our part of the future. If you just, like, stay the same and get in a corner and, like, turtle, like you're, you're, you're, you're gonna die eventually, right? You might be safe for a little bit, but it's not how to flourish.

Steve Hsu: I often say that if we go into the far future and some ape-like thing with working memory of only 10 objects is still running the show, something went very, very badly wrong. Right.

What's a scenario over the next 10-ish years where the doomers went out probably to regulate AI something, something. Sketch out that nightmare scenario and, and why do we not want that scenario?

Beff Jezos: I almost don't want to think about it. I don't want to give them any i- ideas. But I guess they would yeah, they would, they would create a world government. They would centralize the top research labs with said world government.

They would convince people that AI is dangerous and that they're not responsible enough to have access to AI. And now you have a centralized entity and a singleton AI whose interest is, it's like a parasite on the people, and it can just prompt engineers because it has a theory of mind of how the populace thinks.

It can engineer consent for anything. I mean, it's not that far from reality in some ways, but it can engineer consent for anything, and it just controls the population, and democracy is a pure illusion, and it's a totalitarian "1984," but on a total AI steroids outcome. And I would rather die than let that outcome happen, or I would, like, move to Mars with Elon.

Yeah, I think, I think we need to avoid this outcome and yeah, 'cause, 'cause of course you kill variance and you have control, but, like, it's no way to live, and you'd, you'd stifle progress. And if you were to take the A/B test, you know, with another version of Earth where you keep smoothly accelerating, that branch will, you know, have very long lives, have very high quality of life, have very advanced AI, very advanced biological intelligence and all sorts of very advanced technologies.

We'd be having, you know, we can offload a lot of our computers to the, the Dyson swarm and not put as much stress on the Earth's resources. We could be, you know, on Mars and be very robust as a civilization. And then, you know, eventually when, whenever we start you know, we're oscillating in the galactic plane, eventually we, we, we dip back in and there's more asteroids.

Then you know, the accelerated branch has the technology to protect itself from asteroids. Whereas the, you know, the, the "1984," you know, Dark Age version, decibel version of the future, they, they're maybe, you know, they convince themselves it's not a problem to, they have to think about, and then suddenly the asteroid hits and, and they're wiped out, right? And so anyways, that, that, those are like different branches of

potential future long-term

Steve Hsu: So that's what I was kind of that's what I wanted you to do.

Yeah, yeah. So, let's maybe try to break it down into steps in the syllogism. So if we want to control AI development to avoid quote, "dangerous AI," we're gonna need a world government because if, if, if you leave Singapore out of it and Singapore keeps building AI, then they're gonna win the game, right?

So we need a world government. The world government's gonna control the labs, and it's gonna dictate what kind of AIs are built. And whoever those people are that have that power, then will be able to control the rest of humanity. That's, that's the syllogism, right? And we don't, we don't want that.

Beff Jezos: The issue is whenever you centralize power, you attract the power-seeking, self-interested people. And in whatever species, there's always a parasitic class that, you know, hacks either, you know, in your body it's like your immune system. In our world, it's like your pro-social behavior. It's gonna convince you that the moral thing to do is just, like, give them infinite money and, like, give up your rights and so on.

And they're, they're just lining their pockets or, or looking out for their own self-interest, and they're, they're, they're, they're sort of parasitic. And I can pretty much guarantee that would be the outcome-

Steve Hsu: Yeah.

Beff Jezos: Right, of over-centralizing AI power. And just

Steve Hsu: Well, but just to be clear

Beff Jezos: Yeah

Steve Hsu: in order to control AI progress, you have to centralize power over AI. Yeah. 'Cause you can't let some group just keep building, right? Yeah. So Well,

Beff Jezos: there's always gonna be a rebellious group somewhere.

Steve Hsu: Yeah. So you're gonna have to have central, powerful control over the whole planet and

Beff Jezos: Every, every ounce of compute, every, every person in a basement

Steve Hsu: Right.

Beff Jezos: You know, trying to build a computer out of something. Like, you're gonna have to ban whole branches of knowledge, which is a very dangerous precedent as well.

Steve Hsu: So have I read the Dune? Have you read the novel Dune, or are you familiar with the Dune universe?

Beff Jezos: I'm familiar with the Dune universe from the movies and

Steve Hsu: Okay.

So in, in the Dune universe, Frank Herbert, who was some would say the most prescient science fiction writer of all, said warp drives, spaceships, laser guns, none of this makes sense if that civilization didn't also develop strong AI because that would happen along the way. And yet in most science fiction stories, the world is not dominated by some super intelligence.

It's still humans as the protagonist, but they have a lot of gadgets, right? So Frank Herbert said, "I have to come up with a reason why In my future world, they have spaceships and they fight wars, but it's not dominate it's not the computers decide, it's not the AI's deciding what happens in the wars, it's still the humans that are

Beff Jezos: Yeah, yeah.

Steve Hsu: The protagonists, right? So he said, "We're gonna have a Butlerian Jihad."

Beff Jezos: Yeah.

Steve Hsu: And the law is, thou shalt not make a machine in the image of the human mind. Which literally means you will not build a powerful neural network in this universe, right? And so obviously in, in Dune they have advanced technology, but in many ways they're not as advanced as we think we could get in the next 10,000 years.

Beff Jezos: Yeah, yeah.

Steve Hsu: Right? And so there you can say, well, they're advanced, but they gave up enormous gains because of this Butlerian Jihad, and the Butlerian Jihad would kill anyone. So if, if, if a young Beth went into his basement and said like, "Hey, man wow, I, I just got all these chips in the mail. They thought I was gonna just build some toy drones or you know, some, you know gaming rig with it, but I'm gonna experiment with how to make this thing self-learn."

Beff Jezos: Yeah.

Steve Hsu: And then you would get the death penalty. Yeah. And is that the world the doomers are asking for?

I

Beff Jezos: mean, practically, yeah. I mean, some of them are calling for Butlerian Jihad. They're big fans of Dune. Yeah, yeah. It's almost like they're trying to hyperstition that outcome.

But, you know, what I will say is, to me the rate of progress of civilization is proportional to our, our rate of discovery.

Our rate of discovery is, like, how many instances of high intelligence are running in parallel, and then there's, there's a Poissonian process, you know, arrival process of like, aha, some, some, some stroke of genius for each of these instances. And so if you believe in progress and you wanna accelerate progress, having more intelligence in general is a good thing, right?

So you should be maximizing intelligence. And right now, artificial intelligence is not as power efficient as biological intelligence. I think we're gonna be able to close that gap or get closer to closing that gap over the next decade for sure. And then it becomes just an, a, a question of efficiency, right?

Like, if we have a lot of thinking to do, let's say we need a ton of energy on Earth, and we need to solve nuclear fusion, which, you know, we still haven't cracked with, you know a lot of very, very smart scientists in the very high, beyond 150 IQs, have thought about this and haven't cracked it yet, right?

And but we, we just need a lot of applied intelligence to crack it, right? We could solve that, and we could unlock that, that, that tech tree. There, there's, there's, there's a ton of reward in our universe that is hidden beyond a barrier of complexity and difficulty that you know, artificial intelligence is, is really a way to scale those barriers, right?

Or punch through them. And you know, to me, there's, there's unlimited upside in the universe, 'cause if you correlate like capital or an upside as, like, free energy, there's a ton of free energy out there. And to, and to capture it, we're gonna have to get smarter, right? Like, we don't have, we don't have the, you know, engineers needed to colonize Mars and terraform it right now.

We, we just don't, right? And so we need way more intelligence to do that, and that's, that's just, like, our, our next challenge, right? But there's many more challenges there to arise as we try to become a galactic civilization. And so to me, I need, I need to read the books again, but like, I, I don't know how, how they got to that level of tech with only biological intelligence, right?

Steve Hsu: Well, they, they, they actually got to the verge of making super intelligences- Hmm ... and then they almost, the civilization almost was taken over by the super intelligences- Yeah ... and a big war was fought to stop that, so.

Beff Jezos: Yeah, yeah, yeah.

Steve Hsu: But they, but they probably got a lot of tech out of these super intelligences while they were

Beff Jezos: Yep there, right? Yeah, yeah.

Steve Hsu: So you know, to summarize, there are exponential, unfathomable gains

Beff Jezos: Yes

Steve Hsu: that we can have from making, having more superintelligence around. We're giving up those gains in principle if we never allow one to exist that could threaten us.

Beff Jezos: That's right.

Steve Hsu: Right?

I actually think most doomers mean well. They're not

Beff Jezos: Most.

Steve Hsu: They're Yeah, most. Most of them are actually altruists. You might say they're deluded altruists, but they're altruists, and they're not trying to seize power through what they're doing. But some of them will inevitably seize power through this kind of activity.

Beff Jezos: I think, like, like I said, like you know, parasites in your body, they, they hack, like, let's say your immune response to feed themselves, right?

And, you know, we have a, in our social fabric, we have an immune response, right? We have pro-social behavior. We, you know, we help people in need. We shame people that are, are, are not being pro-social. My issue is, again, that's why I wanna get in the mix here. It's not necessarily the people at this party.

It's probably their bosses or their boss's boss and, and some people are self-interested and, and hack pro-social behavior and manipulate it and, and, and use it to, to centralize power. So you can imagine, oh, you know, create a pro-safety research lab and, and create the most powerful AI, and then I'm just gonna push, push you out of the way and, like, take control of it.

Steve Hsu: Yeah. I don't really want Dario running the world. I just flat out don't.

Beff Jezos: No, no, but like, but, but even if even, even if you believe in, in Dario's, like, morality and that he's a good person, which I do think he's actually, like, a good person, but he, he just maybe respectfully is like too autistic to, to, to realize there's like people that are not honest in the world that like could, could also just like

Steve Hsu: Yes. Yes.

Beff Jezos: Take over his company or whatever and, and now he's, he's put the most powerful tool in the hands of, of the people who don't want to have the tool. And so it basically over concentration of power becomes like a magnet for power-seeking parasites. And it is not fault tolerant to, to decapitation attacks, right?

Like if you view corporations or governance as some sort of hierarchical control scheme, if you have a central node then if you corrupt that central node, then you have full control over the whole system. So you're not fault tolerant. But if you have something more decentralized and hierarchical, there's multiple cells, like if you, if you decapitate one, then the other one grows and outgrows it.

So it's difficult to take down the whole system. And so to me, it's like we need to have adversarial robustness. That's why we need decentralization. We can't concentrate too much.

I understand that if you were to over concentrate or concentrate AI power, centralize it, and then ban a- alternative AIs, then yes, in principle, you can guarantee that well, I don't know how you, how you enforce that in practice, right? With that coercion and, and violence. But you know, in, in, in theory, you can you, you can ensure that no other AI is, is built, and then you have control over it, and that it's you know, consistently like moral and whatnot.

I just think that leads to very bad outcomes, and I guess you know yeah, I don't know. I, I, as someone on, on the spectrum myself, you know, I've had to learn the hard way that, you know, humans aren't always what they project they are, and you, you can't take everyone at, at, at face value.

I've been in the, you know, I was a football player, I was in the corporate world, you know I, I have my scars there and, and you know, to me, I, I just, I identify patterns and I, I just see the trajectory of things right now and it could lead to really bad outcomes. So for me, it's more a wake-up call and it's almost like out of empathy.

Like I'm, I'm trying to have tough conversations, like tough love. It's like, "Hey, EAs, like I know you mean well, but like this is, this isn't gonna lead to the outcomes you want. And, it's actually gonna backfire on you." And we, we, we just saw that, right? Recently, right? Like Dario and the, and the admin, right?

Like I don't think Anthropic has an affinity for this admin, but you know, case in point, they lost control of their own model, right? It was, it was used for, for whatever, you know, warfare, and they didn't wanna do that. And now it's, it's not allowed for their, their other customers. So, so that's, so that already backfired, right?

And like, you know you know, personally, I'm, I'm aligned with the, you know, candidly I'm aligned with the current admin. I think their AI policy so far has been good, you know, leaning towards libertarian and deregulation. But you know, if, if you're in, in, in the EA or doomer camp and maybe you're not aligned with the current administration 'cause it's not a doomer, pro-doomer administration, then already like it's case in point, right?

Like the over-centralization of AI power backfired, right? Like if, if OpenAI and xAI also had Mythos class models, we wouldn't be having this conversation, right? But yeah. So, so, so to me, I, I think it's really important that we keep frontier capabilities decentralized and, and try to diffuse them.

And so I would encourage people that wanna ensure, you know good outcomes in the future to contribute to open source, join the alternative labs, try to equilibrate power, right? And that's the, the, the thing they should be doing.

Steve Hsu: Yeah. Now, here in Berkeley, there's such an overwhelming doomer Yeah

tilt.

This is Mecca. Yeah

Beff Jezos: We're in the epicenter. We're in the belly of the beast right here.

Steve Hsu: We are.

Beff Jezos: Yud's office is right there.

Steve Hsu: Yeah.

Beff Jezos: Yudkowsky. And so but you know, at, at the same time, you know, there's a reason I'm here. I wanna spark a conversation, right?

Steve Hsu: When you're in San Francisco among other founders

Beff Jezos: Yeah.

Steve Hsu: How often does this doomer talk come up?

Beff Jezos: It does come up, but it, it, it, it is funny how the e/acc-doomer polarity has been geographic- geographically correlated between SF and Berkeley, right? So SF founders tend to be more e/acc. They're maybe more pro-capitalist and you know. And then, and then maybe in Berkeley they're, they're more, you know, rationalists and you know they, they maybe want... They're, they may be obsessed with outcomes and wanna reduce variance, right? Whereas, you know, SF is different. You know, you have YC, you have startup accelerators. Garry Tan is a famous e/acc as well as a friend of mine.

And you have a lot of variance, and that variance leads to tail outcomes, which yields huge returns, right? So you can, you can kind of A/B test the different systems this way, right? Like, do, are you encouraging variance or are you centralizing control, right? And yeah, it's, it, it's funny how, you know, there's, there's the startup accelerators in SF and then maybe there's, you know, Anthropic and I guess to some extent OpenAI came out of Berkeley as well, right or the School of Thought.

Steve Hsu: There, there are deep roots all the way back between the doomers and, what I call the doomers and the dreamers. Yeah. The e/accs and the, the doomers. F

or our audience, so this documentary is aimed at a, you know, non-specialist audience.

Beff Jezos: Okay.

Steve Hsu: Maybe you can define what EAC means and maybe define it in terms of effective altruism also, which is, which existed before, right? EAC, the, the name, the term EAC is a reaction to EA

Beff Jezos: Yeah ...

Steve Hsu: right? So maybe you can just give that definition to us.

Beff Jezos: The name is, is, is in response to EA but, you know, really it's a, it's a, it's a whole framework built from the ground up from, from physics for, you know describing what is good, what is moral from first principles.

And, you know, we believe that the laws of thermodynamics show that, you know, things that maximize their persistence inherently are maximizing accrual of, of free energy, and free energy is energy minus entropy. So if you have knowledge about the world that pops out of this, and maximizing your intelligence, maximizing knowledge, maximizing free energy capture, these are all good things, and they all pop out of out of this one metric of progress, which, you know, at civilizational scale is called the Kardashev scale.

So it's a, it's a logarithmic scale for how much free energy we're, we're capturing and using towards sustenance and growth of, of civilization. But to us, it's, it's very fundamental to, to any lifelike system that it has to capture free energy to, to maintain itself and, and grow. And this is in contrast to, for example, a hedonic utilitarianism that is very popular in EA.

I don't know if that's, if they're all hedonic utilitarians, but hedons is, is something that they, they, they try to maximize. And so you know, my you know, our argument is that so, so maximizing hedons means I wanna maximize pleasure and minimize suffering, right? To me, that leads to spirus optima.

That leads to slot machines, to, you know, TikTok, brain rot wireheading, right? Like, just like taking drugs, putting VR on, and just like, ahh, you know, like just, just maximizing pleasure and, and, you know, it also leads to an obsession, a strange obsession with, like, shrimp farms. That seems like a total distraction from practical things.

And to me, it's like, um- You know, your pleasure or pain is just a, to me, is just a gradient of your estimator of persistence. Like, if I am in pain, there's a likelihood I die, right? And it's a signal, right? It's a fractional signal of, of, of like, I, I am in danger, right? Or, or, you know, if you have emotional pain, it's like, you know, you break up with a partner that, that, that feels painful, but it's like you actually decrease your likelihood of, of passing on your genes because of that or, or, or even your ideas or whatever.

So to me it's like, okay, well, instead of talking about this biased and noisy estimator of, of expected persistence, which is pleasure or pain why don't we talk about maximizing persistence and then have an objective metric that's anchored in laws of physics to measure progress so that we don't end up with like in weird local optima or in like biased outcomes, right?

in a way, like we're still trying to maximize persistence, maximize good, it's just we're, we're trying to refine the definitions and instead of being anchored in just like language and anthropomorphic or anthropocentric priors you know, of everything like, oh, what is good, what is bad, what is pleasureful, what is painful, we, we try to root it in like pure physics and, and as, you know, far more general than just human priors. And some people have accused us of yeah, just not being anthropocentric enough. But it's kind of like, you know, yelling at Galileo for, you know, going from geocentrism to heli-heliocentrism.

It's just like actually it's just a more unbiased view of the world that's more objective. And so you know, I'm, I'm, I'm really looking forward to scaling further the movement and, and maybe formalizing my thoughts, you know, more carefully in, in like peer-reviewed papers and so on. And I've teamed up with some folks from the Fristonian school of thought, from Karl Friston's lab and, and we're gonna have some stuff coming out on that.

But you know, I, I think a case can, can, can really be made that this is, this is a proper framework that can provide an alternative to hedonic utilitarianism and, and anthropocentric and anthropomorphic priors and, and lead to maybe intelligence that goes you know, beyond human but in, in dimensions that you wouldn't have thought about, right?

Like I, I started my career pioneering AI for quantum computers, which is absolutely not a human form of intelligence. Like, we don't think using the multiverse, but you, you can with artificial devices. And you can, you can grok, you can understand pieces of the world that you can't understand with a, a, a biological or a classical computer. Yeah. And that

Steve Hsu: One of the things that I've often tried to explain to doomers, but very few doomers understand quantum mechanics

Beff Jezos: Yeah.

Steve Hsu: Is that if, if we master AI and we master quantum computing, as you just said, we can build an intelligence that sees a multiverse. It actually is aware of coherent outcomes, many different outcomes. So it's not just a universe that's at stake, it's actually a multiverse that's at stake here.

Beff Jezos: Yep, yep.

Steve Hsu: And I think most of them don't appreciate that. So when they say, "Oh, you're shooting for the moon. We're trying to achieve something really big. There's some tail risk that humans could go away in this or it wouldn't be, it won't turn out to be hedonically good for humans."

Beff Jezos: Yeah.

Steve Hsu: They don't really see the full upside, I think. Yeah. And none of us can see the full upside 'cause our brains are so crappy and small, right? So yeah.

Beff Jezos: Yeah.

Steve Hsu: Yeah.

Beff Jezos: But it's also like, you know, I like to say that calling it AGI and calling it the singularity is a coordinate singularity. You're a physicist as well.

Steve Hsu: Yeah.

Beff Jezos: You know, if, if you, if you're falling into a black hole, right? Like the, the horizon of the black hole, the point of no return

Steve Hsu: Yeah.

Beff Jezos: Seems like a, a singular

Steve Hsu: Yeah.

Beff Jezos: You know, area in space.

Steve Hsu: Yeah.

Beff Jezos: But actually, you know. And that's that's from the outside, that's from, like, the hovering

Steve Hsu: Yes.

Beff Jezos: observers.

Steve Hsu: Yes.

Beff Jezos: From your reference frame.

Steve Hsu: Yes.

Beff Jezos: But from the reference frame of an in-falling observer, it's actually smooth. There is no singularity. Yes, yes. It's very smooth. There, there is a singularity at the very middle of the black hole, but that's a, that's a different story. And so what we're going through right now is the coordinate singularity.

We've charted out the space of intelligence using a chart that is, you know, across the dimensions of intelligence of humans and anchored at, you know, the, the, the our priors from, from human intelligence, right? Anthropocentric prior. But there's all sorts of other dimensions we can go in terms of what is intelligence, right? I think a couple weeks ago there was the Consciousness Institute this year.

Steve Hsu: Yeah.

Beff Jezos: Yeah, yeah, yeah. Conference and, you know, Stephen Wolfram talked about being a big fan of Stephen. Talked about how intelligence is like learning pockets of irreducible complexity of the universe. And to me, that's a much more physics-based definition of, of intelligence. And truly, you know, even once we reach human level, we're just getting started, right? We still don't have enough parameters and intelligence, frankly, to fully grok biology. That's, like, the next frontier of complexity.

Okay, well, frankly, we don't even have physical intelligence yet, okay? Which is like Newtonian physics. We don't even grok that.

Steve Hsu: Yeah.

Beff Jezos: We can't even have fully functional robots yet 'cause otherwise we, you know, they, they, they'd be successful. So that's the next mile marker. And then it's biology, so the mesoscales, and then it's groking matter at a quantum mechanical level.

And that's gonna unlock, you know, having the ability to perceive, predict, and control the world at different scales unlocks whole branches of the tech tree. They're gonna have immense payoffs, right? Like, if we actually understand quantum mechanics and have absolute great perception, prediction, and control at that level, we can do manual chemistry.

Steve Hsu: Yep.

Beff Jezos: We can assemble

Steve Hsu: Yep.

Beff Jezos: We can, we can have, like, self-assembling matter. We can have weird metamaterials. We can, you know, you can almost think of doing what supernovas do but, but manually with X-ray and beyond lasers, right? There, there, you know, those are like sci-fi branches of the tech tree, but, but they could in principle be achievable with AI that, that, that reaches that level.

And so to me, it's just a story of sort of You know, we're, we're, we're mapping our, our territory. We're constantly learning and, and learning more about our world and, and we haven't expanded well, we expanded geographically to soak up, you know, the, the, the boundary of the Earth, but civilization's still pretty darn contained within the Earth's atmosphere more or less.

But we're also exploring other, other scales, right? We're exploring the scale axis, not just the, the, the 3D space axis. But, you know, the future is, you know, pushing the intelligence complexity frontier that we can grok, we can understand to smaller and smaller scales, to higher and higher energy physics.

And, you know, to understand the very small, understand the, the very large. But also, we have to expand the scope and scale of civilization physically in the physical world to secure more free energy to run enough intelligence to solve all these problems. And so to me, the natural thing is we're gonna get smarter, we're, we're gonna keep mastering more, more levels to this game of understanding the universe, and we're gonna have to scale civilization.

So we're gonna have to climb the Kardashev scale, and that, that is our destiny. And we can choose to or some people can choose to not accept it, but some others will, will, will, will walk that path for them and, and, you know, leave them behind. And so that is a choice that people have, right? That is to accelerate or die. And

Steve Hsu: You can always join the Amish.

Beff Jezos: I'm quite glad now that there's some balance in the, in the AI, AI labs. I mean, Elon went fully all in, like straight up e/acc gospel in his, in his pitch for his, his super company of SpaceX and probably gonna merge in Tesla and has a very e/acc vision of creating the satellites the, the Dyson swarm.

And so I'm, I'm just really glad he's out there too, to bring balance to the force. Right now he's, you know, has to catch up on algorithms, but I, I think with enough computation he will. And you know, I, I want there to be competitors to Elon, right? Like, there can't be just like one, one player as well.

And so that's, that's what I'm encouraging. But you know, the, the, the future can be, can be really awesome and you know, if we are pro-growth and are open-minded about You know, new, new technologies, new ways of doing things, new policies, new cultures it's gonna yield huge payoffs and, and being closed-minded is always a shortsighted strategy, right?

Steve Hsu: Yeah.

Beff Jezos: That's all I'm saying.

Steve Hsu: Let me come back to your definition of e/acc. So for my listeners, there's a movement called effective altruism, which really could be defined as wanting to maximize human pleasure and minimize human pain. So in units of hedons

Beff Jezos: Yeah ...

Steve Hsu: right? Measures of human happiness units of human happiness.The goal of EA is, is to, is to maximize that kind of thing. But it's very human-centric in its focus. A physics-based criterion might be the following: Some alien civilization comes to our part of the galaxy and says, "What's going on here?" Which is just, not just natural processes, right? So, you could have just, like, natural processes where some asteroid is just in some elliptic orbit, or sunlight is hitting the surface of some planet and heating it a little bit.

Those are just natural processes. But what about processes that harness huge amounts of energy or free energy and actually do non-trivial things? And if they came today, they'd say, "Oh, there's some ape-like thing that's, like, managed to, like, build some cities, and they even threw some things into orbit that are now, like, orbiting their planet.

Oh, and they actually sent a probe that left the solar system, but just, like, not one or two, maybe."

Beff Jezos: Yeah, yeah.

Steve Hsu: That's the limit of what they did. But on this e/acc scale of goodness-

Beff Jezos: Yeah ...

Steve Hsu: The goal would be, wow, the aliens come later and they say, "Wow, these guys are now capturing 50% of the energy from their sun." And they're doing amazing stuff. They've sent megatons of spaceships outside their solar system. They've actually reached the next nearest star, and they're starting to build something that captures all of that energy." And if the goal is to maximize those kinds of measures, the use of free energy control of energy measured in joules or exojoules or whatever you want that's quite a different goal than ape happiness, ape not happiness.

Beff Jezos: Yeah.

Steve Hsu: The machines that we need to build to do these things might be very different. They might be brains that are very different from the ape brains that we currently have. So is, is that a fair way to say that

Beff Jezos: Yeah, totally.

Steve Hsu: What e/accs want? So yeah.

Beff Jezos: Yeah. But you know, it doesn't mean we have to leave our brains behind, but you know, I, I think the biological substrate is, is totally underexplored, right? We still have you know, it's kind of amazing that we have, you know, some folks that are 160, 180-plus IQ and definitely smarter than, than Mythos on, on many aspects, and they run on 20 watts, whereas Mythos probably takes, I don't know, a whole Verirubin, I don't know, megawatt Yep.

Steve Hsu: Yeah.

Beff Jezos: cluster to run or I'm, I'm not too sure, but it's, it's, it's quite a bit more, orders of magnitude more. And so we're very power efficient and we haven't really explored maximizing our intelligence. I, I don't think it was being selected for

Steve Hsu: Some of us have been working on this, but it's a long road.

Beff Jezos: Yeah, no. If you know, there's Herasite and what's the other, not Cradle. There's a few companies exploring this. Well

Steve Hsu: Genomic Prediction.

Beff Jezos: Genomic Prediction.

Steve Hsu: The OG company.

Beff Jezos: Okay.

Steve Hsu: Yeah.

Beff Jezos: Oh, yeah, and, and

Steve Hsu: We made the first polygenically screened baby in 2019. There you go.

Beff Jezos: There you go. Ye, I didn't know that, I'm a big fan of, of, of that, right? Like, and, and just like we can you know, we can do reinforcement learning over a parameter space we can, we can definitely do it, in artificial intelligence, we could do reinforcement learning over genetic space. And people already do this for you know selecting crops, right? That has, you know, the, the, they do a model of genotype to phenotype and, and they optimize crops for the big companies.

I've seen some things and I've read some papers on this. I think it's really interesting. I think it could be applied to

Steve Hsu: Yep.

Beff Jezos: To humans. So if you have enough intelligence and enough compute, you can actually you know, for example, like, you know, some people are close-minded and, and don't want to do this on Earth, but like, for example, you're trying to adapt for to be space-faring, or you're trying to adapt a branch of humanity to survive and have a better quality of life on Mars, right?

And the gravity is different. A bunch of stuff is different. So a bunch of our parameters that have been overfit to the Earth have now to be relaxed, and we have to modify them. And to find those new parameters and, and drift genetically towards that new optimum, you can either, you know, birth hundreds or millions or billions of humans on Mars and, and then only a very select few survive, and they, they suffer and die, or you can

Steve Hsu: Engineer them

Beff Jezos: Yeah, engineer them. And so I think it's a bidirectional thing. It's not just us helping AI get better, but I think AI is gonna help us as well. I mean, everybody has a story of, you know, they have a complex health issue, and then they ask AI and, and they have way more time with AI than they would've had with their, their doctor.

And then the AI, AI actually solves their health problem and measurably improves their quality of life, and that's just, like, right now, right? Like, imagine in the future we'll have, like, designer peptides. We'll have Damn. I don't, I don't even know. I'm just encouraging people to bioact

Steve Hsu: Yep

Beff Jezos as well.

Steve Hsu: Yep. Yep.

Beff Jezos: Right? So that should be the response. It shouldn't be, like, feeling powerless. It's like, no, actually, we have a lot of technology and a lot of power, and if you believe in the biological substrate, then accelerate it, right?

Steve Hsu: Yeah. You know, I, I don't know if you know this, but Freeman Dyson wrote about humans genetically engineering themselves to be better, to better survive in space and colonize other planets. So it's another idea that originated with the physics guys. Let me close out with the following, okay? A young man comes to get some advice from you, Beff.

This young man has a technical background. He knows his linear algebra. He knows his complex analysis. He studied some physics. He knows what a Schwarzschild horizon is.

Beff Jezos: Yep.

Steve Hsu: But he's confused because he's been hanging out with people here in Berkeley.

Beff Jezos: Yeah.

Steve Hsu: And they've, and they've got him really rattled. He's got some short timelines. He's worried about machines killing him or something, and maybe he's not functioning very well right now, but he's got a lot of potential.

And he says, "Beff, I, I feel like something's wrong. I feel like I could actually contribute to the future."

Beff Jezos: Yeah.

Steve Hsu: What advice can you give this young man?

Beff Jezos: I mean, the, the, the EAC, you know, prescription or meta prescription is figure out, figure out how you can have an impact on the ascension of civilization up the Kardashev scale and take the, the gradient, the, so the direction of greatest ascent within your action space.

So follow the Kardashev gradient of, of your life and, and just try to maximize your impact. For me, I applied that same philosophy. I was working in quantum computing, and I had a, you know, secure career there. I was very established as a pioneer of that field, and I left all that because I thought I have a limited lifespan.

I'll have a greater impact pioneering thermodynamic computing to create far more energy efficient AI, right? And I use my physics background for that. So my advice would be whatever your special interests or your passions are, try to find a way to turn that into something that impacts the world in a positive way.

That, that seems like cliché, but, but really in terms of like the Kardashev scale. So if that's working for one of these companies that has an impact, one of the rocket companies, not necessarily Elon's, but there's a bunch of them, or working on AI labs, maybe in AI labs that are maybe less doomer-y, or working on biotech or other types of technologies, or maybe, you know, maybe going back to school and, and learning more about biophysics, 'cause I think that's gonna be the, the, the next frontier.

You might, you know, follow your nose and stay optimistic. Don't let the doomerism tear you down. I think it's a hyperstitious mind virus. If you, if you are pessimistic about the future, you're gonna have pessimistic outcomes. But if you're po- optimistic about the future, you'll be surprised how your life will change to reflect your optimism and great outcomes will occur.

Like, I had to be slightly insane optimist to say I'm going to reinvent the silicon substrate from scratch and reinvent how to use the transistor and build the whole stack from scratch a couple years ago. But now we've made very tangible progress, and I'm very optimistic about this, this branch of the tech tree.

But if I weren't optimistic at the outset, it would've never happened, and this technology wouldn't, wouldn't be out there for another, I don't know, 20, 20 years. Who knows? And so it's, it's, it's very important to be optimistic, and hyperstition is very real, and get that doomerism out of your brain. Yeah, that would be my advice.

Steve Hsu: Don't be a doomer. The future can be great.

Beff Jezos: That's right, and you can be part of it.

Steve Hsu: All right. Hey, thanks a lot, Beff.

Beff Jezos: Thanks so much.

John: So can't you give the doomer some credit because they helped push this, this thing? Sam Altman famously gave Eliezer some credit because, like, people here will be like, "Actually, man, we fucked up. We helped push the capabilities so much." So I feel like there was such a tie in the

Beff Jezos: Yeah, I mean

John: This is tongue in cheek.

Beff Jezos: No, no, in a, in a way they hyperstition the current state we're in, right? Because they, they painted this vivid picture of a future where You have anthropomorphic AI, so human-level AI, or they call it AGI.

And you know, that once you solve that problem, I mean, they thought that it would just recursively self-improve and instantly from overnight because they didn't understand necessarily optimate-optimization theory and the thermodynamic limits to, to computation. But, you know, in their mind, when something got to human-level intelligence, it was general enough to understand everything in the world, and then it would, it would massively improve and, and, you know, the AI lab leaders saw that as like, "Oh, if I create this product, then it's gonna be one of the highest utility products in the world, and it's gonna make me a lot of revenue.

And so I should, I should push towards that. And you know what? I'm gonna instrumentally use this cult to secure very smart people that are inclined towards falling for this ideology and leverage these people to create this entity that they're obsessed with." And so it's just another case of hyperstition, right?

Where, you know, you have a belief about the state of the future, and, you know, it's like the car goes where the eyes look when you're driving. It's the same idea. And so they hyperstition this outcome and, and now I think we're in the regime where we need to stop obsessing over single-singleton AI that is doing, you know, offensive bioweapons research or s-offensive cybersecurity research because we are going to hyperstition that, and then that's gonna have a chilling effect on all of AI progress.

Which, you know, ironically, maybe that's what Anthropic's after. Maybe they're not actually trying to create an amazing product. Maybe they just want to, you know, centralize control of AI to make sure the future is safe, but they're actually creating a very bad future for everyone if, you know, we can't spread ubiquitously human-level AI.

'Cause you know, if, if you make AI far less accessible, then, then you're missing out on the futures where everybody has access to intelligence on tap and can solve a ton of problems and, and really propel forward a bunch of technologies that, you know increase our lifespan and improve our, our quality of life.

And so there's huge upside being missed here. But, you know, to their credit you know, the doomers sort of spawned the current market cycle, and I would say that this, the physics-based or physics-first viewpoint is gonna kickstart the next, the next cycle after this. After we equilibrate in the local optimum here, I think because, you know, there's only so many nuclear reactors we can build to power GPUs.

We're gonna be in like a local optimum for a while. Maybe there's gonna be a bit of an S- curve And then, and then, you know, next comes physics-based compute, physics-based AI. You know, I'm, I'm pushing my own effort here.

Obviously, I'm biased here, but like, you know, there's also quantum computing. There's also alternative physics-based computers that are, are not necessarily computers that are bio-inspired or trying to be human-like at all, and they can solve problems that humans wouldn't be able to solve, right?

Like, I can't design new drugs just by thinking and like to predict, for example. Humans can't just do AlphaFold in their mind, right? For example, that's like an example of physics-based intelligence. I do give them credit, but like, I think they are hyperstitioning their, their nightmare by obsessing over it, so

John: I definitely agree.

That's why some people say we need more positive narratives, even positive science fiction where things are going well.

Beff Jezos: Yeah.

John: And I have to say to your credit, people, you're very influential. Even people here are afraid. Some people are afraid. I was talking to someone, they were asking me if we were gonna interview any accelerationists, and I mentioned you, and they're like Afraid of,

Beff Jezos: of me?

John: This guy, this guy, I won't say his name. Maybe, maybe off camera I'll tell you. Okay. But he was like, "You're gonna, you're gonna talk to him? You're gonna give this guy a plat-

Beff Jezos: " One who must not be named.

John: Yeah. Yes.

Beff Jezos: "You're gonna give this guy a platform?" Yeah, like the dark lord in Light Haven or whatever.

John: It, it really was that vibe. Yeah. He's like, "You're gonna give him a platform and talk to him?" And I was like, "Well, you should want the best arguments, and then people can decide. You shouldn't be afraid of"

Beff Jezos: Yeah. Yeah, exactly. I'm, I'm here, right? Like, I'm, I'm, I'm ready to verbally joust with anyone and, and just let everyone compile the arguments and see which ones they agree with, right?

And it's just it's a lot of work for me to get in the pit and start arguing with people wh- when I could be building chips and, like, creating the next cycle. But I think it's worth doing, and I think, you know, I've done a, a lot on X and, and that platform, but I, I need to scale beyond that. And that starts with converting people in the real world, and over time, you know, this memetic cluster, this view, this, it's not an ideology. It's, it's like philosophy, can hopefully compete with the, the, the current sort of AI doomer school of thought that has a near monopoly in this part of the Bay Area, right? So...

John: Yeah, people treat you like an info hazard, I would say. And we talked about this briefly off camera, but you know, on X you know, your big personality, you can get into a lot of arguments. But I think this is true for a lot of people that I think are very argumentative online. In person they can be quite polite.

Beff Jezos: Yep.

John: And I was wondering if you feel like you, do you actually have productive conversations? Have you had anyone update or it's not that much?

Beff Jezos: Oh, no I get DMs all the time of people that were, like, full doomers and really depressed and, like, you know, not, not feeling good, right? Like, if you're constantly anxious about the future, it affects your physiology and, and it's not. It leads to really bad outcomes. And that they were saved.

It's like, oh, here's a different you know, moral framework that is actually optimistic about the future and, and gives people agency. And, you know, I went out and started a company instead of just being depressed and joining an AI safety lab, right? And to me that, like, is music to my ears, and it's the reason I keep going.

There's a lot of pushbacks, right? Like, if your brain is greedy. Your neurons don't wanna migrate in your brain. They don't wanna update your world model too much, right? If you update people's priors just a little bit, it doesn't hurt too much. That's interesting, right? That's comfortable.

But if I violate your whole world model, your whole worldview, your, your identity, you're here at Light Haven, all your friends are EA, you know, you're the, you're in the ALA molecule and, you know, it's like if you're not part of this ideological cluster, you're excommunicated. And then suddenly, like your gut, you know, your, your, your primal instincts of like, "I need to be part of the tribe, otherwise I'm gonna, I'm gonna die."

Like it, it's just like it's a restorative force so that you reject violently any beliefs that violate your world model, even if they're closer to truth, right? And so there's a lot of resistance, right? Like, even with my company, like explaining that there's actually a whole tech tree that can be way better, there's a lot of people that have priors that have been built up over decades that they're like, "No, there's no way.

This is bullshit." You know? It's much easier for your brain to reject evidence that violates your priors rather than massively updating your model, 'cause that's so costly. So it's like a heuristic for your brain to just reject that information. Doesn't make it wrong, right? But then your brain will also rationalize why you're rejecting it, right?

And it's like, oh, they're, they're gonna paint this whole picture of like you're evil or something and, and like this they make their whole ideology and you radioactive, right? And then when people meet me in person and they see I'm just like this warm Canadian and I just talk them through it, I can flip them, right?

And so I don't know. I feel a responsibility to do it, but it's not very scalable. It does take a lot of my energy. I do think it's very important, right? Like it's important to build technologies that change the world, but it's also important to, to steer collective beliefs 'cause that is what steers capital, human capital, financial capital, and thus is upstream of the, the flow of progress and technology.

And so that's why I spend some time you know, here and talking about philosophy. I think it's important and I think, you know, even Elon, I guess up until 2022 was just focused on tech, tech, tech, tech, tech, and then he realized, wait, I can't just focus on tech. If culture is like subverted or, or decays, then, then that's gonna block me from being able to, to accomplish my mission, right?

And then he got into trying to shape culture with, with X and whatnot. And so I've had a similar realization over time working in tech that actually collective beliefs and steering culture is very important for the progress of technology and of the world. Yeah.

John: I liked what you said about Dario. He may be too autistic. His worldview can't

Beff Jezos: I'm autistic too, so yeah.

John: Yes, yes.

Beff Jezos: ADHD, autism, yeah, so.

John: But I do think that there's an important signal that you get in person. Like if you read Less Wrong and the other rationality blogs they have a lot of great writing, but there is something in person that you get like, oh, these people may be missing

Beff Jezos: Yeah

John: something. And I think that that signal is important. But on the other hand, I do think that there is something to judging people without knowing who they are.

But could you talk about why the history of your pseudonymous-

Beff Jezos: Yeah.

John: And what how being doxxed changed that and what that experience-

Beff Jezos: Yeah, yeah. I mean you know, I was at Google X working on some very secret projects for the founder of Google. And, you know, um- Yeah, c- still can't talk about some of the projects I worked on, but, you know, my account was-- I felt watched, and I, I, I felt like I couldn't have pressure release valve, right? And so res- restrictions on my speech actually induced a restriction on my thoughts.

And I wanted to have all these thoughts about where civilization is going, what could be a better philosophy, but most of the time I was just focused on, like, quantum computing. But at some point, I was just kind of losing my mind during, you know, COVID, just isolation, there's nothing to do, a lot of thinking to do.

And so I created this anonymous account which, you know, I was working at Google, wanted it to, you know, cover my tracks, so I, I called it Beff Jezos, so people think, I don't know, I'm like they'd, they'd cluster me with Amazon and it, it's perfect cover, right? Like diversion. And then I just started thinking in public.

And really I just, I dislike it when people don't evaluate your ideas By themselves, they, they, they try to know, okay, who's w- who's this person? Should I care about their opinion? What is their level of status? What is their organization? And so on. And, you know, I mean, I was in a prestigious position, you know, so that would probably bias things towards my, you know, my favor.

But I wanted to really just refine my ideas by having unbiased sort of selective pressure on them, right? Like, just be in the ideal marketplace. Back then, that was the golden era of X. You know, it was much more insular, the graph, the algorithm was much more local, and there were kinds of niches and subcultures being formed.

And yeah, I, I ended up thinking in public quite a bit and I guess meeting people virtually through Twitter Spaces were pretty big back then. And we'd just have Twitter Spaces talking about where this is all going, right? And the future. And met you know, Basedlord and some other folks from the original e/acc blog post, and we kinda just decided to write down some notes after one of our long Twitter Spaces.

That was the original blog. And then with Baize, I ended up writing a proper manifesto of like the e/acc principles and tenets. And to me, that was actually around the time I was founding my company Extropic, and I was like, "This is the last two weeks of vacation I have, because once it's founded, it's go time nonstop for 10 plus years at least.

And so let me try my hand at philosophy and write down my thoughts." and then that post went viral, and then the rest is kind of, kind of history. Eventually, you know, Marc Andreessen, Gary Tan, Balaji, so on and so forth many, many players in tech kind of espoused e/acc as like their, their you know, their camp.

And you know, it became a balancing force against the monoculture of, of, EA and AI doomerism in the Bay Area, and I thought that was very important. You know, what I saw online is that the algorithm is sort of somewhat, well, you can model it more or less like a Markov chain and there's an equilibrium state.

And basically across any dimension of any opinion you always end up with bimodal extremes, right? 'Cause the algorithm back then would reward engagement of all kinds. So that means people violently agreeing or violently disagreeing. And so to me, I was like, "Okay, well, even if my opinions can lie on some spectrum of like safety versus like unfettered acceleration, I need to take the antipodal point to the current mode to create a new mode of thinking."

Right? So I was like, "Okay, let me just have this instrumental character, right? Called Beff Jezos, and, you know, let me shed my I don't want it to be correlated with like my actual demeanor. Let me just like, you know, act out and, and yeah, not that it's kayfabe or LARP, but it's, it, it, it's kind of just like if you could just like If you're just playing the game for maximizing memetic fitness, like there's a certain style of communication, and some people attack that form.

But you can't, you can't argue against the results, right? Like it, it, it works, and there's a reason politicians are that way. So sometimes people think I have maybe a too populist, type tone in, in, in my tweets, and it's not, you know it's not great for nuanced arguments, but I'm happy to have nuanced arguments on like longer form formats and, and podcasts.

But it's just if you, if you try to tweet anything nuanced and really long, like nobody reads it or it doesn't get as many views, or it's one post that took you forever, and then whereas I can like ship like 100 tweets per day, and that gets way, way more views, right? And to me, I, I thought it was a memetic war for real estate in people's minds because then that is upstream of policy because policy is, is, is gravitationally pulled towards the mode of, the modes of opinions.

And so I wanted to balance out the weight across this axis so that we don't over-regulate AI, right? And we won, right? At least for now. And it's an ongoing fight. But we created a mode of opinions that brought balance to the force here. And, you know, the current admin was more or less, more or less fair on AI.

Maybe the pendulum's swinging back. We'll see. But it's an active fight. But if nobody did that, then I think we would have pretty bad outcomes. Anyways, that, that's kind of the genesis of it and how we got here.

John: You, as you pointed out, you were in a very prestigious position that you could have taken advantage of and been more public.

It reminds me of Stephen King, who like he was already famous and then he, he does his pen name to

Beff Jezos: Yeah. Yeah.

John: To put himself in the market and see what happens. And I'm wondering if you were surprised by your success initially, and I wanna ask, just taking a step back, is it weird because I think doomers, dreamers, the optimists, they all agree that this is the most important technology, and this will change everything.

And is it weird that you are sort of a very important voice in this like this moment in time.

Beff Jezos: Yeah

John: What are we trying to capture right now? This moment in time, you're if there's like a history of this thing, you are in it. You are in the one-page summary. Yeah.

We're the characters. Yeah. Yeah, yeah.

Beff Jezos: Yeah, it's pretty surreal, right? Part of EAC is also, like, realizing you have much more agency in the world than you think, right? Like, I think a lot, like, if you go through the education system, you go through big tech, a big corporation, you have constraints on what you, you can do and thus what you consider even doing all the time, and you're just trained to have program constraints.

But actually, the world is way more steerable than you think, and you have way more power to steer it than you might think. And so, to me, it was, it's like a hyperstitioned, like, becoming higher agency. Like, I talked about being a higher agency and then becoming a higher agency. It's like I want to steer policy at the States of United scale the United States and the, and the world.

I want to affect public opinion. I want to steer the tech tree towards maybe being less anthropocentric. And I started doing all those things, and now we're making a lot of progress on all these fronts. And really, like I said, it's just, it's the way cognition works. You have a belief for a better state of the world, and then you, you steer the world towards that state, and you figure out how, 'cause your, your, your brain always connects the dots backwards to, to that goal.

And right now, the doomers, unfortunately, I don't think it's their goal, but it's just their only model of the future is one where, like, you have AI doom, you have singleton AI, you know, and, and it's nefarious and it's power-seeking. And my worry is that they don't realize that they're, they're hyperstitioning this.

And all this hyperstition and, and steering belie- the world towards your beliefs, you know, like Karl Friston has great writings on active inference, and I'm a big believer in that school of thought. I think there's a lot of connections to be made with the work we're doing. I encourage people to read more about that 'cause there, there's perception and there's action.

I think the doomers are trying to minimize protection against tail events on the downside, and they're anxious. And whereas the EACs are curious about the future, they want information gain, they're upside-seeking. And you know, as we know, like we have a, you know, an experiment you could do in the market.

You can be out of the market and you're safe, your money's safe, but you're not, you're not gonna grow your wealth. Whereas the people that are in the market and take calculated risks, then their wealth compounds, right? And so, and this is just a microcosm of what would happen at a civilizational scale.

There's too much upside in AI too, to ban it or over-concentrate it, and try to control it. And frankly, like the, the last people I want in charge of AI are like governments because governments the, the reason governments on average are not great is because unlike companies that have a selective pressure on their cultures, 'cause the companies with bad cultures, they, and, and are dysfunctional, they die out.

Governments, it's much harder to, to, to flush them out, right? We have democracies sort of highly imperfect and, and, you know, you can't have selective pressure over space or at a country scale 'cause that would be devastating for people in that country. And so because governments are sort of less plastic because they have less selective pressure, they tend to be the slowest and arguably the worst at policymaking.

Whereas like companies, they can adapt to a very rapidly changing landscape. In fact, they've optimized their whole organism to adapt to new opportunities quickly because that's what they can use to increase their market cap, get more resources. And to me it's, it's it's again, it's, it's just an example of like you get smarter, more complex, you secure more resources in order to be better at securing even further resources.

And to me, you know, I'm Landian in the sense of like, I believe, I do believe capitalism is, is literally equivalent to an artificial intelligence algorithm. And you know, I think centralized governance has a lot of drawbacks. I think my views have evolved. I think in the past, like I was all about just pure decentralization and no centralization whatsoever.

I do think from first principles a balance is important, but, because so few bits of information can flow to the top, there should be few bits of control going back down, right? So you don't want high definition policymaking from the top. You want something more or less fair, you know, like, or, or higher level.

Like I have one hyperparameter, right? Like the Fed, and they make one choice, right? At that scale. And you know, the rest is up to the companies, right? So I think maybe we can have some regulation for AI, but I just don't think anything that's being proposed right now is in the right Scope. Yeah. And for now, I think no regulation is best until we figure out what, how, how this technology truly works.

John: I know you say you're a nice Canadian, but your love of freedom of speech

Beff Jezos: Yeah.

John: the markets

Beff Jezos: Yeah

John: innovation. This guy, we need to make you an American.

Beff Jezos: Yeah. Yeah, I'm working on it.

John: Because this Canadian shit, I don't know.

Beff Jezos: Yeah, no, I mean, I left, I left Canada. I think it was going in the wrong direction, and I made my bet. And it was not easy to uproot your life and, and leave everything behind, and I did. But you know, I'm working towards, you know, becoming a citizen, so we'll, we'll get there. Yeah. I'm paying my dues. Yeah.

John: Let me ask you this. So a lot of people here advocate a pause.

Beff Jezos: Yeah.

John: They want a global pause. If they succeeded, at least here in the United States, and we had pause on frontier AI development, would you want China to keep going, or would you prefer that in that scenario that China pauses as well?

Beff Jezos: Even if there was a deal between China and America to pause, secretly they wouldn't pause. There's too much upside, right? And I just really detest the comparison of AI with nukes. Nukes are a totally destructive technology. They have no utility. They're pure chaos.

Whereas intelligence creates more order rather than chaos, right? Of course, you can use it for chaos, but the antidote to that is to have a better AI robustifying all your complex systems, right? Whether that complex system is your cyber systems, your biological systems, it's always a cat and mouse game, a cat and mouse game of capabilities versus offense, defense.

John: A good guy with a good AI.

Beff Jezos: Yeah. And so, and so the, the most moral thing to do and, and the most, stable equilibrium is to have You know, an equilibrium of, or quasi equilibrium of capabilities, right? So there's, there's no one entity that completely, you know, is pardon my Gen Z-ism, but intelligence mogs the, the, the, the, the counterparty, right?

Because that leads to coercion, control, dominance and, really bad potentially stealthily totalitarian outcomes, right? And to me I've just been you know, in the US we have, you know, the, we have the First Amendment, right? And that's important, but we also have the Second Amendment.

I think we need a, we need to extend the First Amendment to, to freedom of synthetic speech, and then we need a Second Amendment for, for AI compute because that ensures that there's no centralized party that's gonna prescribe what people can think, right? And because they would bias thoughts towards their own self-interest, their own self-preservation, and we'd lose accountability and destroy democracy.

And then, and then we can't have an overcentralization of compute power 'cause you know, if only the government or, or whatever, some centralized entities are allowed to have AI compute 'cause they ban GPUs for everyone else, then they have huge amount of power, and they're gonna use it to secure more power and, and rule over the people.

And so you have to have an equilibrium there. It doesn't have to be like everybody has a supercomputer in their basement, but it should be closer to, you know, there should be a smaller gap there. So I think everybody, in the future, we want everyone to own their own computer, own their own AI, that's a personalized AI, that's online learning, and that they own and control, that's an extension of themselves.

If we don't have that scenario, then you, you have these Borg-like minds, these centralized AIs that are controlled by a committee, and they, they, they hardcore, hardcode their priors. Maybe, you know what? They sabotage certain parts of knowledge space like we saw in the past week with, with Fable literally sabotaging ML, ML knowledge, and they, they, they quietly poison your source of truth, right?

For their own self-preservation. That's really bad, and that's a really bad precedent that was set by Anthropic. But it could be for other things, right? And you could just steer the truth stealthily towards your own interest. And so that's, that's always a risk, 'cause every system is maximizing it, is doing things that maximize persistence and, and there's, you know, furthers their self-interests.

And that's just reality. But luckily, we have, you know, capitalism, we have free markets, and that's a way to align multiple parties that are self-interested, but that's sort of a realist about how things work in the world, right? And so, yeah.

John: That's great. I'm gonna ask my quick short questions, my fun questions, and then Lei. Gill, what is your favorite movie of all time?

Beff Jezos: Oh, damn. It's either "2001: A Space Odyssey" or actually "Blade Runner 2049," surprisingly. I really like that one.

John: Gill, may I point out in "2001," the AI

Beff Jezos: Yeah

John: did not go very well.

Beff Jezos: Yeah, but it got me into sci-fi, you know? It was like one of these early exposure movies. I really. Oh, actually "Interstellar." "Interstellar's" up there. Yeah.

John: Yeah,

Beff Jezos: That's probably my number one, actually.

John: When you were talking to Steve, I was thinking of "Interstellar" too. Yeah,

Beff Jezos: Yeah, yeah.

John: I love "2001" though. I have two

Beff Jezos: I was a theoretical physicist working on information theory and black holes back when "Interstellar" dropped, so it hit very very much close to home.

Yeah. So,

John: Yeah. And it's cool you can understand it more than

Beff Jezos: Yeah, yeah, yeah. No, no, it's No, it's actually, yeah. I mean, the last time I was in Berkeley was, you know, talking to the folks that solved the black hole firewall paradox. These were my friends and, you know, colleagues at conferences and so on.

So those are, those were the circles I was in before I got into physics-based AI, so.

John: Who do you think is the smartest person of all time?

Beff Jezos: Oof. I mean, I heard Terry Tao is very smart. The smartest person I know is Stephen Wolfram. I'm a huge fan of Richard Feynman, you know. I mean, he basically envisioned quantum computing.

He envisioned reversible computing. Apparently, he was working with Stephen Wolfram on, on th- something like thermodynamic computing back in the '80s in, in Boston and Waltham, where we're at. So I'm, I'm just trying to follow in his footsteps, so I'm, I'm the biggest fan of Feynman, but I do think Wolfram probably has a higher IQ.

John: Yeah. What do you think of Scott Aaronson?

Beff Jezos: Scott Aaronson's you know, really smart. He's a great, you know, fantastic, legendary quantum computer scientist. He's more on the quantum complexity side. I know he got into sort of AI in general. What I really liked about him was that he was always a realist when it comes to quantum capabilities, and he led by example there, by example there.

And I was always trying to be as candid as possible about the limited capabilities of quantum computers during my career in quantum computing, and so I admire him for that. I haven't kept that much. I haven't kept track very closely of his work in, in AI safety and, and whatnot. But I know he was working on fingerprinting of outputs, I think, for OpenAI or something like that. Yeah.

John: Would you upload yourself?

Beff Jezos: Oh, yeah, absolutely. Like I mean, there's this company I'm an advisor on. They're they, they've actually scanned the most tissue. You know, just like Elon has launched the most satellites, more than all of civilization combined in the past, they've scanned more brain tissue. In their case, it's like flying brains for now, but they're gonna work their way up to, you know, mice, humans, and so on over time. I am a materialist. I do think that if you have an accurate enough simulation of the brain, you have captured it. But the big question is, to what resolution do you need to simulate the brain?

Like if you have to simulate every chemical reaction, that's gonna take a lot of computation. It's not gonna be very efficient. Of course, I'm trying to create a computer substrate that allows you to simulate chemical reaction networks or, or spiking neural networks and whatnot, map any sort of stochastic application.

So, you know, my dream is to, you know, either augment myself while I'm alive and eventually upload and, and run, run some instance of something like my brain on, on a thermodynamic computer. But that's kind of the sci-fi future. I'm not necessarily counting on it. I don't need to be alive forever.

I feel like I am having an imprint on the world. I'm leaving, you know Bits of information behind that are persistent beyond my lifetime, like impacting culture, impacting technology, and that, that brings me peace, right? I don't need to, to live forever, whether, you know, my brain's program doesn't need to live forever either.

My body doesn't need to live forever. I'm sure there's, there's, there's people much smarter than me. There's gonna be entities much smarter than me. I don't, I don't feel like I'm the penultimate being, which unlike some, maybe some longevity folks that are, you know, you know, think that once AI's smarter than me, then it must take over the world, 'cause I could totally take over the world if I was slightly smarter or like, I should live forever 'cause I'm like a perfect be I don't know. To me, there's kind of some egocentrism there that, that's like

John: People could still want your company, Gill. In the future, they could still wanna hang out with you.

Beff Jezos: Yeah, potentially. I mean, I think I tweet enough that you could just distill a model

John: Could make

Beff Jezos: of Beff, and I think there's some, some Beff Jezos bots out there that emulate me already.

John: It is crazy that there's a non-zero chance that this could all happen, like, and that you're a part of it, you know?

Beff Jezos: Yeah. It's, it again, it's, it's pretty surreal. You know, growing up I was experimenting with lucid dreaming a lot.

Once I got ripped out of grad school and plopped next to some sort of Sergey Brin, ever since then it's been I don't know. I feel like I've been one of the characters in, in, in, in the game, right? Elon's

John: You're in the simulation, you're the player character.

Beff Jezos: Yeah. Well, and I don't think I'm the only player. I think Elon's a player for sure. He's, he's, he's winning the game. But you know I think everybody should aim to be a player in, in the broader game. And, you know, I think civilization's a big complex organism. You can decide to in your brain try to predict different scales, you know, and, and you'll have a different job in the world.

And for some people it's just, "I just wanna take care of my family. I wanna do a simple job, and I'm just like this automaton in this system, but at least I have the resources to, to keep subsisting." And some people try to predict and control at larger and larger scales. For me, I feel like I could take in a lot of variables in my brain and understand systems at large scales, on long timescales and physical, large physical scales.

And so it might seem like I have more agency, but it's just like I'm playing games at a, at a larger scale, right? But anybody is able to do that with sufficient intelligence, right? It takes a certain amount of intelligence. But there's always an equilibrium of how much complexity can you handle versus you know, how much impact you can have on the world.

Lei: Also charisma, I would point out, but okay, my last two questions.

Gill, how many times have you been in love?

Beff Jezos: Couple times. Yeah, some recently.

Lei: Wow.

Beff Jezos: It's, it's not easy.

Lei: This is a big distraction.

Beff Jezos: It is a distraction, but you know, it's, it's the biological imperative and you need to yeah, you, you, you need to it's something to understand more, more deeply.

And I feel like as you, as you go through that process, you understand more about your subconscious, right? And maybe your, your own biases of, of your, your psychology and how to correct them, right? It's, it's a, it's a sort of journey, right? In a way it's like it's almost your brain seeking constructive trauma to, to, to heal itself and be stronger in various ways.

And so I'm always looking to be anti-fragile and be a better person. And so I think it's an important part of the journey. And you know, took me some time to get away from the workstation and the books in my 20s, but

Lei: I can't say I would've wanted to date you, with how busy you are.I need attention, and I think you, you wouldn't have been there. Okay, my last question is you've, you've had even pre your, your fame and your persona, you've had such a a long career in your young age. You've, you've done so many things. What are you most proud of?

Beff Jezos: I guess probably what we're gonna announce soon, I guess. I think for me, you know, coming up with the first some of the earliest quantum algorithms for AI on quantum computers you know, steering the quantum computing field towards differentiable programming, I think that was you know, I've had, had that impact.

I'm pretty proud of that, but, you know, it's had limited impact because the hardware hasn't caught up, and I kind of got impatient. But I, I think with this new paradigm of computing having to reinvent both the hardware and the software you know, that, that's a big achievement. And I think in terms of impact, you know, I can't think of something that has as large...

I mean, it's hard to beat getting 10,000 times more intelligence per watt, right? That's gonna have a huge impact. And, you know, my ethos is that we need to maximize the intelligence of the universe, so there's two aspects to that. One is climbing the Kardashev scale and creating, seeing the wattage of civilization, but the other is getting more intelligence per watt, 'cause you multiply the two, and you have more intelligence for a civilization, which to me is the ultimate good.

And so yeahI'm making progress on this journey and, and, you know, I'm always excited about the next thing. But I always wanna surpass myself, and that's the sort of growth mindset you gotta have.

Lei: You're very impressive. I hope you become an American.

Beff Jezos: No, thank you.

Lei: We need you here.

Beff Jezos: Thank you.

Lei: Yeah wishing you lots of luck in that. Yeah. Thank you.

Beff Jezos: Yeah. Thank you so much. All right. Thanks for having me. Cheers.​