Jaan Talliinn: One kind of important disclaimer that I should say is that like, really, like, nobody knows how the, how the future is will be unfolding. There are some kind of, like, reasonable arg ents, and the AI kind of keeps improving to kind of make those arg ents actually more strong and salient.
the main argument basically is that we might be heading for a future where AI no longer needs humans. And by that, I mean we might get a self-sustaining AI, AI that can build its own infrastructure, can do its own improvements without any need from, from human, any help from humans.
And in fact, humans might be too slow for it. And obviously, from the human perspective, I think that's a very negative future, just like human s have been, like, very unfortunate for many other species that we have driven extinct.

Steve Hsu: Welcome to Manifold. My guest today is Jaan Tallinn. I believe he is the first three-time guest on my podcast. Jaan, welcome to the show. Thanks for having me. For the audience, Jaan is a founder of Skype. He was an early investor in DeepMind, and he was also an early investor in Anthropic. He has been very active in AI safety and to help humanity survive AI existential risk, and that is the topic that we will focus on today.

Jaan, we're here at LightHaven. For those of you watching the video, you can see the LightHaven sign. We're in a podcasting studio at LightHaven. LightHaven is a focal point in the East Bay and in the Bay Area for people who think about existential AI risk. Jaan, let me start by asking for you to maybe give a s mary for a layperson of the current situation.

Like, where are we in terms of the development of AI, heading toward what might be, in the future, real existential risk? Just maybe start by level setting there.

Jaan Talliinn: One kind of important disclaimer that I should say is that like, really, like, nobody knows how the, how the future is will be unfolding. There are some kind of, like, reasonable arg ents, and the AI kind of keeps improving, to kind of make those arg ents actually more strong and salient.
the main arg ent basically is that we might be heading for a future where AI no longer needs humans. And by that, I mean we might get a self-sustaining AI, AI that can build its own infrastructure, can do its own improvements without any need from, from human, any help from humans.

And in fact, humans might be too slow for it. And obviously, from the human perspective, I think that's a very negative future, just like humans have been, like, very unfortunate for many other species that we have driven extinct.

Steve Hsu: So you mentioned the possibility that AI could be self-sustaining. could you talk about a scenario perhaps where it escapes from a data center or escapes from one of the major labs and then- Manages to copy itself around the world. So maybe just lay out explicitly what that scenario would look like.

Jaan Talliinn: I think it's really hard to lay out kind of very explicit detailed scenarios because almost all... any detail you add would, would make things m- less unlik- less likely. But like we already see with existing, AIs their kind of ability to cunningly escape, various constraints, sandboxes.
I think there have been now multiple cases where that has happened with existing AIs. Soin some ways I think it's dangerous to be dismissive about AI. "It's all just a bunch of n bers in a computer and, and like there's nothing to worry about because if you think about it AI is currently really going for our cognitive abilities. and like AI is not going to get worse. The main ways how we are kind of keeping AI in, in check really is using the fact that it lacks certain cognitive abilities which means that like this strategy really cannot be it's not really sustainable long term. I think there is like like there's like one scenario that I do think is,probably not very likely because again, it, it includes like, too many steps, but it's like at least like fairly easy to understand, I think.

Is that you just have like a economy pivot towards serving AIs rather than humans, and then just like replacing humans as they become bigger and bigger bottlenecks. So instead of having humans build data centers, you will have robots to build data centers. And instead, like human armies and, and police enforcing the laws and regulations, you will have like AI-protected, robot-protectedAI infrastructure.
at which point human s might no longer be just able to resist, the, the proliferation of, data centers. And they might not even be like data centers in, in kind of human terms again, because AI has like very different constraints, very different preferences, so to speak. Like for example humanoid robots are probably too big for AI.

AI running much faster than humans at least in principle, it will probably need much, much smaller robots because they can be faster.

Steve Hsu: They probably have heard about huge data center build-outs. Mm-hmm. And of course, it's not easy to audit the compute that's running in those data centers. And so one could imagine if the whole planet gets covered by these data centers, and most of the activity there is AIs doing stuff for people, like helping you book an, a flight or answering some query in a chat. But some fraction of that compute could be AIs that actually escaped and are sort of just living on their own, and, and already we're kind of on the verge of something like that happening.

Yeah. I believe that currently something that the half of the compute on the planet is dedicated towards serving AI, mostly in terms of training next generation AIs. but probably increasingly so for also different logistical and whatnot needs.

I think in the last the last time you were on Manifold, you said something like $100 billion plus or hundreds of billions a year are being spent on improving the capabilities of AIs by these leading labs. Mm-hmm. And only a very, very small fraction of that is being spent on safety. What do you think the n bers are today in 2026?

Jaan Talliinn: Yeah. I think like the rough n ber is like $1 trillion per year now when it comes to, like, you know, building out AI infrastructure and including all the energy energy needs and then like, that kind of channels into training, et cetera.

And, yeah, AI safety efforts certainly have like increased. But I think Definitely not in, in multiples, or at le- at least not comparable to the actual AI capability.

Steve Hsu: Just to take a rough n ber, if, if it's a trillion that's going into AI infrastructure and training more capable models, could it be a billion that's spent at most on AI safety research?

Yeah, I think like billion at most. Yeah.

Jaan Talliinn: I think it's pretty certain less than $1 billion. So it could be

Steve Hsu: 1,000 to 1 expenditure.

Jaan Talliinn: Yeah, probably more.

Steve Hsu: The, the

Jaan Talliinn: ratio

Steve Hsu: is more than 1,000 to 1. Yeah. 1000x making the thing more powerful relative to making it safe.

Jaan Talliinn: Mm-hmm. It is, like, hard to count these things. I think depending where you draw the line Yeah ... and what is kind of what counts as AI safety. I think, like, $1 billion. It's unlikely to go over $1 billion.

Steve Hsu: Right. So we sort of walked through your first point, which is that AI might get to the point where it can exist independently of humans, so maybe hiding out in data centers or in robots.
Related to that, though, is its ability to improve itself autonomously. So without humans trying to help it, the AI itself may be thinking about, "How would I improve my internal architecture? How would I improve the algorithms I use? Maybe what data would I go and try to find to make myself better at X?"

Mm-hmm. talk a little bit about how that might come about and what the consequences would be if the-- if we get to that point.

Jaan Talliinn: So the unfortunate situation currently is that leading AI companies are in, in a cutthro- cutthroat race. There are some, like, mitigating factors just like as of, as of recently that have like are making situation more complex like the war in, in Iran that actually I think is, is having like a nontrivial effect on, on AI buildout. as you saw with Mythos, like, there is like some restraintnow being exercised when it comes to, like, publishing the models. But, like up until this point, it really was that as long as you have some- have something capable to show, you sh- need to publish it as quickly as possible in order to have basically in order to as- build up your revenue and investments as quickly as possible.

Steve Hsu: Since we're in a competitive cutthroat race- Mm-hmm ... companies are racing to try to get ahead as much as possible, and maybe they will be very happy to allow the models to modify themselves. Exactly. Yeah. Yeah, you were

Jaan Talliinn: talking about self-modification. So, like, in this kind of cutthroat competition, there is actually kind of like explicit demand within the companies to kind of use these AI capabilities or kind of like feed, feed those AI cap- capabilities back as, as much as possible to kind of accelerating the race. So, like, there's like a very natural economic pressure to head towards self-sustaining AI.
Steve Hsu: So one of the things we're gonna do in our doc entary is make an animation that tries to convey to the average person. What is meant by recursive self-improvement of a model. Mm-hmm. And if you could maybe explain the way you would explain to a smart but a non-expert person- Mm what is recursive self-improvement, and why is it dangerous? So one very

Jaan Talliinn: sort of simple but not fully correct, way of explaining this is even now you might have heard about AI coding agents, and like the AI development even though the central thing isn't about coding, but it's still the engineering involves quite a lot of coding.

So as the AI gets more capable the ability to code you know, infrastructure that is necessary for further AI training becomes easier because you're using the, the AI from the last generation to, to build the scaffolding and infrastructure for next generation. That's an example where you have, like, AI capabilities feeding back.

I've also heard that, like, chip design is now kind of using more and more AI in order to come up with next generation of chips. That's another example where AI, AI being powered by chips actually feeds back into, into next next generation of chips. And in general, like, whenever you have, you know, a process as a physicist you should know a process kind of feeding back to its own speed, you will have basically exponential- Yes exponential situation at your hands. Just like with viruses, like the more viruses there are, the more infections there are, hence, hence you will get, like, exponential speed up. And, that's just like a sort of mundane way how AI can feed back to itself. Ultimately, you might have some kind ofvery tight loops where you have something on a data center suddenly start starting to just improve itself without actually going through the real world loop of improving chips and manufacturing chips, et cetera.

Steve Hsu: So imagine you're talking to a political leader or maybe even another billionaire who but who is coming from some old economy industry and doesn't understand AI or software very well, and that person just says to you, "Jaan, I don't see why you're concerned about this. These are just things that humans make, and we can just shut them off. I don't understand what you mean when you say recursive self-improvement. How could these things get out of control? How could they improve themselves beyond our control?"

Jaan Talliinn: Yeah. So this is like, there are so many kind of different questions there. One is that like when people say that like, "Oh, we can always pull the, pull the plug." Like, that's where, where I want to kind of stress like the self-sustaining bit. We kind of pulled the plug on COVID but like not really. It's still going. Right? So once you have something that is like self-sustaining, o- once it no longer needs humans to continue existing it's much, much harder to pull the plug so to speak.

The reason why we currently still have the ability to pull the plug is that the AI just doesn't have enough capabilities to develop and maintain its own infrastructure. But again, economic pressures are towards ensuring that human s will be cut out from this loop unless we actually counteract that pressure.

Steve Hsu: The way I phrased the question, I brought in both self-sustainability, but now I wanna switch to self-improvement. Mm-hmm. So this, this person is asking you, a very sincere person just trying to understand- Mm-hmm What is going on with this. Why do you think these things will eventually self-improve themselves and actually maybe race beyond our control and become much, much more capable on their own? Like, why, why do you think that's possible?

Jaan Talliinn: It's kind of hard to imagine an AI that is self-im- self-improving I, I suggest you imagine an AI company that is like self-improving and increasing automating, increasing or removing humans, firing humans and like ultimately replacing the CEO with, with AI and all the processes that involve humans.

For a sudden, once you're at one point you're down to like zero human s, and you have this company whose, whose like development just continues. It just continues to develop AI as it was up until this point. So I, I don't think it's like super hard to, to imagine a self-improving AI company at least.
And there's only In some ways it's like that's, that's all you need. Like, I, I can make a further argument that the AI might not even need a company because it might just like we don't Like, just like grass doesn't lead eco- need economy to grow. Once you have something that is self-sustaining, it doesn't really need human economy.

That's already kind of like argument that's could be like unnecessary. We can just talk about AI self-improving AI companies.

Steve Hsu: Is a scenario where the AI is controlling resources or getting access to resources through hacking, trading in financial markets, or mining Bitcoin- Mm ... are those all plausible ways that you think this might happen?

Jaan Talliinn: Yeah.

Steve Hsu: I think the

Jaan Talliinn: It, it really is a question about how capable AI we are talking about. So if, if an AI is not sufficiently capable, it will need some support from the economy. And like, one way to get that support is, is to make money and then, like, just order the things that it, that it needs.
But it's, like, also, like, really important to say that, like, human like, one reason why humans kind of need the economy is that we are not self-improving. We cannot kind of go beyond beyond the capabilities that we were, you know born with and what we, what we learned. But also, like, we don't really trade with ants or rabbits because, like, we don't need their economy or their ecology to do our thing. So like once you are sufficiently capable you might not just need a human economy anymore to to get everything you need for sustain, sustain yourself.

Steve Hsu: I believe your organization advocated for pausing AI research or frontier improvement of frontier models already a year or two ago. Is that- Yeah.

Jaan Talliinn: Future of Life Institute we have done you know, two We have participated in several sort of open letters but two of them we have initiated. In twenty twenty-three, we called for this six months pause to, in kind of like frontier development of AIs, just in order to kind of Because it was very clear back then that like GPT four ChatGPT basically, was a big big deal and we should kind of like

give society some time to react to this new situation that, that, that they're in. Well, that didn't happen. Part of the reason why we included. There was like internal debate, and part of the reason why we decided to, like, put like a deadline like six months was just to make it very legible that like no pause is kind of feasible without government intervention.

And the second letter was this superintelligence statement released last year which is just explicitly saying thatthere should be like wide buy-in like from society and scientists before we kind of rush to develop superintelligence or potential superintelligent AI that we likely would lose control of.
Steve Hsu: So those pauses didn't happen, and how would you describe the current situation in twenty twenty-six?

Jaan Talliinn: I mean, the situation keeps changing by day, right? So it's, it'sAnd the year is still young, so, so who knows what what the situation will be in a few weeks or a few months.
There are some like some positive signs that again, like this, you know, self-restraint from Anthropic in releasing the, the top model I think is an example of a company kind of like going against the incentive of releasing everything as quickly as possible. And like once one company starts doing that, it might actually kind of relieve the pressure, raise pressure for, for everyone else.

So that I hope that will kind of lead towards more kind of slow and deliberate AI progress. But up until this point, yeah, it was like, I think, pretty bleak situation because like there was. The way I kind of describe the game board so to speak, on a, on a global level, I think there are like four players.
If I were, were doing like a, some kind of tabletop exercise, I would have like four, four players unlike the one that, that we participated in London two years ago. So there would be basically the AI companies is like one player. Then there's US government second player. Chinese government is, is third player and rest of the world is, is the fourth player. And they have they each have kind of their own constraints. Like the, up until this point, really, the AI companies don't really have much free will. They need to like everything, then they do need to be in service of putting out the best model as quickly as possible in order to kind of sustain the moment .

US government is, at this point, the most powerful of the, of the four players, but they are deeply conflicted and like almost all their kind of economic and competitive advantage comes from the first player. So there's like a interesting dynamic there and yeah, unfortunate unwillingness to do, to kind of steer the situation.

Chinese government, I don't know much about. They, they say very reasonable and adult, adult things, I would say, but it's very hard for me to know like what's actually behind those reasonable sounding words. They're a bit of a joker in a, in a game.

Steve Hsu: Yeah.

Jaan Talliinn: and rest of the world is, yeah, asleep at, at, is asleep at the wheel. Like they just don't understand that their immediate future is going to be decided, by the three other players now or like the at least two of them. And if they were More awake or understanding, they could be like a formidable f-four player, but currently they're not.

Steve Hsu: So you mentioned Anthropic doing the right thing in delaying the release of their super capable Mythos- Mm-hmm model. In anticipation of our interview, I asked an AI what your stake in Anthropic was, and it estimated it, this could be totally wrong, it estimated at about twen- $10 billion stake in Anthropic. Mm-hmm. Mm-hmm. And I think a permanent observer seat on the board. And so did you play a role in getting them to act responsibly and delay Mythos? What's the status of your communication with the top leadership there?

Jaan Talliinn: Yeah, not directly. Like one thing that has happened is that Anthropic as it has scaled the board has, has increased to the point where there is like a, every kind of individual stakeholder, so to speak has like less and less voice.

And also like the processes has- have sped up so much that unless you're in town, which I'm not it's actually hard to, hard to follow what is happening. So yeah, I didn't have like much.
Steve Hsu: I mean, you named four players. Mm-hmm. And of course, we know that, you know, the President of the United States is the single most powerful political figure in the US.

Xi Jinping is the most powerful- Mm-hmm .player in China. Among the labs, do you feel there's incredible concentration of power in just a few people for one of those four most im- three most important players, I think, that you mentioned? Is there, is there too much concentration of power?

Jaan Talliinn: Yeah, my, my friend Sean O'Hegarty has this saying that like the future of the world will be decided by a, a group of people who fit in an elevator.

Steve Hsu: Yes.

Jaan Talliinn: So, so yes, there is a lot of concentration of power. But within the labs, I think it's, I think I believe it's rather different, because of like the culture is different and the structure is different. For example, like DeepMind, I have like a lot of respect for Demis Hassabis. I've known him since like 2011, I believe.

But I- I'm pretty sure he's kind of, a lot of his effort goes into navigating this entire huge Google Byzantine infrastructure or infra- politics and whatnot. I think that the OpenAI is probably like mostly a Sam show at this point. but I don't have a lot of visibility there.

I mean, for example, like the, the seven co-founders, they have been like fairly equal, in, in, in, in the company, from the start. Obviously, there are kind of like role differences and whatnot. So, so I think Anthropic is probably Kind of more consensus-based than than the others

Steve Hsu: Do you think it's something that average people should protest over in the streets about this concentration of power that's growing? Should we, should we trying to control and regulate these companies?

Jaan Talliinn: Yeah, up until this point where it was pretty clear that companies will not self-regulate again, with this, like, MITOS non... like, delay of Mi- MITOS release is, like, a first, like, serious counterexample to the... to what I'm about to say. I think it's, like, very important, to indicate that like, the current direction is not okay, and, and thereby kind of, like, build up, political demand for actually putting some reasonable regulations and constraints in place because the companies are kind of, yeah, in a tough, tough position when it comes to self re- self-regulation.

In fact, like, one, one of the encouraging things that I keep repeating is that both Dario and Demis in Davos in January kind of called for some kind of slowdown. I think we really should, like, hear, hear what when they're, when they're asking for that.

Steve Hsu: in a positive scenario where we do get to pause before it's too late-
Mm-hmm How much of that is elites becoming convinced that this is the right thing versus the general population? Do you need both components, or will one of those components be enough to get us there?

Jaan Talliinn: It's very hard for me to say. I suspect, like, if, if, like, su- if there's, like, sufficiently intense conviction on either of them, it should be probably sufficient. And especially if there's a combination it, it should be sufficient. But again, like, before, like, anybody in their mind goes, like, "But China" Yes, we need, we need kind of international coordination on that. And, and again, Chinese, in their words at least, have been saying that they're open to that. So, so, like, we shouldn't kind of dismiss them that, that we definitely cannot coordinate with China.

Steve Hsu: Yeah, you know, I was just in Beijing and Shenzhen with some of the top AI researchers. I was based at Tsinghua University when I was visiting, and they're just less AGI-pilled than we are.
So I think they're... in a way, they're not as far along in the thinking. But I think that, you know, they're very used to the idea that the government can and should regulate the companies. Mm-hmm. And so I think if they get to the point where they're convinced that, yeah, this could be dangerous, I think it'll be a, actually in a way, more natural, easy move- Mm-hmm for them to just restrain the companies but you just have to get their political leadership to the point where they really feel deep down that there is existential- Exactly. Yeah. Exactly.

Jaan Talliinn: That- that's why I said there are four players. Like, yeah, the Chinese AI companies are not a player in this. Yeah in this global Exactly ... global situation. There is, like, one I don't have, like, a lot of visibility into, into what's happening in China, but yeah, I do hear that, that they are just, like, less, like, behind in their, in thinking in general. And there is, like, one cultural thing that I, that I've kind of noticed over the years, is that in Asia in general but also in China, there is...

People tend to confuse consciousness and competence. so they think that, like, the natural trajectory of AI development is that they get, like, more and more competent up to the point where they become conscious, and then they are just like us. And then we, then we need to, like, think about how can we kind of, like, welcome our new, new, new members of society, which is just deluded thinking. I think-

Steve Hsu: Let's put

Jaan Talliinn: it that

Steve Hsu: way ... I think we might have touched on this in the last, the last time you were on the show, and- Mm-hmm ... I s- might have said something like, you know, Shinto, the, the sort of sort of folk religion in Japan, has a very animistic- Yes .Sensibility. And even the, the China, China's not a very religious culture, so they don't have this monotheistic mass religion the way that some of the other countries do.

But still, people, when they're worried about something, might go to the local temple and pray to some, like, kind of local deity- Mm-hmm for good luck or something. And that is also a kind of animism. Like, this little thing, this little statue has some consciousness and-
Yeah ...and volition. So I understand what you're saying.

I think there's a sense that, like, oh, these... If these things get smart, they'll just be like, the character Astro Boy- Yeah. Yeah, yeah, exactly ... in anime, right? Yeah. So yeah. Which I, I don't think is kind of

Jaan Talliinn: working in their favor.

Steve Hsu: I don't think it's gonna play out that way. No. No. Yeah. It's a... I think it's... well, comment on this, please. I mean, I think it is very likely to be a very alien intelligence. When these things reach this sort of full superhuman - Mm-hmm ... intelligence capability, it'll be a kind of alien intelligence that we won't really understand. It won't be like a cute monkey that talks to us or something.

Jaan Talliinn: Exactly. Like the, again, the really important bit about current AI is that it's not built, it's being grown. All these, like, evaluations and capability, like cards, sheets, they are basically done using a process that just evaluates this new alien that has stepped out of the, out of the machine. So it's not... And, and already we can see from these evaluations that it's, it's, it's a weird, it's a weird

Steve Hsu: intelligence. Yes. So we're growing these increasingly powerful, weird alien minds- in our labs. We're spending a trillion dollars a year doing it, and we're giving them access increasingly to everyday decisions that we make, like Mm-hmm trading stocks or buying an airplane ticket for me, choosing what restaurant I go to, giving me advice about my love life. That's what we're doing. you're actively working to slow this down. So you're deploying your own financial resources and your own life energy to slow it down. what's your strategy?

So, so how much of it is aimed at changing what the public thinks? How much is aimed at informing the elites about what's going on? Or maybe there's a third category that I should be thinking of.

Jaan Talliinn: Yeah. I, I think my, my strategy kind of has evolved over time. Yeah, when I started like seventeen years ago, it was like almost entirely about like supporting just like a few niche groups that existed back then, and kind of trying to create like a more, mainstream awareness and especially in academia.

So I started this Center for the Study of Existential Risk in Cambridge, UK, and, and then Future of Life a couple of years later. So, and, now really, yeah, I kind of keep repeating this, like no self-sustaining AI, please. And the for a while, yeah, the strategy was to just like talk to the legislators, either kind of directly or via, yeah, via the work at Future of Life Institute, and then try to kind of shape the regulation in a way that would actually lay down some kind of rules just like we have for other industries to ensure that they will not, not be exposing the society to like massive neg-negative externalities.

But that effort really hasn't gone far with this US administration. So yeah, like the current strategy is pivoting towards more just like creating kind of, more popular demand for some kind of rule-making, and then perhaps kind of creating, Yeah, basically like once you have political demand for that, it's kind of easier for politicians to, to actually do something about it.

The one big problem has been just the low saliency of AI up until this point. It is now changing. but it's like the way I put it is that when you ask, ask AI-- ask people in the West, like what do they think about AI, the median answer you get back is, "Oh, it's bad." But when you ask them what's bad, they're not gonna say AI, right?

So, so it's it's has been so far, like, people are concerned, but not in a way that like they think about it all the time.

Steve Hsu: I, I think though we are starting to see people very concerned about the economic impact of AI. Mm-hmm. So loss of jobs. But still I think for the average person j ping toward X or existential risk is, is still a big leap, and most people haven't- Mm-hmm made that leap. But does the pushback that we're starting to see about the economic consequences, does that make you more hopeful that we can make that leap and then pause things before we have existential risk?

Jaan Talliinn: I mean, on the margin, yes, a little bit. but it's I think it's also, like, important to like, there has been, like, this debate in AI safety, circles for a while that, like, look, that if there's, like, something that is kind of adjacent to the X-risk concerns, like should we like... But it's also, like, more legible, more understandable to public and whatnot. Should we just, like, pivot to supporting that in the hopes that, like, there's, like, a positive side effect?

But I, I belong in the camp that, like, no, you should just say what, what you're actually worried about. And, and, and while admitting that the other, other problems are also problems that we need to address. Butyeah, I'm fairly convinced that we need, need to kind of while addressing the other problems and not dismissing them not rest assured that if you're gonna, like, a-address the adjacent problems, that we're actually going to, be saved.

Steve Hsu: Well, it our documentary is gonna be almost 100% focused on S- X risk. We're barely gonna talk about the economic aspects of it. So it's, it's mainly trying to figure out a way to make the average person understand why these leaders, whether they're accelerationists, people at the labs, or they're, quote, "doomers" who are worried about existential risk.

Both sides have a non-zero probability P doom in their heads- Mm-hmm. Mm-hmm of something going very wrong. I think even Sam will publicly say there's some chance things are gonna go terribly wrong. Mm-hmm. So that, that's our focus. But By the way, the- Yeah ...

Jaan Talliinn: the doomer moniker is, is horrible. Okay.

Steve Hsu: You're... You don't like the doomer moniker. Oh, I

Jaan Talliinn: don't. Of course not. It's like Yeah, I think Eliezer Yudkowsky, he says that, "Well, if, if, if we are doomers, like, the, the opposites, opposing side is AI death cult." Because, like, they really don't care about Right humans going extinct Right which makes them a death cult.
But like, yeah, I think anti-extinctionists

Steve Hsu: perhaps. Yes. So you mentioned the death cult. So recently I've been hearing r ors that at the core of some of these big labs, there really is a successionist death cult, and let me explain what that means. These people don't like humans so much. They are really looking forward to a s-a worthy successor to humanity, and some of them, I think, believe that they will upload themselves to become part of this machine god, and thereby achieve immortality and, and you know, great power.
Does that sound far-fetched to you, or do you think the personalities you know all these people. Do you think there are actually people who think that way at the core of some of these big labs?

Jaan Talliinn: yes

Steve Hsu: there are

Jaan Talliinn: So there is a kind of selection effect that, attracts people like that. I mean, I don't think they're like in a majority or anything.

Yeah, reasonable to assume that like 10%, 15% are like that. I mean, I have like one illustrative sample is like someone like in AI, well, very fairly known, fair fairly known AI researcher like said to me, like, "Jaan, don't worry about this. humans are disposable species."

Steve Hsu: Now, when he said that, was he imagining that he personally was also disposable, or did he have a, a plan for escaping the disposal? I mean, I didn't, we didn't, I

Jaan Talliinn: didn't like, we didn't kind of ex- I didn't kind of elaborate this. but yeah, I, I ... From what I understand, like he's totally fine of going extinct. He also has children, so like that's like one interesting usually like the extinctionists or successionists, they don't have children.
but yeah, I think he's like fine of, of being exterminated along with his children.

Steve Hsu: So he's, he's fine being exterminated along with his children? Yeah.

Yeah.

Wow. I mean, they have like mental issues, let's put it that way. Yeah. Okay. So turning to mental issues, so you have children. You have a lovely family. Do you worry about the future of your children? Is, is that foremost in your mind?

Jaan Talliinn: Yeah, I do. It's, it's, it's not yeah. It's not, it's not a great time to be a child in terms of the world situation, where we're currently heading. I, I don't want to be like a fanatic and, and and kind of literal doomer in the sense that like nothing can be done.

Don't want to create like a self-fulfilling prophecies. I definitely do think that there is hope and, and things, can change like one important realization is that and we might talk more about this, is that before we get this like super intelligent system that we can't do anything about it, we might get something that is like just barely able to kind of self-sustain or, or do like massive damage- but it's kind of tempted to do it.

Mm-hmm. which means that the actual dynamics might be very different from, from what's predicted from this, like, asymptotic considerations. But yeah. And also, I have to admit that my kind of system one level, intuitive level doesn't really, like, resonate, with what my system two conscious part kind of thinks it's a horrible predicament that we're in. So on this, like, day-to-day level, I don't stress too much.

Steve Hsu: Yeah. So let me break that down for the audience a little more. So at the intuitive level, on a day-to-day level, you can be quite happy and enjoying what's around you. Mm-hmm. But when you think too much about it or you- you're concentrating on it, you realize humanity is facing a very, very difficult challenge. Yeah, the

Jaan Talliinn: most

Steve Hsu: the biggest challenge that we've ever faced. But, s- does the s- system one or system two influence, how you feel late at night when you're trying to sleep? Do you, do you lose sleep over- Yeah, I don't. You don't.

Jaan Talliinn: So

Steve Hsu: that

Jaan Talliinn: that's, that's kind of the I guess, like, that's why I'm lucky with that my system one.
although some of my friends are trying to argue- argue me out of it that, like, I should kind of train my system one to be more alarmed, because that should make me more effective. which is probably true, but on some kind of, like, selfish grounds I think I'm kind of happy with my system one not being super stressed. Yeah. I,

Steve Hsu: I would like to enjoy what finite time I have on this planet. Exactly. Yeah. Yeah. Yeah. speaking of enjoyment, I know you're a, a big hip-hop ... What, is hip-hop the right way to say it? You're, you're ... You like to dance. I mean, like, street

Jaan Talliinn: freestyle. Freestyle street dance that does involve, like, yeah, hip-hop and, and popping and

Steve Hsu: And is the, is the, the moment when your happiness and able to really get outside of everything, when you're on a dance floor, maybe in Tokyo- Mm-hmm doing your freestyle? My favorite place. Yeah. Yeah. That's your favorite place? Yeah, Tokyo is my favorite place, yeah, in general. And you're headed there after, after, this meeting?

Jaan Talliinn: Yeah, I mean, a couple of weeks, actually. Yeah go back to Estonia for, for a couple of weeks and then Tokyo. Have

Steve Hsu: You ever been videotaped dancing?

Jaan Talliinn: I think there are thousands of videos of me video. Oh, okay. Excellent. But, like, luckily they're, they're kind of like scattered on, on people's phones and Okay.
Not

Steve Hsu: We can't get it on YouTube. Y- yeah, probably not. If, if we play, if, if we play a little music, can we get you to freestyle a little bit? I mean, like, a, a little bit perhaps. Okay. Awesome. At the end, we'll let's try to do that. Okay.

Jaan Talliinn: Awesome. I'm just, I'm just

Steve Hsu: wanting to see it. We wanna see, we wanna see happy Jaan. Yeah.

Jaan Talliinn: Yeah. I was, was actually just, couple of days ago dancing here in SF in, on Sunday at some like, outdoor thing. And, and, yeah I touched, somebody. Yeah, I got, got really nice compliment even It's like, how can you be so good?

Steve Hsu: That's awesome.

Jaan Talliinn: Yeah. I'm, I, yeah, I live for the applause. Yeah. Oh, that's great. On the dance floor.

Steve Hsu: But now when did you start dancing? Where, like, when you were in college, were you already doing it, or it was later in life? Oh, later. Yeah

Jaan Talliinn: Yeah. I think it was more all in my 30s, yeah, like about 10 years ago.

Steve Hsu: And what, how did the thought process go? So in college, you would go to a club with your friends sometimes, but you weren't systematically trying to be a good dancer.

Jaan Talliinn: Yeah, I think I ended up, like, going to nightclubs with my Skype colleagues. I mean, I was, yeah, in, like, my mid-30s, and, like, rest of the crowd was, like, in mid-20s. So there was, like, a lot of, like, nightclub activity. So at one point I just, like, noticed that I actually like dancing, and, then I started getting feedback.

So, like, I only half jokingly say that I learned to dance using RLHF. Just, like, people giving me more and more feedback, and, like, yeah, I get. People are very nice when I, when I dance. That's awesome.

Steve Hsu: Do you, and am I remembering right? You actually practice, right? You- I do. Yeah, yeah

Jaan Talliinn: Yeah. Yeah. And, I have a dance instructor.

Steve Hsu: Yeah

Jaan Talliinn: Yeah. So he he's also a good friend. Sometimes we go, go out and kind of ... I joke that, that he does all the hard and impressive stuff, and I do the easy and impressive stuff.

Steve Hsu: And is this mostly in Tallinn that you guys are out, or all over the world?

Jaan Talliinn: All over the world pretty much, yeah. He's, he's based

Steve Hsu: in London. Okay. Yeah. Well, that's great. Let me come back to the dark stuff a little bit. one of the people that is very popular among the super accelerationists here in Silicon Valley, and maybe the people that believe in a worthy successor or some kind of successor, is a concept sorry, this person who's popular in that way is, is, is Nick Land, the philosopher.
Mm. I, I don't know if you're familiar with him.

Jaan Talliinn: I've heard the name, and, like, I, yeah, I like one quote that he, that he has.

Steve Hsu: Oh, what is the quote that you like?

Jaan Talliinn: I don't remember the exact quote, but it was something that seemed true basically, that what we're witnessing is, like, the future. How did, did he say?

Like, future AI civilization or something like that, kind of like reaching back through- Yes techno-capitalist means and kind of to bring itself into existence. Yes. Which, like, yep, that's how, that's, that's, that's what we are witnessing right now. Yes. That's what I wanted to ask you

Steve Hsu: about. Yeah. So because Land had these thoughts in the 1990s. Mm. Now, I don't know what you were doing in the 1990s, but

Jaan Talliinn: I ran a games, games company.

Steve Hsu: Yeah. Yeah. But, but at that time, like, that was pretty deep in the AI winter. Mm-hmm. Very few people were seriously looking at neural nets. We didn't have enough compute to really do much with neural nets. And so I think Land is quite unique in having that vision, maybe drug fueled, but in any case- Mm-hmm it turned out to be correct all the way back in the 1990s. And so I want to just talk about this idea of techno of techno-capital. So the idea that. The markets have their own dynamics and logic, which is independent of any one person. Yeah. So it's independent of Sam Altman- Mm-hmm it's independent of Elon.

The markets and society have their own dynamics and logic, and capital makes new technology.

Steve Hsu: Technology allows aggregation of even more capital, and you have this positive feedback loop that Land thinks would, all the way back in the '90s, thought would lead to the emergence of a super intelligence. And it seems to be what's happening.

So maybe you could just comment on that.

Jaan Talliinn: Yeah. So like that's like g when you earlier asked me about like self-sustaining AI I basically used that framework, that you have like this company that kind of works within the economy but then kind of gets like increasingly automated and just economic pressures will push out human s, and then like the entire economy pivots more towards, even more towards serving AI.
So eventually you basically have a situation where there will be competition over land, between, you know, like data center build-outs and agriculture, and agriculture will lose- Mm-hmm because it's no longer aI doesn't care about agriculture, it doesn't care about healthcare. So yeah, will perhaps just starve because, because the, the, the economy will be less and less interested in them.

And in fact, there's like a potential flywheel effect. The more are out of jobs, the less economy cares about them, and the, the more there is kind of value in, in re-replacing.

Steve Hsu: Yeah. So in a way, it's a very, transhumanist and sort of, you know, soulless, machinic kind of logic that's gonna lead to all this stuff regardless of what

Jaan Talliinn: Unless, unless there are shortcuts, right? Yeah. So like this is like, again, I'm not ruling out that you might have some, you know, extra economic event or, you know, external to economy that actually makes AI self-sustainable in a way that COVID was not really part of economy.

Steve Hsu: Yeah. I, I mean, it sounds like you're counting on to wake up, to develop some kind of awareness that we, we wanna, we wanna short-circuit this techno capital loop before it's too late.

Jaan Talliinn: Yeah, yeah. And the, and the good news is that we kind of have done it before in, in some cases. Like the, like in some ways, like easier cases the, like human cloning Mm-hmm is an example that like you can certainly imagine like profitable companies doing human cloning but we just didn't do it. at least I know, I mean, there have been like some examples.

The entire nuclear like in Like 90 years ago or so, people were pretty pessimistic. as Leo Szilard said that the world is headed for grief. But we kind of muddled through because we well, and that's not just because we kind of automatically muddled through, but there was like a, like significant amount of effort put into, into making sure that nuclear doesn't proliferate and whatnot.

Steve Hsu: So my impression, you know, because it's, it... Well, I don't know how sunny it is. It was sunny for a while today in Berkeley. It's usually sunny here. You seem like you're in a good mood. So I, I d- I don't think I could apply the word doomer to you even for your- ... your external, demeanor. You seem very, pretty happy today yeah.

Jaan Talliinn: Yeah.

Steve Hsu: I have, I have a good life. Great. All right. Well, thanks a lot. It's been great having you on the show again.

Jaan Talliinn: Thank you.

Steve Hsu: How did you get into existential risk and AI and all of this?

Jaan Talliinn: Yeah. I think in 2008, or was it even 2007 when I was, was still at Skype, but kind of starting to look around, like, what, what is the next thing for me to do and be interested in, I stumbled upon the writings of Eliezer Yudkowsky. You know, he has written, like, 1,000 essays.

So at one point I kind of developed my own software to make it easier to read his essays and kept reading, and at one point thought, "Okay, I... This man can write faster than I can read," so I, - Just shot him a cold email and we met up in, in 2009. And after that basically I was sold that, yep, I, I need to do something about this. This is going to be the most important topic of the

John: world. Eliezer's often the gateway drug for existential risk. Many people,

Jaan Talliinn: Yeah.

John: Could you just say about what, what did your software do to help to assess this?

Jaan Talliinn: Yeah. So Eliezer's writings were, kind of interleaved in this co-authored blog with Robin Hanson on Overcoming Bias.

And they were sort of like in random order, and there were like lots of comments. So basically what I did, I, I kind of scraped all the all his writings and kind of ordered them chronolo- chronologically, so it actually would make sense. And then like annotated all his links. He really likes to put kind of like cryptic links in his writing.

So like the links all had a- annotations whether I should click on them or not. And then I also like formatted the comments cutting them off at the point where Eliezer stops responding to their comments. So I knew like yeah, basically like make it made it more usable. Wow, yeah.
John: So we're doing this interview in Berkeley at LightHaven. Mm-hmm. Could you, talk about for people who don't know sort of the outsized influence this has on AI safety, like what LightHaven is, what your thoughts are on it and your role?

Jaan Talliinn: Yeah. So I'm probably not the best person to talk about like what LightHaven is exactly. But yeah, I've been a supporter of the group in general because the, the group behind it is you know, with like Oliver Habryka and his, his team who were kind of behind reviving Lesswrong. lesswrong.com it .com, I think it's .com was basically the blog that Eliezer started writing on after, you know, s- splitting off from Overcoming Bias that I just mentioned.
And then it kind of at one point it sort of like decayed. but Oliver took it and and re- made a very successful restart. And I think shortly before COVID he started this, a co-working space in Berkeley. what was it called? I forget. And like basically I saw that this is, is a pretty amazing, amazing place.

So at one point where he, you know mentioned that he would actually like to create like a more permanent and more like weighty physical manifestation of the, community that has been interested in kind of AI safety and, like, all adjacent topics to me sounded like a, like a great idea.

John: Yeah. This is sort of the center, I think, of everything. There's so many meetings here and, and people doing things. Could you so we're doing this interview towards the end of April 2026. Could you talk about what your timelines are?

Jaan Talliinn: I mean, they're highly uncertain. Usually it's the weakest thing I can say is that, like, they're probably years, not decades.

So but, yeah, I have like double-digit uncertainty. As in, like, I, I'm not, like, 90% sure in kind of any of the, claims. But I did support the AI 2027 report, which I think still is kind of the most, kind of thorough, prediction about timelines. so I'm mostly kind of deferring to that or, yeah, other kind of people in the community who are thinking about timelines.

John: It's been pretty accurate. Yeah. Yeah. which doesn't make me feel good, but,

Jaan Talliinn: Same here

John: Yeah. But I think it's bought a lot of credibility in how accurate it is. Mm-hmm and it's been one of the most influential, I think, things recently that I've heard people talk about.

Jaan Talliinn: Yep.

John: Now this question, you may have deeper thoughts on it, but it's one of the most common questions here.

What is your P doom?

Jaan Talliinn: So I try not to be very explicit about my P doom because, because, yeah, I, I don't want self-fulfilling prophecies. and I... If there's like something that I can still do to lower it, I don't want to sabotage my efforts by actually nailing the number to something. but, like, one thing I could say is, is that in this hypothetical world where I kind of stop dealing with with AI risk, like, completely and just, you know, spend my time dancing or something like that, yeah, it doesn't look good.
We are probably like 80, 80% doomed or something.

John: Yeah. Because like Steve said, you're devoting a significant amount of your time and resources into Mm-hmm trying to do something about it. So it can't be trivial.

Steve Hsu: Yeah.

John: Who are the people who you respect in the, in the field and who you listen to?

Jaan Talliinn: Mm.

Yeah. I mean, I really do like the AI 2027 people. They're just, like, so clear Have like so clear thought and, and are like so grounded. And yeah. I mean, Eliezer Yudkowsky has been kind of like a controversial figure from time to time. but it's like just still, like he was the guy who kind of brought me into this topic, and I haven't seen him kind of be, unfortunately, I should say be just like completely off base, in, in anything really.

So like still kind of he, he continues to have my respect. Yeah. Then like I'm pr- on the o-other hand, I'm pretty well, I have like pretty good relationships with, with all the people leading the companies, AI com- top AI companies. So certainly they, they also deserve. Yeah. They, they also I respect them.

John: Speaking of Eliezer, and what do you think of MIRI's, you know, suggestion or their push to have an international treaty? Mm-hmm. and basically their approach, because I know you supported a pause, and I'm just wondering, you know, there's the different movements. You have like Pause AI, you have MIRI, and what more, what specifically you think would be best?

Jaan Talliinn: Yeah. I think the main, main thing that-- One big thing that is missing is just like the international realization how deep trouble everyone is in because of this AI race that is happening mostly in the US. And like if there was like a much bigger realization about this, like creating these international treaties would be easier.

It's now possible. Like I've seen some positive signs with Claude Mythos Tropics Mythos thing, that people are now starting to take this more seriously. yeah. I mean, just, you know, just today I got like some incoming pings from non-US people inquiring about the situation with Mythos, et cetera.

John: You talked about, the Chinese government being an important player. For I know people in the audience will go, "Jaan, you're a well-connected guy. You know a lot of these people. Why can't we just talk to China and say, 'Here's how it is.'" Is it, the incentives are too strong until there's enough political support or the inferential gap is too great?

Jaan Talliinn: Yeah. I think the main missing piece is still kind of like I would say respect for what AI could be. So as long as people are kind of like dismissing. Like people who could be in a position to talk to China are dismissing AI as something that like some piece of nerd math.
Then yeah, we're not there yet. So like they should start taking it much, much more seriously. And, and luckily it seems to be that that this process is underway.

John: And for the US lab leaders, you know, we, we talked about Dario and Demis coming out and saying that we'd be open to a pause. Mm-hmm. people, when you talk to them, they're going, "Well, if this is so dangerous and that the lab leaders have acknowledged that this is-- has a great risk, why are they charging forward?"

If you could say something about that, and then i-if you think that Sam hasn't, I think, spoken publicly about being open to a pause and why you think that is, if he, he has publicly acknowledged the existential risk. Mm-hmm.

Jaan Talliinn: I was thinking I kind of like lost the first part of the question. What was the first part of, part of the question? It had like two, two part of the question. So,

John: why are, why are the lab leaders who are acknowledging the risk racing ahead? Oh, yeah, yeah. Okay. Yeah.

Jaan Talliinn: Yeah. I mean, it's, it's sort of like a game theoretic situation here that it's like not really possible to slow down the race unilaterally.

If you like slow down your company, it means that you will just be not participating in the race, whereas the risk race continues. It for a while, Demis had this kind of strategy of just like building a company that kind of develop like a-- that can develop comfortable, comfortable lead in AI, and then basically at one point stop and kind of spend down the lead in terms of increasing the safety of the situation.

But that didn't work once like OpenAI got created and then like Anthropic and xAI and whatnot. We have like so many players who are, because of the movement of talent and like ideas in general, are like roughly, you know, they're not like very far from each other. which means that, yes, like if, if any company would, would slow down, they would just like remove them from the race, which is not really having the effect that people naively would think.

And there's like another kind of when you. There are two reasons why a, why a lab or like AI company leader could acknowledge the existential risks. One of them is that that, that they are kind of genuinely they just speak the truth. They, they care about the, the AI not going massively wrong. But yeah, there's another reason.

It's kind of instr ental reason, is that they, that leaders who do that, they are more legitimate to their team because the team understand that. Like, at least a significant part of the team they're smart people. They understand that AI can get out of hand. So if you are, if you are, like, someone at the lead who is, like, dismissive of this risk, like Jan LeCun, et cetera, it's just much harder to find talented people because, like, you're not really a legitimate leader.

I think, like the way how Mark Zuckerberg, like, ended up paying, paying billions in order to kind of buy, buy up talent is, like at least some evidence for being dismissive about AI x-risk being, having like a real monetary cost. So that's, like, a, a little bit unfortunate that, that sometimes you have, should have, like, some uncertainty like why the leaders are talking, talking about it.

But I'm pretty sure that many of the leaders genuinely believe that.

John: That this is a genuine risk. Why do you think Sam hasn't talked about a pause? And do you think he would be open or if, if he saw capabilities he was-- that made him more afraid? Or what do you think the difference in psychology there is?

Jaan Talliinn: Yeah, I don't know that. Yeah. I haven't spoken to Sam for like two years now, so yeah, I don't know.

John: Is it... Personally, is it hard to, sort of be friends and close with the lab people, but also be so worried about what they're doing?

Jaan Talliinn: Practically, not really. Because yeah, as I said earlier to another question, that, like, I have my kind of system two conscious part that is doing all the worrying, and my system one intuitive part is kind of clueless about, about the AI risk. so I think like in interpersonal relations they are kind of like very, a big function of the, of the system one intuitive bit.
So it's, it's not, not difficult to be friendly with people.

John: Do you think that's just innate, like you just have an optimistic personality, or did you have to do some, some psychology hacking to deal with the

Jaan Talliinn: I, I never have done any psychology hacking. I guess my system one is kind of naive. Yeah.

Yeah, by default.

John: I want more of your system one.

Jaan Talliinn: Yeah.

John: You had talked about the war in Iran having an impact on AI. Could you say more about that?

Jaan Talliinn: So just at a meeting here in Bay earlier this week where someone pointed out that a lot of the Infrastructure build-out was meant to happen in Middle East, and that's now a war zone. So it's unlikely to progress as planned. Which I think from like the AI city perspective is, is great news. Yeah. It's like we're happy about this. Yeah. That's War. And then also, like, the energy flows have been, like, significantly disrupted which apparently affects both the, kind of data center growth as well as the chip manufacturing actually, from what I heard.

John: You had also talked about that, the fourth player, the rest of the world is asleep at the wheel and could be more of a player. Yes. What were you thinking when you said that?

Jaan Talliinn: Mm. I mean

John: Or if you were talking to the leaders of these.

Jaan Talliinn: Yeah. Almost... I mean, the vast majority of people and economic resources are commanded by this fourth player on this planet.

So there is definite kind of like latent bargaining power for the rest of the world. But yeah, the rest of the world really isn't using it at this point. So, like, if there was really kind of like alert in a sense that, like, I don't know, Denmark was, like, super alerted by, by the threats by the US to annex Greenland. If the rest of the world were in that kind of level of alertness, I think, like, the situation would change pretty quickly.

John: Do you think the reason, they aren't is similar to China's reason where they're just not caught up on
Jaan Talliinn: Yeah, yeah, yeah AI? They just, like, don't understand what AI is Yeah
and, and, like, how it actually could have international impact rather than just some people for, for some electronic services in the US. Yeah.

John: That's a part of why we're making this doc entary, is I think so much of this is proximity based, is even though these things are in the news. Like I was just in New Orleans and I was talking to some software engineers there.

Mm-hmm. And they just were not caught up at all about, the state of AI, even though they, I think, use LLMs

Jaan Talliinn: Oh, interesting in their

John: work, but they just had, had no idea about any of the conversation about this. Mm-hmm.

Jaan Talliinn: Mm-hmm.

John: Even though they're, they are software engineers. And so the average non-tech person I still think is just.

Jaan Talliinn: Yeah. So, like, even if it, it, if that's the situation even in the US, right? Yeah. So, so yeah, the rest of the world is, like, still even, even less informed about this. Yeah.

John: When you picture the future, what do you picture?

Jaan Talliinn: Yeah, I'm just, like, so... Yeah, we're getting, getting into this like what's the, what's the word? Like, Vinge-an uncertainty where the future will be dominated by processes that are smarter than I am so it's very hard to tell how it unfolds. I do think that, like, a positive future would be something where we would slow down a lot and then just kind of adopt the current AI tools that are indeed

John: Presenting the best arg ents of both sides, the accelerationism and the safety people. Mm-hmm. And want to, as we said, kind of like educate the mass. Mm-hmm. Would you say yes? I would just, like, tell it to my team to investigate and depending what they come back with, yeah. Okay. Do you ask AI, "Do I trust these two people?"

Jaan Talliinn: Team lead. Hmm. Okay. We're good. Good? Yeah. The first question, I wanna get to is, I only have a few, very quick one. Mm-hmm. So the, this communication of risk to people or the perception and communication of risk to people, there are a lot of, like, techno-optimism or techno- Mm techno-optimists out there.

John: To me, it feels like an easy sell, abundance, all these cure cancer, et cetera. I ass e you two, I ass to some- I, I don't think the selling is going well, but go on, continue. can I ask why? What, what would you mean by that? I mean, the public is not buying it, is it? Yeah. So yeah, they're not, yeah.

Jaan Talliinn: They can do the selling, but it's not working. What do you think the public sentiment... Like, how would you describe public sentiment towards AI? In the West, they... I mean, there have been, like, lots of polls that just show that people are very uneasy now about, like, what it means, what is potentially mach- what those machines that are potentially smarter and more capable than they are will mean for their future and the future of their children. Okay.

John: So, but they are not... Are they thinking about existential risk? So far, I understand most of the public is not thinking about existential risk. they just worry about their jobs and you know, social structure in general. Is that your... Is, is it x-risk your main concern? Yeah. My, my main concern is that people will not live very long now.

Jaan Talliinn: Yeah. How do you understand people who seem to have access to all these arguments and facts, however seem to be oblivious or resistant to this kind of a risk belief? Yeah. I think there are, like multiple, multiple groups, that have very different reasons to be kind of immune to these concerns.

The obvious one is just, like as U-Upton, Upton Sinclair has said, "It's very hard to, to explain something to someone whose job depends on not understanding it." so they might just have financial interests. that's like the most obvious reason to be dismissive. Another one is that you have people who, in principle, can't imagine that machines can be competent in a way that can be dangerous.

I think that's also, like, pretty large. So that's so, like, they, yeah, they say things like, "Oh, why don't we just, like, unplug?" Like, as long as it's just gon- some kind of, ... It's just like they don't understand what AI could potentially be capable of, of. They... For them, it is just like an app on their phone, right?

Like, what, what, why are you afraid of an app on your phone? So that's just, like, not informed. But of course your question was, like, about, like, people who are not informed, obviously they haven't, like, like, been exposed enough to these arguments or, or, or are somehow incapable of following them.

John: I was more thinking in terms of people like Yann LeCun. Mm-hmm. Pi- Piotr Diamandis Alexander Wissner-Gross. These kind of people, they... I ass e they have the intellectual capacity to actually understand these nuances, these kind of arg ents. However, they also seem genuinely believe. I mean, first of all, like, I think, like, Yann LeCun's position has changed over time.

Jaan Talliinn: So he was, like, completely dismissive of any risks, and now he's, I think, like, dismissive. Says that, like, nothing bad will happen in five years or something like that. I, I don't actually know what his latest position is. But there could be... I mean, it's very hard to, like, psychoanalyze people and, like, know, know exactly what's what's kind of, what's kind of driving their immunity to these arguments. But I think, like, some people, including some of my good friends, they are... sort of, like, built a brand around, being dismissive about these.

So in some ways, it's almost kind of like a part of their identity to not buy the arguments. Well, because they have to live up to that- Yeah ... false- Yeah ... image. what... On this, what any thoughts on what would be the best way to actually communicate these kind of high-stake risks to the, to the mass, I guess?

The mass or the, the, the people who are dismissive? to the mass because- The mass. Yeah. Hmm. Yeah. I don't- I have, like, very good prescription here. I do think that it's valuable to... I mean, Future of Life Institute kind of continues doing

I don't know, educational campaigns. One area that I think is valuable is basically demonstrations of capabilities that AI companies themselves aren't really interested in doing or interested in. Just don't, don't have, like, a... It's not a priority for them. and, so I just came from, from TED Conference and Peter, I forget the last name the creator of OpenClaw gave a talk there, and a lot many people asked me afterwards, like, what did I, what did I think of that talk and, and OpenClaw in general.

And my, my answer is that I do think OpenClaw is kind of net positive at this point, even though obviously many things can go wrong, because it serves as a demo of what AI, AI... What the latest AIs now are c-are capable of. If we didn't have OpenClaw, we still would have the risks, most of them at least, but, but no, no visibility into them.

So I do, do think that demos is a good, at least one good potential approach to educate people. Earlier, Steve brought up this name, Nick Land, and you said you know a sort of snippets of his beliefs. Mm-hmm. This cap techno-capital, machinery, and you think this sounds about true.
John: and also I think for different intellectual orientations, like physicists studying, you know, the, the cosmos and different theol-theology traditions all believe in something bigger than just human kind. Mm-hmm. I think it's quite a human -centric worldview, and value seems to be a quite recent and in some sense isolated kind of, you know, phenomenon.

I'm curious to what extent you feel sympathetic to this kind of, not necessarily like a misanthropic, you know, - Mm-hmm ... kind of value, but just, like, believe in something bigger, and as a result, I guess some kind of resignation or some kind of, like, just, like, try to align themselves, like along with this, like, so-called cosmic trend.

Mm-hmm. I'm curious about your take on this. So I don't want to, like, fully dismiss it. just I have, like, some uncertainty, especially when my- Friend Nick Bostrom kind of seems to have pivoted into, into worrying about, like, whether we kind of fulfill our responsibility to develop something bigger.

Jaan Talliinn: But that said, yeah, I think it's mostly misguided. Like, in some ways, we are humans, and, like, human values is what we have. And so, like, if we are, if you are kind of painting a picture of a horrible dystopia in the future and according by human lights and calling this is something bigger, I think you're just, like, sup- super confused.

Like a grand future with, like transcendent beings, et cetera, should still be something that is also good by, by human lights, not by just some kind of abstract. I mean, I grew up in Soviet Union. It was like a... It, it was a dystopia ruled by, by this, like abstract loft-sounding ideas.
like, abstract loft-sounding ideas is like a... I have, like, a bit of skepticism of all of them. Let's put it that way. I grew up in a communist China. Oh. I can relate to that. All right. I moved to the States only- Yeah ... after I finished college in China. Yeah. And, like, though my kind of visits to China, like have been...

Obviously, like, I, I wasn't part of, like China has seen, like, much, much worse situations. But, like, my visits, I think my first visit was in, like, early 2000s and, and I was, like, kind of bracing myself for this, like revisiting my childhood. But then was like, "Oh, it's actually much more, much more soft and liberal place than, than I remember the Soviet Union to be."

John: Yeah. Yeah. 2000 was already, you know, kind of, - Yeah. Yeah after WTO and, yeah. Exactly. Things were changing. Yes. You seem to have this. You occupy seems a very unique place in, it seems, like, as an investor, both in the frontier labs and at the same time very tapped in, buy-in to this safety risk arguments and, and investment.

Is navigating through this space having this kind of dual identities tricky? Or what's your... What has your ex- experience like in this, in the context of these, like, dual identities? Yeah. I don't think it's, like, super tricky. a little bit. Like, for example, I deliberately didn't want to take a board seat at Anthropic even though I led their first round Because I, I, I perceive that that would be like a conflict of interest, w- and would interfere with my ability to work with, like, Future of Life Institute and whatnot.

Jaan Talliinn: But yeah, I kind of see my role as, as someone who kind of tries to catalyze positive things rather than someone who kind of like coordinates things behind the scenes or something. so if the... If any of the companies doesn't want to do a thing, I don't think I can make them to do a thing.

John: Mm-hmm. My, my last question would be since we're now at Lighthaven, I think some of the earlier questions also touched upon your involvement in this space and AI safety ecosystem in general. Do you mind sharing some thoughts on perhaps what you think are, some of the space for improvement s- for some optimization in the whole setup of the AI safety ecosystem?

Because I re- Mm-hmm I, I was listening to your last podcast to with Steve, where you made a comment that a lot of these kind of investments in the AI safety is immediately channeled back to the capability. So now it's kind of a bit dubious, you know, what impact is. So, like, what-- how do we actually optimize it or improve it?

Jaan Talliinn: Yeah. I

It is like the way I've been putting it is that, like, sign uncertainties are everywhere. So like you can be much more certain that you are gonna have a impact, but is it going to be a positive or negative impact? Like, much more uncertainty about that. So yeah, like many of the techniques that or many of the, like, technical research that the AI safety community has done actually has helped the AI capabilities research.

So you can't kind of draw a, like, very distinct line. but now that we are in a, again, as I mentioned earlier, the AI safety is kind of comfortably or uncomfortably behind in terms of like technical research of the, of the frontier. when we are more thinking about how to actually help society navigate this, like, upcoming period, I think what would really help is having just like a broader, broader composition of people.

Like people who've been kind of like self-selected into, into this AI safety group that are worried about this abstract thing they tend to be very much kind of not like median people. So, so it's like I think there, there is, there is-- could be a kind of like, lot of value in kind of broadening, for the coalition that is worried, worried about AI.

John: Hmm. Okay. Lastly, besides dancing, how do you, how do you cope with existential dread these days? I know you have system one, system two, but if you one day get bogged down by that. Yeah, I, I

Jaan Talliinn: Since I started working on kind of AI safety like seventeen years ago or so, like the entire job basically has changed. Like I think the first ten years were just like, so interesting and fascinating. You kind of like get to talk to philosophers and talk about these abs- abstract ideas and learn about the mind and, and, and like like novel things that computers can do and et cetera, et cetera.

And then like participate in these like crazy-sounding startups that are actually making surprising progress. And it, it, it was, it was so interesting. And now it's just a job, basically. Like talk to politicians who don't really understand what's going on. and so like the way I put it is that like if it, if AI wasn't a thing, at this point I would be actually much more happier just dancing and doing building software, which I always loved.

So yeah, like dancing and building software are my two things that I actually care about in terms of, you know, professional, like, hobbies, I guess.

Earlier you said you are not sure about the direction of impact. You know, we're making impact, but we're not sure of impact.

John: Mm-hmm. I hope our effort today, your investment today, like your time with us, I hope will maximize the positive impact for your time invest-investment with us today. Yeah, I hope so too. Yeah.

Thank you very much. Thank you. Okay, a couple quick questions from John. Yeah. Is that what you wanted? Yeah, yeah.

And then we're... And then hip hop. Then hip hop.

Jaan, I wanted to thank you because you had a not insignificant impact on my younger years with Kazaa. Oh, really? It was ... I have such nostalgia for that, for that time, and downloading things, music, and whatnot. And I was wondering what that experience was like for you, and what it was like to be a part of something that was, like, very much in the news and controversial.

Jaan Talliinn: Mm-hmm. Yeah. I mean, we were... It was m- these are my young and reckless fear- Yeah ... y- years. And also, like, they went quite quickly. I mean it was, like, a year, a year and a half or something. So it wasn't, like, a big period. Yeah, it was the first time you kind of saw this global phenomenon basically.

And, like, I remember sitting in my apartment with my co-founders and one of them claimed that now we are at, like, 60% of the internet traffic in the world. I don't know where he got this, got this data, but it sounds plausible. - Yeah, this was, like, a huge deal. Yeah, yeah, yeah. Like- It was... I mean- Everybody I knew was downloading.

Yeah, yeah. No, because i- it kind of... It was, like, media, right? Yeah. Back then that kind of probably was, like, significant- Yeah ... contributor to- There was no YouTube. Yeah, yeah. It was before YouTube and stuff. Yeah. yeah. And then, like, the lawsuits hit. Yeah. We were sued in three, on three continents. Yeah.

So I was exposed to the intricacies of various legal systems and did, like, depositions and, - Were you stressed, or were you just like, "Eh"? Yeah, that wasn't super stressful because like, the companies were, kind of, sued.

John: Yeah.

Jaan Talliinn: But I was always just, like, a contractor

John: Yeah

Jaan Talliinn: To the companies like, kind of in formal terms.
So I wasn't, like, personally kind of targeted by the lawsuits. And you have the ChillSystem1,

John: So you're- Hmm? You have the ChillSystem1. Yeah, that too. Yeah. Yeah. Yeah, so- Would you download things from Kazaa yourself? Yeah, I did. Oh, really? Okay. It was, like, such a, such a phenomenon, and then it, it

Everything died after. There was so much attacking of it, but

Jaan Talliinn: Yeah. I mean, also like so iTunes, - Yeah. Yeah ... started

John: And then Spotify. Yeah.

Jaan Talliinn: Yeah.

John: What's your favorite movie of all time?

Jaan Talliinn: I have like a list of, of movies jaan.info/ fa- fav movies. Oh, really? Fav movies. Yeah. Yeah, there are like several, several that, I mean, yeah.

It's hard to pick like one. Yeah. I do like one, like one that just comes to mind is, is Before Sunrise. Oh, yeah. And like one, - Steve and I always talk about- Really? Yeah. Before Sunset. The Before trilogy.

John: Yeah.

Jaan Talliinn: Yeah. And the funny thing about this is that the third movie in the trilogy starts with a discussion about Skype.

John: Yeah.

Jaan Talliinn: Yeah. How did it feel to be- Yeah, it was, it was like, when I was watching this I was like, "What?" Yeah. Yeah, yeah. Did I just put myself in my, to my favorite movie? Yeah. So yeah. Linklater
John: is like- Yeah, yeah ... channeling. Yeah. Yeah. Those are some of the most beautiful and sort of real or accessible romance movies.

Jaan Talliinn: Yeah, first time I went to Vienna, I actually like did a tour of the places. Yeah ... where the movie was filmed.

John: Oh, really?

Jaan Talliinn: Yeah. It's, i- it's funny because like the, in the movie, they're gonna make it look as, as if you're just like a... You turn a, turn a corner and then- Yeah ... you're another set. Yeah.
Yeah. Like five kilometers apart. Yeah, exactly. So it's- Exactly ... it's not a, not a very intent or not very concentrated tour.

John: It's

Jaan Talliinn: fun.

John: Was it weird, like you mentioned being mentioned you know, Skype being mentioned in the movie, it, was it weird becoming very successful? Or like, was it just pretty much amazing and enjoyable or surreal?

Jaan Talliinn: Yeah, because we had seen the success with Kazaa before. Yeah. We kind of like in some ways... I mean, after Skype was launched then just in the first few days we thought, okay, this is exponential. Yeah. And and I even draw like a, do like a chart on a, on a log scale- Yeah ... and see like how many months we can go before we run out of, all of people on this planet.

Yeah. I was like, "Okay, yep, here we go again." Yeah. So, so yeah, I think the experience with Ka- with Kazaa, yeah, took the edge off of the, of the Skype thing. But Skype was obviously like much more, much bigger, much more positive- Yeah ... because Kazaa indeed was controversial like all

John: Tthe time. But it, for the biggest j p in like your personal happiness or utility, do you think that Kazaa provided the biggest j p or Skype or something else?

Jaan Talliinn: I don't think they kind of kind of, inter- interacted or coupled with my happiness too much. Really? Yeah. I mean, o- one thing is that even before Kazaa we were k- Like, the team behind, the engineering team was, like, we were pretty successful. I ran a games company- Yeah, yeah ... for, for 10 years, and yeah, we were, like, pretty, pretty. Like, I wouldn't say, like, well off, but, like, well, yeah, we were, we were not, not bad. Not, not, not, like, yeah, we were not doing too bad. Yeah. And so we, yeah, it was kind of, again, there was, like, no big sort of moment- Yeah ... of, of success. It was kind of, like, gradual- Yeah ... over time. Yeah.

John: Is it awesome being, like, a billionaire, or are there huge downsides or?

Jaan Talliinn: No, it's, it's fun. Yeah. But what do you think of all the... I mean, I think you're different because, you're sort of one of the good guys. But what do you think of all the billionaire hate that people just have this sort of, like, primal reaction about? Like, oh. Yeah.

I mean, yeah, I think, like, many, many billionaires, they are also sort of like leaders of companies. Yeah. And they're like, you're, I think you're much more kind of constrained. You can't be, like, chill. Yeah, yeah. so, so, and also, you know, Musk, I guess, seems to be kind of, like, a- addicted to, - Yeah ... the social media, which is just- Yeah super unhealthy, I think. Yeah. so I think, yeah, my, my advantage is that I don't have, like, these constraints of, of needing to perform in a certain sense. Like, either I'm on social media, I'm not on Twitter, for example- Yeah ... or, or like perform in front of

John: Like a company or something. Yeah, and I think your companies were also more loved.
I mean, even Kazaa is, like, the- Mm-hmm ... the average person, you know, was- You know? Or Utorrent. Yes, yes, yes, exactly. Yeah. who do you think is the smartest person of all time?'

Jaan Talliinn: I mean, that's, like, a fairly common answer to that is that John von Ne ann- Yes ... was, which is, like, just super likely. Yeah. Yeah.

John: Steve was responsible for me and, you know, everybody else getting super into John von Ne ann, and I got so into it that I tracked down his daughter, who was, I think, 87 at the time. Okay. And I interviewed her in her retirement home- Awesome ... in Boston. which was difficult because I had to go through all these hu- hoops, and she actually, like, forgot the schedule, you know, because of her age.

So I had to, like, get in there. But it was, it was cool to... You know, she had all these stories about Einstein and all of his- Mm-hmm. Does he have gra- grandchildren? Does what? Does he did John von Ne ann have grandchildren? Yes. Oh. Yes. I don't think any of his progeny was as successful as him. Yeah. His daughter was pretty notable, especially for a woman. You know, she was I think the chief economist or something under Nixon.

Jaan Talliinn: Mm-hmm,

John: mm-hmm. But so far, I don't think they have the- Yeah

Jaan Talliinn: yeah. I mean, clearly it was a fluke.

John: Yeah, yeah, the super, super genius. Mm-hmm. And who, who's the smartest person you, you know?

Jaan Talliinn: It's hard to tell. I mean, David Touretzky is, like, a, a strong candidate, I think. Yeah. I don't know if you know David. What does he do? He used to run ARIA, ARIA- Mm-hmm ... in, in UK. Or, like, was a, at least some leadership position there. And but he just, like, stepped down a couple weeks ago, so now he's gone.

Yeah. Yeah. Doing.

John: planning to do some other things, I guess. He should j p into the AI safety, - Yeah ... AI safety boat.

Jaan Talliinn: I mean, he, I mean, when he was doing ... He was, like, a, one of the couple of figureheads of this, like, guaranteed safe AI. Oh, okay. Yeah, yeah. So, like, if you, if you look him up, he's okay.

Yeah. I gotta, I gotta research him. Yeah. He's, he's like a... He's the person who has the, has achieved MIT graduate degree at the ripe age of 16. Ah. Which is, like, the world's record. I hate those

John: people. All these prodigies are making us feel bad. He's smart. He's smart. Yeah. Mm-hmm. You have six kids?

I have six kids, yeah. I have four daughters. Oh. Do you have parenting advice for me?

Jaan Talliinn: Mm-hmm. I mean, one, one thing is that your kids usually are smarter than you think they are because you're kind of anchored. They keep developing, right? In some ways a kind of, like, similar situation with AI. Yeah.

Like you're probably underestimating their capabilities. so try to kind of, like, extrapolate a little. Yeah. And treat- I gotta posit, posit- Yeah. ... here. Treat them more like an adult than, than you would by default.

John: Yeah. I think David Friedman says the same thing- Mm ... about, ... It was, it was easier for him because he just tr- treated them like little adults and

Jaan Talliinn: Mm-hmm

John: Mm-hmm, that worked out. Yeah. Yeah.

Jaan Talliinn: Yeah.

John: Yeah. How many times have you been in love, Jan? I don't know. Something like three. Three? Something like that. Okay. Three's pretty good. Three I, I think as an average number. What are you most proud of? You've done a lot of stuff.

Jaan Talliinn: I mean, a couple of things that already have come up, I think it's... I'm, I'm pretty happy how they have gone. like, yeah. There was like... I remember after Skype there was some kind of, like news- paper article about some, like, rich list or something like that in Estonia, and the commenters were of course like, like, "All rich people are criminals," et cetera, et cetera.
And then somebody goes like, "Do you also mean the Skype guy is a criminal?" And I was like- Yeah. You're like- Yeah ... yeah. I feel pretty proud of that- Yeah, yeah ... that people don't think I'm criminal. and yeah, I mean, when people give me compliments for dancing I'm like, "Yeah, I can do that. That's fun." Whenever I go to a wedding or

John: Something like that, I'm always jealous of the people who can really, like, show off.

Jaan Talliinn: Mm-hmm.

John: You know? I feel like I wanna take a page out of your book and yeah,

Jaan Talliinn: I mean, dancing is, like, so underappreciated for guys. Yeah. It's like, like, almost, almost no guys can dance. Yeah.

And if you just, like, put a little effort into it, like, really fun- Yeah ... stuff will happen. Yeah. I think,

John: it's some low-hanging fruit for- Yeah, it's just, like,

Jaan Talliinn: so low-hanging

John: fruit. Yeah. - And who could... Who would you meet if you could meet anybody,

Jaan Talliinn: in history? In history? Oh Feynman, John- Feynman ... yeah. Okay. Richard Feynman. Steve got to meet, got to

John: be buddies

Jaan Talliinn: with him. Oh, nice.

John: and as a game developer, I have to ask, what's your favorite game?

Jaan Talliinn: I mean, I play Civilization. - Yeah ... like so in Microsoft as a revealed preference. Yeah. Yeah. Civilization. Just the first one? No, I played, like, many of them.
- Yeah ... not I think I played also first one, like, briefly, but

John: yeah. Did you play Ages of Empire, or Age of Empires or Command & Conquer when it was- No, I haven't played it. I know it, but I haven't

Jaan Talliinn: played it. Yeah. Yeah. Also, like, I, since I stopped doing games, I actually, like, yeah, with the exception of Civilization, I just don't really play much.
Just hard to find time. You have better

John: things

Jaan Talliinn: Yeah.

John: Okay. And a few questions from the audience. What is your advice for politicians who are concerned about the future of AI?

Jaan Talliinn: just educate yourself about it. just use AIs to see, like, what they're about and how they are, how their capabilities are increasing.
And please, please do not underestimate where this trend is going.

John: That was from Max. And Robert asked, "Do you think transformers are an existential risk, or do you think we need one or more additional insights? Like, do you, how important do you think embodiment is?" Yeah. Many questions there.

Jaan Talliinn: I obviously have uncertainty.

It's possible that the transformers will not get us all the way there, but currently we do not see signs of stopping. so it's, it's, ... Like, if I had to, like, make a side bet then I would bet that, yes, transformers is all, all we... Attention is all we need. but yeah. And embodiment, I mean, there are, like, there is, like, some very interesting work by Michael Levin and, and people like, like him, that there is, like, the human is more than just the brain- on the other hand, I think, like, there are counter-arg ents that like, like some human s do not have, like whole, whole body.

And like some of them are paralyzed some of them are paralyzed from the, from the birth. So like, it's... As a friend of mine said once that, like the embodied embodied intelligence people, quite often they make claims that are interesting but obviously false. Or, or claims that are obviously true but uninteresting.

Mm. That you need, like, a- Mm ... you need a physical manifestation in order to be, in order to, in order to actually run the intelligence. Yes, sure. and then like the claims that are interesting that actually need, like hands or, and legs- Mm-hmm ... in order to be as capable as human s is like, no, that's obviously

John: false.
Yeah. How concerned are you with S risks?

Jaan Talliinn: Hmm.
Not a lot. And, and partly because, partly because it's just, like, a v- seems like very narrow target- Mm-hmm ... to hit. And partly because of, sort of like game theoretic, decision theoretic reasons. - Yeah ... if you're gonna get too fixated on this, you might be just becoming, you know, dysfunctional- Yeah

in, in some general, general sense. I was just having a w- walk a couple of weeks ago, I think like last week even, with someone who we, we got started talking about this, and he was like, like, "I'm not, not afraid of suffering." I was like, "Till they prefer like a life of external suffering to like not being alive at all."

I was like, "That is badass." Wow. You know? I wish I had that fortitude. Yeah, yeah. That constitution. If you have that, that kind of attitude, I think like, yeah, you're just, like, invincible.
John: Yeah. Wow. Diego asked, "Can you explain your 2013 presentation, 'Why Now?'" Yeah,
Jaan Talliinn: that was the most fun presentation that I've ever given.

Yeah, it was just thinking about this, like, weird coincidence of seeing the most potential, the most impactful time in the history of the universe. And then, like, offering one kind of explanation for that, that we might be living in a simulation or a set of simulations where these, like, mature civilizations run by AIs and whatnots that are separated from each other by space-like distances in the sense that they can't directly commu- communicate.

They are basically attempting, attempting to communicate by simulating each other. In order to simulate other regions where there are, like mature civilization, civilizations, you need to need to model the creation of those. similar s-advanced civilizations. So you need to model basically what are the grow lengths of 21st centuries.

So in that sense, we might actually not be-- shouldn't be super surprised to find ourselves in the 21st century because, like, this is, like, very attractive place to simulate. Great. Thank you so much for talking with me. Thank you.

John: Can I ask one more question? Sure. What do you think are the reasons for your success?
Mm.

Jaan Talliinn: a lot of it is just-- I mean, okay, well, I think two reasons. Like, one is just, like, seem to have won the birth lottery in, in terms of just being smart. And the other is just, like, trying many things. Like, I mean, known for Kazaa and Skype, but, like, there were, like, several projects that we tried before and and after and between.

So it's in some ways just, like, a bunch of selection effects. And then just, yeah, like these two things combined basically have helped me to just, like, make connections with other very smart people. and then kind of that kind of-- There's like a self-enforcing, self-reinforcing kind of flywheel, that,

John: helps you do more and more interesting things that way.
When you were younger, was-- did you feel more disconnected from people because you had fewer smart people that you were connected to?

Jaan Talliinn: Yeah, I, I think so, yeah. And I-- Well, there were two things that kind of made me feel disconnected. One was that I was in behind the Iron Curtain. Yeah. And, like, growing up in a, in a nation that didn't really seem to have, like, lots of future ahead of it.
And and the other thing is I was, like, a massive nerd, so I was just, like, reading a lot mostly, like, reading books. you know. What's your favorite book? My favorite book?

Yeah, I don't think I have anything that kind of stands out. Yeah, there are, like, bunch of books that I have liked, like Charles Robert Wilson, Spin. -h . just, like, recently I was talking-- yesterday I was talking about it because that's why it's, you know, in my cache is, is because the book, especially the audio version, I really liked.

There is this, I think, what, Matt Suarez Demon. Mm-hmm. which was very, very interesting.
Yeah, I'm probably forgetting a couple of books that actually I, I think are even better. What about music? yeah. My my music kind of taste has shifted over time and but generally re-remain rather broad. But because as my kind of dance hobby, I tend to listen to quite a lot of, like, dance music.
Yeah.
​