Steve Hsu: This message is for some friends at Mechanize, a startup that builds environments for training and evaluating frontier LLMs. Its customers include the top AI labs, and it has contributed to the breakthrough in coding capabilities of frontier models. Mechanize is hiring. See the links in the show notes.

Compensation is extremely competitive. For technical roles, $300 to $500,000 per year in salary, plus additional benefits and equity compensation. They are also seeking smart generalists. For example, a research engineer position focused on alignment. This person will build evals that test for misaligned model behavior.

Salary, $500,000 a year. Puzzle maker. This person will design interesting and original puzzles that LLMs cannot yet solve. The salary component of compensation is $300,000. Mechanize understands that my readership and my listenership on this podcast is highly selected.

There is a very good chance you will be interviewed if you apply via the link in the show notes. I already know of departments that have discussed, not that they've implemented, but they've discussed reducing the number of PhD students that they admit because the professors can get a lot of productivity out of using AI, whereas bringing a student up to speed, taking a student who's just completed their undergraduate degree and bringing them up to the frontier where they can actually do things which are useful to a professor that takes years, and it's super labor-intensive, and a lot of professors would just rather use the models to do their research than go through that process with the students.

Now, in the long run, this is a problem because, we, we need to train the next generation of physicists or mathematicians or scientists. But it's possible we won't need as many given the productivity multiplier that we get from AI.

Welcome to Manifold. We're here at the Mandarin Oriental Hotel in Taipei, Taiwan. I am recording this episode all by myself. The title of this episode is going to be AI Summer 2026. The whole episode will be about things related to AI, and also a little bit about my summer travels. So in the last few months, I've been in San Francisco and Berkeley, Bangkok, Singapore, Hong Kong, Beijing, and Taiwan.

Steve Hsu: I attended these crazy conferences at Light Haven in Berkeley called Less Online and the Manifest Conference. I also worked on a documentary film called Machine God, which I'll talk about in this podcast. What you're seeing on the screen is if you're watching the YouTube version of the podcast, what you're seeing on the screen is images taken from my travels.

So these are all photographs that I myself took, and I've just got them playing in the background here. For those of you that are just listening to the audio version of this podcast, you of course won't see these photographs but you can see them if you go to the YouTube channel.

So let me start with topic one, which is AI and math and physics.

And if you've been following this you, you may remember that about eight or nine months ago I wrote a paper, a physics paper which was published in Physics Letters B. It was about nonlinear modifications to the Schrödinger equation. And the main ideas for that paper actually came from GPT. So what I did was I submitted the paper under my own name, and it was accepted after review.

And only after it was accepted did I post a companion paper which describes how that paper was written and revealing that the main ideas for the paper had actually come from suggestions from GPT. And I got a lot of pushback on that paper. I think a lot of people, because it was an AI-generated paper, really didn't like it.

And I don't think there's anything technically wrong with the paper. I think it's actually all correct and I think, at least to me, quite interesting. But I think a lot of people just had a negative reaction to the fact that AI was involved. Now, fast-forward just six to nine months, and the situation has changed drastically so that now there have been many important results published which, either had a significant amount of AI participation in the generation of those results, or even the results were generated almost entirely by AI.

And these are papers in areas like math and theoretical physics and probably also some other areas, although I don't, I don't track these other areas quite as much. And so I think most researchers now in fields like math and theoretical physics, especially the younger ones, realize that the situation is changing, and these models are extremely useful for researchers.

Most of the researchers that I know are using the models on a daily basis and using them to check calculations, review what's in the literature. Occasionally, the models will actually generate new ideas that are actually haven't appeared already in the literature and are, are interesting or useful.

So, I don't think there's very much doubt that these models actually are useful for research in frontier math and physics. I think there's still some debate about whether the models are capable of generating, quote "genuinely original or genuinely new ideas." What I wrote in my companion paper, and again, I should have mentioned this, there's a, there's a whole Manifold episode devoted to this so you can go watch that if, if you're particularly interested.

Although, you know, that was six or nine months ago, which is like sort of like a lifetime ago in terms of AI, the progression of AI. There's some debate about whether the models are actually able to generate quote, "genuinely new or genuinely original or genuinely novel ideas," or whether they simply combine ideas that already exist in the literature.

And what often happens is you have some area of, say, math and another area of math that are quite distant, and the people that are expert in one area are not also experts in the other. And the model, because it's read everything and it is simultaneously an expert in both area A and area B, the model can combine techniques from A and B to do something which at, you know, at the moment when the model does it and shows it to the guy who the human user who's, say, only an expert in A and not in B, it looks very amazing that the model did this.

However, in the fullness of time, when experts from both A and B have had time to look at the result and think about it and discuss it, at that point, it's, it's possible for people to say, "Oh, well, this was just a straightforward combination of ideas from A and B," or maybe there was some difficult technical calculation, but, you know, a machine could do that technical calculation.

But the ideas came from A and B. They already had appeared in the literature of A and the literature of B. And the description I just gave actually applies to what might be the most celebrated result in math that came autonomously from a model, and this has to do with something called the Erdős unit distance conjecture which you can look up.

There's been quite a lot written about this. but still the question remains: Well, can the model actually generate something completely new that, that humans hadn't already thought of? And until it does that, the model is clearly not as good as the best human mathematicians. Now, of course, the set of all mathematicians, or all reasonably good research mathematicians, is much larger than the set of mathematicians that have actually, quote, "generated some genuinely new idea."

And, I think we have to acknowledge that. And Terence Tao recently, in his comments, on this topic, has actually even been advocating for us to sort of reconsider what human research really is and to what extent, you know, humans are really doing original stuff. And he says maybe we should reconsider this and, acknowledge that humans are actually mostly recombining ideas that, you know, maybe we consciously know this idea because we read it in a paper or book, or it sort of seeped into the background and so our sort of conscious brain isn't really aware that we know about this idea, but it's sort of lurking around in our subconscious or some other, less accessible part of our brain.

And so that when we actually use that idea and combine it with another idea, it seems like it was in a completely, quote, "original de novo thought that entered our head," but in fact, it really wasn't. And I'm very sympathetic to what Terry says there. I think surely there are cases where humans innovate completely de novo, but I think those instances are far more rare than people think.

So summer twenty twenty-six, I think the status is it's very clear that the models are useful to frontier researchers. They're being widely used by frontier researchers. They're maybe not quite at the level of the very, very best human researchers. They still make mistakes. You still have to look very, very carefully at the output from these models, or you can, you know, go down, go on a wild goose chase and waste a huge amount of time.

But clearly there's been a huge advance in the capability of the models in these areas in the last year or so.

And this is, I believe, and I don't know this for sure, but I, I believe this to be true, it's largely due to post-training. So it's, you know, the models, the pre-training of the models has probably made them somewhat better in these areas.

But the post-training where I teach the model to reason following examples of the way that humans reason this is sort of very, expensively prepared data from human experts to that's used in the post-training to improve the models. Some of it's synthetic, but, but probably most of it's actually heavily dependent on humans to generate.

There's a big cottage industry of human professors and postdocs and PhD scientists being paid to prepare these sort of training datasets. It's mostly that post-training that makes the models better and better at actually doing scientific research, say, research in mathematics or physics.

So in any case, we've had tremendous advance in those capabilities, and they're my understanding from talking to the actual researchers at the labs is there doesn't seem to be a slowdown in the rate improvement of the models. Now, it, it's not the ideal situation. The ideal situation would be where the models are just sort of talking to themselves, reasoning, checking things, and then becoming better through that process.

What's actually happening here is that human AI scientists and engineers are trying to make the models better in a kind of brute force-ish way, where we are building datasets that we think you know, training or post-training on those datasets will make the models better. The training sets include really specific examples of humans solving certain problems and talking about what they're thinking when as they solve these problems.

And so all these things are used to make the models better, but in a very kind of specific brute force-ish kind of way. Nevertheless, that process doesn't seem to have run out of juice. There seems to be a fair amount of improvement left foreseeable in the foreseeable future. So, you know, further in the future, we might run out of juice or run out of steam, and this process of improvement of the models slows down.

But I think at least for the foreseeable future, the next year or so, I, I don't think people are predicting that that'll happen And given that's the case, I think most scientists, especially physicists and mathematicians who are familiar with these AI models, you know are anticipating that they'll become, you know, as useful to the senior professor as a very good grad student or very good postdoc who has already finished their PhD.

And these models don't get tired. They can do very large and long calculations. You can have teams of these models doing working on something for a long time, expending huge numbers of tokens and coming back to you with the results. So, so the whole enterprise of scientific research in these areas is gonna change dramatically.

I already know of departments that have discussed, not that they've implemented, but they've discussed reducing the number of PhD students that they admit because the professors can get a lot of productivity out of using AI, whereas bringing a student up to speed, taking a student who's just completed their undergraduate degree and bringing them up to the frontier where they can actually do things which are useful to a professor that takes years, and it's super labor-intensive, and a lot of professors would just rather use the models to do their research than go through that process with the students.

Now, in the long run, this is a problem because, we, we need to train the next generation of physicists or mathematicians or scientists. But it's possible we won't need as many given the productivity multiplier that we get from AI. So that's a little bit about AI capabilities in math and physics.

Now I want to make the connection between what I just discussed and something called recursive self-improvement, or RSI. And recursive self-improvement means that models, AI models like GPT-5 or Mythos actually improve themselves so that the next generation of that model is better, but the, the, the improvement came from the model itself.

And how would this work? So we already know that at the labs and in a lot of software companies, most of the code, maybe ninety percent of the more or more of the code is being written by AIs now. So that's been a big transition in terms of the actual economics, the business of the AI industry, is that they've found an area where they really can replace human labor, you know, not completely, but they can augment human labor and sell a lot of tokens which are valuable to the end user because those tokens are used to actually generate new code, which has economic value.

So a given model, let's take GPT 5.6, that model is made of code, right? So you have code which in which the execution of a neural net process is implemented in the code. You know, the training loop, I've heard said for, one of these large language models, even if the model itself is huge, trillions of parameters the actual code used to run the model or run the training loop for the model might only be a few hundred lines of code because the actual code is calling on libraries and the actual operation of the neural net is really just a bunch of matrix multiplications and computations of certain simple functions like neural activation functions.

And so it's really not actually code limited, right? You might need code to like generate some synthetic data or execute some eval. So there's definitely code that you need to improve the model, but it's really not the bottleneck. To me, the bottleneck is either the data. So I was just talking about how to improve model performance in math and physics.

You need very, very specific training data. That training data tends to be made by really expert human researchers. You could imagine a day when that data could be made synthetically. So you go to some program like Mathematica, and you, you have the LLM model running Mathematica, something like Mathematica symbolic math program, or if you're improving theorem checking language like Lean, and you have that process then generating reliable results, which then you can use as training, synthetic training data for your LLM, right?

So, so that, that's much more of a bottleneck to model improvement progress than the code itself And the second aspect, which I think is very important and, and underemphasized, is the architecture of the model itself. So I don't think we are anywhere near the optimal architecture of the neural network for our specific AIs for maximum learning efficiency, maximum inference efficiency, maximum inference performance.

I just don't think that at fixed, say, compute budget, we have the best architecture, and there could easily be factors of ten or a hundred or, you know, orders of magnitude between where we are now and, and the, the best possible architecture. And the search through that space of architectures is something that is ongoing, and I will argue now that the, the capability of the models to do research in areas like physics and math is very closely connected.

It's very similar or analogous to the ability of a model to think like an AI researcher and think about model architectures and propose and improve model architecture that might have a different information flow than, or information dynamics than a the current neural networks but might be an improvement.

And so to conceptualize an improvement or modification in the neural net architecture is analogous to research in areas like math and physics. Understanding the dynamics of information flow in a model, training dynamics, gradient descent, those are all very, very similar to concepts that appear in math and physics.

Writing code, generating training data to test a hypothesis about a modified architecture, those are similar to things that physicists do for example. And so the one of the reasons other than just self-interest for using the models in my own research, the other reason I'm tracking very carefully the capabilities of the models in sort of math or applied math kind of things, is that I think it's directly tied to when we will first see really effective RSI, really effective recursive self-improvement of models, and I think we're just getting to that threshold.

So in that scenario, the model itself would be asked, "Oh, propose some possible modifications of your own architecture. Test those modifications to see if they actually improve, you know, the efficiency of learning of the model or the efficiency of inference or the max intelligence capability of the model at fixed model size parameter size, et cetera."

So, so those are all things that then, you know, given a proposed modification of the neural network architecture, the model itself could do a bunch of research, do a bunch of training runs check to see if in a smaller footprint like you know, ten billion parameter, eight billion parameter model, like making this modification actually makes the eight billion parameter model smarter at fixed, you know, training data size, et cetera, et cetera.

So the, the model itself could run those experiments autonomously and then decide this proposal for the new architecture is better than my current architecture. Let me implement it in the version N plus one, so the next version of me. Let me then train that next version of me, and we'll, we'll determine whether my hypothesis was correct.

And if so, I will have just made a better version of myself, right? So we're talking about a model autonomously or semi-autonomously, could have input from human researchers throughout this process but maybe less and less over time. A model which is autonomously or semi-autonomously improving itself, coming up with an improvement on its own architecture and then testing that, and then we end up with the next version of the model.

And of course, this is a positive feedback loop because the better the models are, the better you would expect them to be in suggesting and testing improvements. So, this is the RSI scenario. I don't think we're quite there yet, but I think we're getting closer and closer. I don't think it's really that far off.

So I think we could possibly see some really significant RSI activity in the next year or two.

I can give you a nice example. So, now most cases where I can explain the example in detail. It's usually a Chinese lab that did it because they're the only ones who publish their results now, right?

I have no idea what the actual internal architecture of Mythos is. I don't know what the internal architecture of GPT is because those are closed labs. They're in a very competitive situation. They don't reveal their tricks. On the other hand, companies like DeepSeek or ByteDance, well specifically DeepSeek, I think is the best example, publish their models.

Their models are open weight. They also publish very well-written papers explaining their decision-making for, you know, certain design decisions or algorithm improvements, et cetera. And so let me give you an example called MHC, which is Manifold Hyperconnections. So this is an architectural thing. You may know that a neural network is a sort of feed-forward thing.

So you have a layer, and then there's inputs to that layer, and then there are outputs from that layer, and those outputs are fed into the next layer of the network, and you may push through hundreds of layers of this network before you get the final inference tokens out. In addition to just a feed-forward structure where you're feeding from layer N to layer N plus one, you could have some side channels where information is flow, is flowing outside of that just layer-by-layer processing.

And so you could have channels called hyperconnections, where a particular layer is allowed to write some information out to this hyperconnection channel, and that's available to some lower layer without having it, it having been pushed through. So maybe it's available to layer N plus five without having been that information having been pushed through N plus one, N plus two, et cetera.

So it's just a different architecture. It's a different way to have some side information flowing through the neural network, and maybe this improves things, right? So maybe just having some important stuff written off to the side, kind of working memory-like stuff or some side information which should affect how neural networks the different steps are taken in manipulating information.

You know, maybe that turns out to be useful. And of course, the parameters controlling how this is done are all trainable. So they're all things which, you know, as, as we do gradient descent, we, we always design these things so that modifying slightly a parameter which controls, say, the hyperconnections is itself differentiable.

And so when you do gradient descent, that parameter can be adjusted to make the model better, right? Based on the, the training data. So, this is just a different architecture. Now, my understanding was these hyperconnections were proposed originally a couple of years ago, in a paper by ByteDance, but it was found that adding these extra hyperconnections led to an instability, gradient instability in the training of the model.

So it's a technical thing there. It just, it just means it, it made the training of the model unstable or difficult. And so even though it seemed like a good idea for an improved architecture of the model it wasn't really pushed that far forward because it caused problems in the actual gradient descent training.

Now, what DeepSeek did for V4 is they came up with a very mathematical way of stabilizing these gradient instabilities, and basically they imposed a constraint on the form. So, so the hyper connection information was, you know, in matrix form, and they imposed a constraint on the matrices, you know, that the norm squared of the rows and the norm squared of the columns all had to be equal to unity.

And this is actually a geometric constraint. It's related to forcing the matrix to sort of live on a particular manifold. And by doing that, they were able to stabilize or remove these gradient instabilities. And so they came up with a, a better way to still have information flowing through these hyper connections.

DeepSeek found that by implementing things this way, they could actually train the model without the instabilities, and that the model actually had improved performance relative to the models without the hyper connections, right? So, so that's an example of a model architecture improvement.

The way the information flows or the information dynamics in the model is different because of this different architecture, but the, it's judged based on the empirical results that this architecture is better. And the insights involved in coming up with this new architecture and modifying it, putting the manifold constraint in, these are all pretty mathematical things things that would be familiar to a physicist or to a mathematician.

And I chose that example because I think that's an example where I could imagine a model, if you asked a model to say, "Here's a description of your internal architecture. Let's study different ways to route the information through the neural network," maybe going beyond feed forward simple feed forward layer by layer.

Propose some ideas, test those ideas. If there are problems, come up with creative ways to fix the problems, and then if you can, come up with a better architecture, test it fully, and then if so, if everything's working, then do a training run, a big training run, and produce a better version of yourself.

And you know, I don't think models are necessarily good enough to do... Well, what I just described is kind of what the DeepSeek team did in this context. And I'm not saying the current models are quite good enough to carry this out, but it seems pretty close. Like, I would not be surprised if in a year or two you can have models implementing the types of innovations, for example, that are clearly described in the difference between DeepSeek V4 and DeepSeek V3.

Again, I emphasize DeepSeek only because they're very, very clear about what they're doing and we know the difference between V4 and V3. V4 is much better than V3. It's much more efficient. It's a smarter model, it's bigger, et cetera, et cetera. It's still quite efficient.

I think if you look carefully at those decisions made by those you know, rather brilliant AI researchers at DeepSeek it's possible to imagine that those could have been made by a model. It's not completely de novo, bolt from the blue, Einstein proposing special relativity kind ofinnovation.

It's more sort of, hey, these are all ideas that have been around in the neural network and machine learning literature for a while, or the linear algebra optimization literature. You know, these things have been around for a while. The model could be aware of A and B and C and D and combine them in a certain way, is seen later to make a lot of sense and be effective and thereby improve itself Right?

So combinatorial thinking of ideas that have already been proposed in the machine learning, AI, neural network literature, but have not all been tried because combinatorially, like if you have, you have 10 of these and 20 of these and 30 of these, all the different combinations, of course, is already beyond what all the different research labs can test.

But now that you have, artificial minds working on this, you could test them or at least have the model think about them and pick the best, most promising combinations, and then run the actual empirical test to see what actually works, et cetera. And, and this process, as I've described, could become more and more automated over time, and I think that would be the onset of RSI.

And what would that look like? Well, maybe the rate of improvement, you know, pick your benchmark. Of course, we have fewer and fewer unsaturated benchmarks, but pick your benchmark and, oh, maybe when the RSI gets turned on and if we have enough, you know, budget and data center power and chips, et cetera maybe it bends up the curve.

The rate of improvement of the models over time as a function of time starts accelerating, and that would be a signal of RSI. And increasingly the, the participation of human experts in this would be less and less and less, and to the point where the models are actually able to largely autonomously improve themselves, and then who knows where that would lead.

So, so I think that's, that's an important thing for us to think about, to keep track of, and it is ultimately tied back to, I think, the ability of the models to think about problems in math or applied math or physics.

Okay.

So let me move on to the next topic I want to talk about which is something I'm calling the agentic phase transition. Now, we already know that at fixed base model capability, by combining different instances of the model playing different roles, like in for, for math or physics research, you could have a model playing the role of a generator, where it generates ideas and possible solutions to the problem, and then you can have the model also playing the role prompted differently the role of a verifier, where it checks to see whether a particular solution that was suggested or a particular method or approach is actually good.

And it's been shown that the performance of a, an AI which is built from many different agents performing different tasks and with a large token budget, that performance can be an order of magnitude or more better than just the model, one instance of the model trying to do something. And so now we have this, this space of different possibilities, which are not just talking about the performance of a single model, but a performance the performance of, in a way, sort of teams of models, and the members of the team could be different.

So they, they could be trying to do different things. for example, one being a verifier and, one being a generator of ideas, et cetera, et cetera, right? For example, I used that approach in the physics paper that I wrote back in October twenty twenty-five.

But now this approach of using agents, many agents, many agents with very large token budgets, some kind of harness which controls what the agents are doing. Maybe a very, very intelligent model is supervising less intelligent models that are doing less challenging aspects of the task. So complicated agent hierarchies, agent control systems, these are all things which really everyone is working on now mostly in the domain of software development.

So the idea is like, "Oh, how do I develop some really complicated code base to solve some problem or implement some solution software solution? What is the right way to combine the agents so they're effective at doing this?" So that's already a familiar thing. There's a lot of effort going into this and because we're talking aboutcombining many different things.

So there, there's a famous quote from the condensed matter theorist Phil Anderson, who won his Nobel Prize for the theory of superconductivity. More is different. So, so he would say like, "Well, if you know the properties of one atom, okay, fine. But if you put Avogadro's number of those atoms together, you could get qualitatively surprising emergent behavior."

And similarly here, although we might understand we don't actually understand, even if we did understand like the intelligence and the capabilities of a single AI model, a single LLM, we don't know what a harness team of 1,000 of these agents working together over a long period of time, we don't really know what the capabilities of that is Noam Brown gave a talk at Manifest pointing out that the evals that we have really don't tell us what the combined capabilities of harnesses of agents or swarms of agents are because it takes a really long time to test like a, a big swarm of agents working on a set of evals and, and with a very large token budget 'cause they're gonna work on it for a long time.

And there seems to be just sort of monotonic increase in the performance on any eval by just giving more agents a crack at it, and giving them longer to work together, and giving them like feedback from some oracle or reward function or external expert. All of these things are things which people are working on, but we don't really know what the situation is.

We're, we're sort of just learning piecemeal, and different groups have different unequilibrated, unequilibrated knowledge about what they've discovered by, "Oh, if we harness the agents this way, and we let them run this way, we get this scaling relationship, or if we do it this way, we get something else."

This is the area, this is the actual frontier, both in practice, so at software companies and at startups and people trying to actually do stuff in the real world and in research labs. So the frontier is agentic stuff, but nobody knows the answer, and there could be a phase transition. There could be a thing where, okay, the individual ingredient is an LLM with some fixed amount of intelligence. When I combine enough of them and in the right way, I get some emergent behavior which I wasn't able to see before.

Like there, there are certain whole, probably whole classes of models or tasks that can be solved or accomplished which emerges out of just having enough agents working together and working together in the right way which we couldn't see before. And it, it would be similar to like you know, if you wanna build the atom bomb at the Manhattan Project, you need theorists, you need some physicists who are specialized in computing things about the properties of the shockwaves in the bomb or something, and then you need engineers and experimental physicists to actually put together the equipment and run the tests.

So you need like different elements with different skills and capabilities combined in a smart way, but then the output is way beyond what any one of those individual agents could accomplish. And, and we're kind of in that phase right now. We're in that phase where both at research labs and in practical settings, like in startups and companies, people are really pushing to see what they can get out of these agentic swarms of AI agents.

So stay tuned. I think that's really where a year from now we'll have a much better idea of what you can get out of that and, and what is the, what is the, what are best practices for combining these swarms of agents. Now related to this, topic is something called tokenomics. So when you start talking about situations where I just I don't just have, like, the I ask the model a question, and then it goes away and thinks for a while, and then a few minutes later gives me an answer. That's, that's what most average users are, are experiencing. People that are using the models in a more sophisticated way with big agent swarms and much larger token budgets you know, they might be expending millions of tokens to try to accomplish some project or solve some problem.

And when you're in that environment the economics of tokens starts to matter, and the term people use for that is tokenomics. And so the point being, if you're using millions of tokens, you actually care what, what they cost and what is the most efficient way to organize the inference that is represented by those tokens, right?

So tokenomics is becoming a more important thing. At the most basic practical level, it's cost. It's engineers, you know, realizing they can only spend so much of their company's money per month on AI inference and thinking about, well, what's the most effective way to purchase these tokens or organize these tokens?

Now, something that has emerged in the US-China competition is that some of the Chinese models, for example, GLM 5.2, which comes from a company called Z.ai. That model is not quite as good at software development and agentic tasks as the best US models, but it's pretty close, and it's very inexpensive to run.

And this is typical of the Chinese models. The Chinese models, because they were typically built in an environment where because of sanctions the Chinese labs typically have less access to compute than the American labs. It's not just due to chip sanctions, it's also just due to the fact that the AI bubble is bigger here, the AI boom is bigger here, and the number of investment dollars available to the US companies is greater than for the Chinese companies.

But for whatever reason, the Chinese are operating under more resource constraints, so their models tend to be more efficient. Efficient both in training and also in inference. And so you have a situation now where US software developers or software developers all around the world have a choice between routing certain queries to a less expensive Chinese model or to a more expensive but also somewhat more intelligent frontier US model.

And by combining by being judicious about which tasks you route to which models, you can optimize your, your token, your tokenomics, the, the amount of money you're spending, you know, to accomplish a certain task. And There was some data released by a company called OpenRouter or a platform called OpenRouter.

OpenRouter lets you route your query to a, a model of your choice, and they give you access both to Chinese models like GLM and also the top Western models. And what's interesting is in the last few months, if you look at the there's been an incredible growth in the number of tokens used on OpenRouter and the amount of money paid to OpenRouter for those tokens.

But most of the growth in the last few months on in the OpenRouter data is from the Chinese models. So you have a situation where the use, the in terms of just at least in terms of tokens, the, the number of tokens which are which are generated by US models for the platform is, is actually kind of flat.

And then you get this big growth because more and more tokens are being generated and delivered through this platform by the Chinese models. And so that's pretty interesting. If that, if that trend, if that's a real representation of what's going on, it's saying that there is substantial growth in use of AI inference, but a lot of that growth, maybe most of that growth, at least in the last few months, has been due to people using the Chinese models more and more.

Now, you have to be careful here because this is counting tokens, and the price per token is quite different. The, so the, the price per token might be five or 10 X more for the US models. And so actually the total amount of money flowing to the US models still could be way bigger than the amount of money flowing to the Chinese companies.

But still, if on the margin, most of the growth is going to the Chinese companies, that means a lot of people are really using their models for a lot of stuff. And so this is something really to keep an eye on both in terms of the US, China AI competition, 'cause if, if the Chinese models are getting, you know, a lot of traction this way, that's important for the dynamics of the competition.

But also if you are someone who's thinking about, for example, buying shares, buying a stake in the upcoming IPOs of Anthropic and OpenAI, you may wanna study the tokenomics very carefully because it could be the case that a lot of the anticipated growth in revenue from OpenAI or for OpenAI and Anthropic, which comes from more purchases of tokens, more use of their models for inference by, say, software developers all over the world maybe that won't materialize.

Maybe much of that growth will actually be captured by these less expensive but still very good Chinese models. And if that's the case, the implied valuation of Anthropic, the applied implied valuation of OpenAI may be quite a bit lower than the trillion dollars that their IPOs are reportedly gonna be priced at.

These companies are only a few years old, and it wasn't very long ago when a any company that was worth $100 billion was a huge company. Okay? Like most old economy companies like Ford or GM, you know, they're only in they're below that range, right? So if OpenAI, the quote fair value of OpenAI, given what is going to transpire in model competition with the Chinese companies, if the fair value of OpenAI is $300 billion instead of a trillion dollars, it's still one of the most amazing stories of value creation, of going from nothing to a $300 billion company within a few years.

That's amazing. It's only been done a few times in history, if ever. That would still be an amazing story, but if you buy the stock at an implied valuation of a trillion, which is what their IPO might come out at, you still might be buying you might be paying three times more for the discounted future cash flows of that entity than what will actually turn out to be the case, right?

So we're not saying that, oh, OpenAI is trash or Anthropic is trash 'cause the Chinese guys are getting traction for their models. We're saying maybe the valuation that Wall Street is asking you to pay for a stake in Anthropic when they finally have their IPO is incorrect because it's not taking into account this sort of competition on the margins competitions competition for revenue, pricing power, et cetera, et cetera.

The situation may be less positive than what the banks are assuming in pricing the IPO, right? So that's very important if you're an investor and you, you wanna think this through. It could be the case that we live in a world where AI is super successful. Everybody wants more AI. It's generating real value in the real economy.

But the Anthropic and OpenAI IPOs were heavily overpriced because people didn't factor in really, really tough competition coming from the Chinese which seems to be indicated by the Open Router data. Okay? All right. Let's see, how long have I been talking? It looks like I've been talking Wow, 45 minutes.

So let me finish up by talking about this documentary that we've been shooting. If you're watching the YouTube channel, some of the photos that have been flashing through the screen are photos at Lighthaven. Photos, I believe there's a photo of us interviewing Beff Jezos at Lighthaven.

There might be a photo of us interviewing Joscha Bach at his California Institute for Machine Consciousness. These are all interviews that we did for this documentary film, and we've been releasing many of these interviews as episodes on Manifold. But now we have a very large number of these interviews, and we're cutting them, we're editing them into what will be a feature-length documentary.

And originally, that documentary was going to be titled Dreamers and Doomers. So dreamers represents accelerationists, people who wanna build AI as fast as possible. They don't wanna slow down at all. They wanna accelerate all tech development. And doomers are the people who talk about their P doom all the time, i.e., the probability that AGI or ASI is gonna kill us all, right? And there's a very sharp divergence between the worldviews of these two groups. So you have people, you know in Silicon Valley, at the labs, in venture that are clearly dreamers, accelerationists. They just wanna build this stuff as pass fast as possible.

I think that's it's sort of definitely gonna be good for humanity to have access to this super intelligence. And then on the other side of the coin or the other side of the bay in places like Berkeley, you have people who are really worried about this, who are worried about existential risk. And originally we thought we would do these interviews in order to capture a moment in history, which might be the moment before AGI or the moment just before RSI or, you know, some important inflection point in the development of artificial intelligence.

It feels, certainly feels that way in the Bay Area. We wanted to capture that moment, and then we wanted to contrast these two worldviews, the worldviews of the dreamers and the worldview of the doomers. And this project was a labor of love. My two teammates, Lei Huang and John Greer, just said, "Hey, we should we're gonna be going to all these events, talking to all these people.

We might as well try to capture these interviews and share them with the rest of the world." And, and since we're, we're all cinephiles, we thought, "Well, let's cut this into a nice a beautiful documentary and release that." So that's the project that we've been working on. Lately, my brother has been actually helping us do a lot of the editing, and so he's actually created, like, the first rough cut of the film, which is about 100 minutes long.

We've tried to raise we spent a little bit of effort to raise money to help support this project, but we haven't really raised much, and basically, we've just been self-funding it with our own sweat equity to do it. As I said, it was just, it's just a labor of love. I'm a big fan of Joan Didion.

Joan Didion wrote an article called "Slouching Toward Bethlehem," which was about the Haight-Ashbury and the Summer of Love in San Francisco in the '60s, and she sort of captured this strange moment which at the time seemed very strange, but then, like, what happened in San Francisco that summer, in a way, transformed the rest of the country and the rest of the world in the subsequent decades.

And in the same way, there's something really magical brewing in the Bay Area, San Francisco Bay Area right now associated with AI, and so our goal was to capture that on film, capture that in interviews and also just footage of crazy events happening ranging from, you know, parties, manifest zaniness people protesting outside of OpenAI corporate headquarters, pause AI, stop AI.

This whole mix of insane stuff all focused on AI in the Bay Area. That's what we try to capture in the film. We have since changed the title of the film to Machine God. Partly because a lot of the people in Berkeley didn't like the term doomers. They claim that doomers is pejorative.

It's, it's, it's actually insulting to them to call them doomers, even though, like, I hear the word doom coming out of their mouth all the time. So to me, like, if someone uses the word doom all the time, it seems to be fair game to call them doomers. But I think from their perspective they want to say, "Calling me a doomer suggests that I'm not rational, that the reason I'm worried about AI is just I'm some kind of depressive person with a weird personality.

It's not the, it's not the consequence of my, like, dispassionate, rational thinking that led me to the, the conclusion there's a problem, that you're, you're sort of suggesting that it has something to do with my mental state or something." As, as near as I can figure, that's why they really object to the term doomer.

Anyway, the film is called Machine God. We premiered the trailer for Machine God at Less Online and Manifest. People really liked it. We'll put a link to the trailer in the show notes. Maybe we'll even merge the trailer into the video for this podcast. Let's see if my editors can do that.

In any case, please support us. So if you want to donate to the project you can do that. We'll put some links in the show notes. If you want to help us advertise, the film, have a showing at your university campus maybe help us make some short video excerpts of the movie that we can put on TikTok to raise awareness of it, I think those are all things that we would, you know, appreciate help in doing. One of the main motivations for me, aside from this historical, cultural goal of documenting what's actually happening in the Bay Area at this moment in time, the other thing I would like to do is, is make some of the core ideas that we emphasize in the movie, like recursive self-improvement, like existential risk, gradual disempowerment of people as we outsource more and more of the most important decision-making in society and in the economy to AIs.

These are all things which I think elites, the people who really have power in our society, are not really they do not really understand these concepts very well right now. If they are worried about AI, it's something like, "Oh, these data centers are gonna damage the environment," or, "Oh, my son might not get a job because of AI competition."

They're sort of at a certain level of awareness of possible negative aspects of AI development. But these really sort of, longer term risks or more subtle, complex issues, I think, really our elites are or and the general population are really not aware of.

One of the main purposes of the, of the, of the documentary is to make these things concrete. To have them hear obviously thoughtful and intelligent people, all the people we interviewed are like that, talking about these topics and seeing that these people are serious and seriously concerned about these topics.

And we hope to just raise the level of awareness in society about these things. Whether you are a dreamer, an accelerationist, or a doomer, someone who's really worried about AI safety, you probably will agree, society could benefit from more attention to these things, which may become important sooner rather than later, depending on the pace of acceleration of AI development.

So let me stop there. I think I've gone on a little over 50 minutes. I hope you enjoyed this podcast. The next episode will probably be back to our normal format where I'm having a conversation with a guest. In any case, thanks for listening.