Steve Hsu: One reason I think that the subset of math that humans like will turn out to be a small subset of what the machines can do is, for example, the fact that our working memory can only encompass about ten objects. Whereas for them, the working memory or something analogous to working memory could be could encompass millions of concepts or billions of concepts.

And so the type of math that the machines can do could have a very, very different style compared to what we're used to and compared to what we can digest. So my prediction for how the future will go is that math and theoretical physics will advance but dominated by AI results, and humans will trail behind and appreciate a subset of what the machines are capable of doing.

Welcome to Manifold. This is another special episode. It's just me. I am on the road. I am in Beijing, China. I'm visiting the Yau Mathematical Sciences Center at Tsinghua University.

S.T. Yau was a professor at Harvard for a long time, Fields medalist. If you're a physicist, you might be aware of something called Calabi-Yau manifolds, which play a role in the compactification of dimensions in string theory.

Yau retired from Harvard and now runs a sprawling network of math, mathematical sciences research institutes at Tsinghua, but also at, in other places in China and other universities. I'm here collaborating with researchers in math, theoretical physics, and AI. We're interested in using the models to do math and theoretical physics.

We're also interested in understanding how to improve the model's capability in math and theoretical physics, and actually ultimately in AI research, which leads to something we refer to as RSI. I'm walking around outside, so you'll hear some funny noises. This place is full of electric vehicles.

Actually, China is a very safe place but probably the most likely way that you would get injured in China walking around is because there are so many electric vehicles, delivery people people just commuting around on electric bikes, electric scooters, and they're pretty silent, so they just sneak up on you, and if you make the wrong move, you could get hit by an electric scooter.

I'm in an apartment complex, which is right across the street from the Yau Center andbeautiful apartment complex with a park and most of the people, I would say probably almost all the people living here are affiliated with the university in some way. I'm here mainly because my wife is on sabbatical.

My wife is in the humanities, and she's on, in, on sabbatical as a visiting professor at Tsinghua, so I'm here visiting her and also collaborating with my research collaborators.

So let me first start by just talking about the shock that the mathematics community is in over these new results released by OpenAI.

You've probably read a lot about it. If you are unaware a number of very important unsolved problems were solved, theorems proved by a version of OpenAI's best frontier model, or at least best in terms of mathematics capability. I've been saying for some time that Astra and actually Claude are superhuman when it comes to math and theoretical physics.

So I, I feel like they're capable of close to flawless computations, long detailed computations, understanding of concepts involved in advanced math and theoretical physics. They have a mastery of the literature that no human has. Stends across many, many subfields. They're able to combine knowledge of those different subfields very productively.

Many of the results that were obtained, when the experts look at those results, they realize the model has used some technique which is quite distant. Distant in the sense that human experts in one area are unlikely to have a mastery of these concepts in a very distant area, and yet the model is able to combine them fruitfully to obtain results in an area where humans have been working for decades to advance our understanding.

For example, in the Riemann hypothesis and yet the model made an advance which, you know, I believe many people feel would I think one mathematician who works in this area said this would be an instant Fields Medal if a human had done this. So we're now in the situation where both for code, like writing software and for doing mathematics and theoretical physics, I think a, a fair assessment is the models are, are clearly superhuman.

By superhuman I mean they're capable of doing things that humans, even the best humans find extremely challenging. It doesn't mean they're superhuman in every way. They might still make mistakes that human coders or mathematicians are unlikely to make. But overall, I would say it's fair to say they're superhuman.

Now, this has caused a huge amount of trauma in the math community, and I mean, Terence Tao, who's a, you know, arguably one of the most brilliant living mathematicians still kind of in his prime. Wow, that was a rare non-electric scooter that just went by. That's probably why you heard that noise. But I'm currently walking down a street. There are like hundreds of electric bicycles, electric scooters parked here.

It's very rare to see anything that's not electric. There are also, by the way, robots, delivery robots, and things like this roaming around too. Back to the reaction of mathematicians to these OpenAI results.

I think the strongest negative case, which people like Terence Tao have advocated, is that this kind of AI advance discourages humans from working in mathematics, discourages training of students.

And if humans stop participating in math or participate only in a very reduced way, then the core training data that is necessary to get it was necessary to get the models to the point that they're at now that training data will no longer be accessible, and the whole enterprise of mathematics, you know, human plus AI, could eventually grind to a halt.

So I think that's the strongest negative hypothesis about the future of math. I find that quite implausible. I actually feel that even if today we started we stopped training graduate students there would still be a fair amount of activity among existing mathematicians trying to understand what the AIs have done And there's still enough training material, training data to continue to make the model stronger.

I don't think we're actually at a limit. I think the, there's still quite a lot of room for the models to continue improving using existing data. And so I think the future actually, even in this worst case scenario, involves super intelligent AIs advancing the frontier and continuing to advance the frontier.

I just, I don't think it's plausible that even if humans get discouraged to do pure math, AI researchers trying to make AIs better at pure math will still succeed. And so we'll just see continued progress, and eventually the frontier will be almost entirely explored by AIs, and humans will lag behind.

I think eventually humans will get be interested in math the same way humans are still interested in chess even though the best chess engines can destroy Magnus Carlsen and the top human players. There still will be human interest, but humans will lag behind the horizon, and there will be some subset of results that humans can digest or find interesting that are originally produced by AIs, but then sort of simplified human digestible, human Appealing theorems will constitute the body of knowledge that humans have access to.

One reason I think that the subset of math that humans like will turn out to be a small subset of what the machines can do is, for example, the fact that our working memory can only encompass about ten objects. Whereas for them, the working memory or something analogous to working memory could be could encompass millions of concepts or billions of concepts.

And so the type of math that the machines can do could have a very, very different style compared to what we're used to and compared to what we can digest. So my prediction for how the future will go is that math and theoretical physics will advance but dominated by AI results, and humans will trail behind and appreciate a subset of what the machines are capable of doing.

So there are sort of two reactions. I think Terry Tao's reaction and that of many mathematicians and, you know, many of whom signed a letter that I think was originated by a group of, I think probably something like over ninety percent of all the living Fields Medalists signed it. And, and that was a very anti-AI letter.

On the other hand, here in China, S.T. Yau so you know, we have this sort of Tao versus Yau thing going now. Yau proclaimed that he and his institutes would create an alliance of a thousand mathematicians here in China, but of course, welcoming people from all over. This alliance would embrace AI and would use I think open source models.

Would try to bring open source models to the frontier to the level where Astra and Claude are, and then use those models to produce open proofs that the human community could appreciate and, and use. And so a very different vision. So the Chinese version very optimistic, pragmatic looking forward to the use of AI to advance the subject.

But I think the majority opinion, at least outside of China, very negative and fearful about the impact that AI is gonna have on math. The people I collaborate with here are actually part, in a way, of this Yao alliance, and the work we're doing, I think, may get sucked into it in some way or another.

One of the things we've been working on is an automated method to produce challenging data that can be used both in pre-training and post-training of AI models, but doing it in an automated way. We think we've come up with an you know, clever methods to do this, The team has already produced new benchmarks and evals, which are not saturated for, even for Astra and Claude.

So Astra and Claude on these tasks still, have, say, roughly fifty percent success rate. The open source model is much lower, maybe twenty percent success rate. And problems like this Can be produced at scale using our methodology. We've been having discussions with several of the Chinese AI labs about using this kind of data that we can produce to close the gap between their capabilities and, and for example, Astra, which Astra I think is by far the strongest math model that exists now. If not Astra, then a private better version of Astra that OpenAI internal people have access to.

So let me turn from that to some commentary about the AI situation in China. 'Cause as, as I said, I've been having meetings with people from many of the leading labs here in China including, for example, DeepSeek.

People here are much more pragmatic than in the US. They're much less, shall we say, AGI-pilled. I regularly encounter AI researchers in China who, who don't not only do they not know the terminology P (doom) they are really actually unfamiliar with the concept. They actually find it very strange that in the US researchers and AI safety advocates extrapolate into the future and think that AI is gonna destroy humanity or human civilization.

They, they find it actually very strange. I've actually shared access to our documentary film, Machine God, which is about AI existential risk but mainly centered on interviews of people who are in, in the US San Francisco Bay Area AI complex. The worldviews are just extremely different.

I think definitely worth exploring. I think people here in China are gradually becoming more aware of safety issues in general, but they're focused on more near-term things like cybersecurity or job loss or even ability of the models to help people create bioweapons. They're not focused further on further into the future on risks such as gradual disempowerment or complete AI takeover of technology on the planet.

So these are things that I think it's very tough to get traction here discussing such things with the AI researchers.Regarding the US-China AI competition, I think the number one constraint on progress of the Chinese labs is access to compute. Well, I should say compute and, and other resources, like just financial resources.

So, so the companies here are not as well capitalized as OpenAI and Anthropic.There's no AI company here that has a valuation on the order of a trillion dollars, and they can't raise hundreds of billions of dollars at a time DeepSeek can maybe raise ten billion dollars, I think which is the reported ballpark for its most recent fundraise.

It's still about sort of one ord-order of magnitude smaller than the sizes of rounds that OpenAI and Anthropic can do. So because they have more limited resources their ambitions are somewhat less, at least in the near term. And so the, the sizes of the models they can train the number of different specific domains of capability of the model, so for example, math or biology or legal work, et cetera, they're more constrained in what they can, for example, pay external data providers to produce training environments for, to produce data sets for.

And so they have to pick their areas of concentration and try to catch up. So for example, several of the labs here have caught up now or almost caught up in coding for cybersecurity capabilities of their models. But overall, their models still lag behind US frontier models. Now, that could change.

One of the big advances here is that with leadership from, for example, DeepSeek there, there now exists a lot of good infrastructure for migrating from the NVIDIA platform to other, chip platforms. For example, the Huawei Ascend series. So running the models to service customers, the post-training of models, and at least part of the pre-training of models here has been done on the Huawei hardware.

And people I spoke to are very confident that within the next, roughly, I think a year, even the pre-training step of models, which is the most computational complex step, because, during pre-training, there's a lot of data that moves around between the individual GPUs. That's something that NVIDIA hardware excels at and the other competitors are catching up on.

But because of infrastructure that has been built here, software infrastructure to manage things like pre-training on Huawei hardware, people are confident that within roughly a year even the pre-training step will be doable on non-Nvidia hardware, on domestic Chinese hardware. So I think that it's likely that China will have a completely domestic supply chain for AI training and inference.

And then the main bottleneck after that will be just the availability, the, the, the ability of SMIC to fab enough Ascend and other domestic GPUs and AI accelerator chips. And so that's a separate issue of, you know, how fast they'll get to the level where they can produce millions of individual chips per year for this purpose.

And, and there are various forecasts for this. I think probably within next couple years they'll reach a point where they are able to supply, largely supply the domestic demand for compute. I don't think it'll completely catch up. I think it'll still be significantly behind the ability of TSMC and Samsung and, and other Western or US-aligned fabs to produce AI chips.

But I think it'll be enough that it'll allow the Chinese labs to be much more ambitious and to pursue, for example, and I did discuss with people here, models that are in the sort of multi-trillion parameters up to ten trillion parameter range. So I think that the qualitative gap between the models that are produced here and the ones that are produced in the US will shrink over time.

Actually, I think the, the problem that can't be solved as easily is the financial problem, which is that because there is a such a huge AI bubble going in the US and because OpenAI and Anthropic and, and even secondary startups that are, you know, startups that are doing things like creating data sets or training environments for models.

You know, all of them are valued at roughly, by the financial markets, at roughly ten X what the corresponding companies are valued at here. That ability of the Western markets to get really excited over some hype cycle and then allocate, you know, what amounts to, you know, percents or multiple percents of GDP toward investment in that particular hype area.

That doesn't happen so much here, and probably that's the main reason the companies here are still behind. It's not a lack of talent. It is still, for the moment, a lack of compute, but just as important is the lack of financial resources. Just don't have the same level of financial resourcesfrom the equity markets that the US, their US competitors have.

Turning to financial markets, I attended a meeting in Hangzhou, which was very, very high-level capital allocators internal Chinese hedge funds that invest primarily in China, foreign hedge funds that have at least some interest in investing in China, and also big entities like sovereign wealth funds, pension funds, et cetera, that allocate monies to hedge funds and venture funds.

Also that at least are either curious about investing in China or, or do actually allocate some part of their portfolio to China. So it's quite a diverse group. The organizers let mepresent a clip from Machine God, and we had a lengthy Q&A about issues like AI safety, AI risk US-China competition.

I think a lot of participants found this discussion very eye-opening because again, concepts like gradual disempowerment and RSI which I think are, are pretty commonly discussed in Silicon Valley are pretty foreign to, I would say, most sort of normie people in finance or investors. This was quite an international group.

There were people from all over the place, Latin America, the Middle East, Europe, the US, but with a, a China focus because it was, it was the meeting was held in Hangzhou. And, so very, very interesting. Lots of discussion about the Middle East, lots of discussion about interest rates, petrodollar how the Middle East will be or-reorganized after the dust settles from this US-Iran conflict.

Also lots of discussion about whether there's an AI CapEx bubble. Very, very interesting experience. Also Hangzhou, as you know, is the headquarters of both DeepSeek and Alibaba. Alibaba makes the Qwen models. So there were, there were people, top people from those companies also around.

So Hangzhou is for, for people who've never been there, it's, it's one of the most sort of, advanced cities in China. It's very affluent. The city's very clean, beautiful. it has a sort of historic lake is, is a place where people like to go on vacation. And the compound that where this event was held, which was very fancy, there's a particular area of this compound where Xi Jinping actually, when he goes on vacation, sometimes stays.

When I was jogging around the compound, someone pointed to me and said, "Hey, that's this is where Xi Jinping usually stays when he's here." So anyway, quite an interesting experience. Highly recommend if you know, when you go to China, you know, people usually visit Beijing and Shanghai but Hangzhou is definitely worth a stop, and it is increasingly actually a, a, a sort of tech hub for China.

Now, interestingly, where I am in Beijing right next to Tsinghua University, a place called Haidan, I would say this is probably the number one center for AI activity, sort of similar to San Francisco area in, in the US. And even though, for example, The Qwen team is part of Alibaba, and that's headquartered in Hangzhou, and DeepSeek's headquarters is officially in Hangzhou.

They have many, many researchers. I was told about half of DeepSeek's team and, and the more research-oriented side of the team is actually based here. And all of the almost all, I think, of the major AI labs in China have pretty large offices right here, just within a kilometer of where I am right now.

So the, the biggest, single biggest concentration, I think, of AI, certainly AI research, is within a few-kilometer radius of where I am right now, and it certainly feels that way. It's, it's sort of like being in San Francisco. When my research collaborators and I have a meeting with one of the major labs, we literally just have to walk a few blocks from where the AI college, the Tsinghua AI.

Tsinghua has not just a CS department, but it has actually a separate college within the university which is focused on AI. And their building which is just outside the south gate, it's just outside the main campus. It's in this area where lots of high-tech offices and companies are located.

We only have to walk a few blocks from those offices to the offices of, for example, Z.ai or, or where the DeepSeek researchers sit. So very, very concentrated. Even more concentrated than San Francisco actually, in terms of how far apart the different companies are and where their offices are.

And of course, people are hopping back and forth between these companies because you might work at company A for a while and then, you know, be hired away or poached by company B. And you might just be moving your office, you know, a few blocks away. One of the things that's different about the AI ecosystem in China, and this has actually been a big advantage for them, is that because most of the companies here, you know, with a few exceptions like ByteDance, they, they generally, they release their models open source, and they write beautiful, very clear papers explaining all the research they've done and what innovations went into the latest model release.

That is very, very different from the US. So on, on the US side the labs are increasingly tight-lipped about what they're doing, what the parameters of their latest model are, what architectures they're using, what innovations, algorithmic innovations they've developed.

All of that is kind of a mystery, and people who are outside of the labs are, are left guessing about what they're up to. Whereas here, almost everything is disclosed. And because things are disclosed in the process of the research leading up to the next model release, they're much less secretive about it because, you know, ultimately the whole thing's gonna be released open weights.

And so there's a lot more collaborative discussion between researchers at the universities, researchers at the different companies. And so it, it creates a more kind of collective large N kind of environment where there's more sharing of research ideas, more openness, and a transmission of skills to the students.

So, for example, the students at Tsinghua University, who are really among the smartest students at any university in the world, those students are sort of bathing, if they're interested in AI, they're sort of bathing in this China AI ecosystem. They can do internships at these companies that are here in Beijing.

They can do research projects with professors or with researchers at the labs. And there's just this incredible diffusion where you, you just meet these, you meet lots of undergrads and grad students who are they know everything about how to do SFT, how to do post-training RL, how, how pre-training works.

Just a incredible diffusion of understanding of both the theoretical concepts, but just also, like, even how to use, like, the particular software packages and infrastructure, how to run things on NVIDIA GPUs, how to run things on Ascend GPUs, et cetera. So there's something to be said for this kind of collective open way of advancing technology and sort, sort of what most academics would prefer because we're used to operating in an open environment where innovations are shared.

And in the Chinese ecosystem, you know, if DeepSeek makes some particularly clever architectural innovation on their models the other companies will, if, if they want to, quickly follow and, and, and maybe use that architecture for their next generation of models. So you know, it, it feeds kind of fast innovation, much more so than a closed secretive ecosystem. Let's see, what else did I wanna cover?

One of the things that I think people outside of China don't understand is you know, because they, they tend to think of the political system here as very top-down, and they tend to think of this place as, you know, at least formal in, in terms of the economy, a formerly communist system that still, you know, is not as driven by market forces as the Western system.

So that is really a misconception because what, what happens here is that there are so many companies in any particular area. There are many, many companies that emerge to compete, and they end up producing lots of innovation, driving prices down, and also driving profit margins down. So one of the reasons why, you know, you might say I don't that you don't want to invest in the Chinese equity markets is because the companies here generally don't produce the same level of profits as the corresponding companies in the US, and it's, it's because there's just so much competition. In part that competition arises because the political system here is very decentralized.

At the Hangzhou meeting, I met the former mayor of Hangzhou and, you know, his job when he was mayor of Hangzhou, which is a very high political position you know, his job would have been to produce lots of world-beating companies in Hangzhou, patents, you know, innovations that he could point to as originating in Hangzhou.

And so the, the local governments act like venture capitalists to invest in their local companies. And so you end up with, you know, a hundred companies making EVs backed by individual local governments. And in a way it's quite wasteful because only a subset of those companies are ultimately gonna reach scale and, and dominate the industry, and there'll be some, you know, messy consolidation.

But on the other hand lots of ideas are explored this way and you know, in a way you could say the system here is very decentralized, maybe more decentralized than the US system or as decentralized as the US system as or more competitive as the US system. And I just think most people who aren't familiar with how things work here are, are just misguided in their sort of mental model of how the, how innovation works here, howcompanies are formed and how they compete.

And in fact, the main problem here is a kind of deflationary situation where there's just so much competition between companies that they're not able to create healthy profit margins. There's a kind of race to the bottom in terms of pricing. And all of this is in a way deflationary and leads to what, you know, people who are critical of the system here call overproduction.

So you just have too many companies trying to produce the same things and, and competing, the prices down and, and once, you know, once a company wins here really the, the foreign competition is less formidable often than the competition within China. So the companies that win here and then emerge to export their products abroad are just super competitive, and they can undercut foreign competition at sort of fixed at fixed quality of product.

They can undercut on price really substantially. And that, that's sort of the problem here that they need to fix. In, in a way, the central government knows this. Central government has actually enacted lots of reforms to try to reduce. Here, the term is juan. It means, like, really grinding tough competition, race to the bottom.

That kind of behavior the central government has figured out, like, oh, actually there's, there's too much local investment in producing local companies and et cetera, and it's, it ultimately counterproductive, and the, the, the central government's actually trying to crack down on it So as always, it's super interesting to be here.

If you've never traveled here it's quite easy to do so. Many countries it's not a visa is not required to visit here for a couple of weeks. And even if you're an American, technically you do need a visa, but you don't need a visa if you're transiting. So in other words, if, if you're visiting China, but then you're going off, going back, going to some other country after that, not directly returning home.

So if you wanna travel here, one of the easiest things to do is say, "Oh, I'm, I'm gonna travel to China and then visit Tokyo." And then technically you're transiting. China, China is a transit point on your way to Tokyo. You're not going directly back, say, to San Francisco. Then you don't need a visa, and you can just come here, I think, for ten or fourteen days.

And I would highly recommend that because I think most Americans are have a very distorted view of what things are like here. And definitely if you get a chance to travel here and and check it out. All right. So sorry for the short episode, but, I'm quite busy here and just thought I'd try to share some of my thoughts about this trip so far.

Might do it again. There are lots of interesting people here that I can interview. So look for some fun future episodes where I interview some guests that also know something about China. Thanks a lot. Talk to you later. Bye.