Steve Hsu: Welcome to Manifold. We have a special treat today. I have with me in this room, three engineers, three AI engineers from the company Super Focus, which you've heard about on previous podcasts. Their names are Ryan, Jordan, and Rahul. And what I hope to do today is have a conversation between the four of us, all of whom, all of us are super interested in AI, both practical applications and eventually AGI.

What's unique about these young guys is that they are in the trenches actually building products with large language models. So, welcome to the podcast guys. Let's start by talking about the Google Memo, which came out, I want to say maybe like a week ago or two weeks ago.

And, interestingly, we just actually interviewed someone who applied to Super Focus, who's currently at Google Research and also had interesting opinions about the memo. The memo roughly says that companies like Google and ironically Open, I'm using Open and Scare quotes, Open AI, because they're keeping their IP and their models closed source, that they're potentially gonna be out innovated by hordes of people who are out there building, even building foundation models, big models.

Which are open source. And by open source, I mean you can download the full model. So you can download the weights or the matrix elements in these models. And, to some extent, these guys got their start because Meta, released some of their very powerful models. So there was a core amount of big compute that was used, really to get this ball rolling.

But it seems like every week we are seeing new models. We're seeing innovation, like people specifying their models. We'll discuss that more in a little bit to make them run on lower end hardware. We're seeing people who have direct access to the weights of the models, doing direct tuning of the models to make them better at certain narrow tasks.

So we're just seeing an incredible amount of innovation and the historical analog that I just want to point to, which I think also Jeffrey Hinton, or ne Lacoon talked about. I think it was Jan. If you look at the history, if you're an old guy like me and you remember the early internet days, the server side software that's running like every data center now, every major website, it's basically Apache and Linux to open source technologies.

But I remember the day when there was a very big dog fight between a company called Sun Microsystems, which ran an operating system called Solaris, which was their flavor of Unix. And they had, at the time, their server side software was supposed to be the fastest. Microsoft also had their own server side software and closed source operating system for machines that we're running it.

but the ultimate winner, which really powers the whole internet today, is Linux and Apache. And so are we gonna see a repeat in history here where AI eventually is dominated by open source models simply because there's just more. capability of bright people who aren't, just, don't happen to be employees of Google or OpenAI, or the few closed source shops to just out-innovate these companies.

So I just want to start, on that topic and, any of you guys, want to jump in, just raise your hand. I'll, I'll point at you and then you guys, you can just, just jump in and talk about it. Anybody want to address this question, this topic?

Rahul: Yeah, so I think open source is definitely catching up at breakneck speeds and for, for now, perhaps like Open AI is like the king temporarily. Even that's disputable, if you like, focusing on a particular subdomain. but I don't think that's gonna last for very long because, and they said this in the memo, but if you look at like, where we were like two weeks ago, and then look at where we were one week ago and look at where we are now.

The gap that is closing so rapidly is nearly exponential. and so I think definitely open source models are gonna catch up. And if Open AI and Google don't address this in some way, shape, or form, then these open models will overtake them eventually. and I think a lot of that is just because there's so much, there's a lot of trial and error that goes into building good AI models.

And you can look at like the open source community and for all these different models, there's like so many different variants for each model. and that kind of trial and error is just like infinitely easier for the open source community to do and put together because there's so many bright people out there than it is for like a company like Google or Open AI with like their limited pool, even though that pool has like very bright individuals.

Steve Hsu: So Rahul, let me drill down on what you're saying here. So, If I talk about the best general purpose model mm-hmm. At any instant in time. Yeah. As opposed to, I took a general purpose model, which wasn't really the best, but I fine tuned it or I did various things. I did even like low rank, adaptation, various things.

I got it to be far superior to say Open AI on a narrow task. Mm-hmm. Okay. Are you saying that even for the best general purpose model at some moment, say six months from now or a year from now, it could even be, it could at that point be an open source model? Is that possible?

Rahul: Yeah, I think that's very much possible.

And I think we kind of saw the trend towards that with all the recent Facebook waits that have been, been leaked because like so many people have been working with those. and I think that model, it maybe is not that model that's gonna be like the winner. but we've just been seeing so many people innovating and improving upon that model, that it seems to suggest that there's like this trend of like, open source models.

We'll get there eventually.

Steve Hsu: So, let's take that case because, so you had a, a, a core amount of like probably pretty significant compute spend on the part of Facebook, but then the model got not first leaked, but then like I think Facebook is just, or meta is just okay with people that officially open sourced it, if I'm not mistaken.

So now are you saying that all the open source people working on it, and this is the, the general model scenario, Are they in effect, like spending the compute money, but in a distributed way that it's gonna push that model beyond, like the best that Bard can do or the best at GPT four or five can do Is, is that the scenario that you're envisioning?

Like does the money have to be spent and is it being spent but it's being spent collectively? Or is it that the money doesn't really need to be spent, it's just cleverness on the part of the open source community? Do you have a feeling for which of those two is more accurate?

Rahul: Yeah. I mean, it's, it's hard to say because it really, you definitely used to be the case that like there was definitely like money involved and, money and time both evolved drastically. I think that's also changing pretty fast and we'll probably touch on this when you like, reach like quantization.

But we're reaching a point where, any, any random individual that wants to like, play with these models needs a lot less computation and needs a lot less time. And so that only helps the open source community edge up, up on this where they can test out all, like whatever they, any individual can test out whatever they want.

And like record speeds, with low cost. And I mean naturally, like all these big companies, will be spending a lot of money because they want to see what they can do. But it's not entirely clear whether that's still gonna be necessary like six months down the road.

Steve Hsu: You know, just to give the listeners a sense of the whiplash that I feel it, you know, six months ago when chat GPT was still kind of new, I was thinking to myself, being an academic professor, I was like, man, I feel sorry for these academic CS groups because what can they do?

Like, they can't do anything. Like what people who are actually at OpenAI or Microsoft research or Google Brain can do or Deep Mind. But now we're kind of feeling the opposite, that if you, if now we're a small company, but if we were an academic group and the PI had like maybe a couple million dollars in grant money, it seems like now the feeling is they could do something interesting, they could make some noise.

Is that fair? That's right. So, for me personally, my worldview on that issue has almost flipped, like 180. But you know, you could say Meta gets a lot of credit for this by basically doing like that first, who knows 10 million chunks of work and pushing it out, open source. But now that that's the case, the situation is quite different.

Jordan, did you want to comment on any of this?

Jordan: Yeah, well I think the money thing is an interesting,kind of thing too. There was an organization called Mosaic that essentially released a 7 billion perimeter model. And the whole purpose of that experiment was to see do you need to spend millions of dollars to get these like super capable models?

And what they found out is that you really don't, I think they were able to get that model done for like 200 k in like a couple weeks as opposed to like millions of dollars in, a couple, a few months. and so I think the question now is like, how, how cheap can you even get the foundation models Yep.

That can compete with like the lamas or like the barns or stuff like that. so I think that'll be an interesting direction as well as to see how far we can drop the cost of even really capable foundation models. so it's kind of like a mix of both, the money and the cleverness.

Steve Hsu: Got it. Ryan, did you want to add anything to that?

Ryan: Yeah, there's also a distribution question here, which is that Open Eye of course owns their models and they distribute them, you know, in the open source community right now. Like we have these great models and you know, how great they are is of course up for debate.

We think they're really great, of course, but, you know, if you want to use GPT four, it's a PIP install and then, you know, one line of code and it works, you know, 99% of the time. Whereas for this, you know, you obviously have to dedicate some engineering effort to that. this might change, and hopefully it will change.

And when that does, you know, you might expect another kind of 10 x leap and interest in these open source models. and that'll be all the worst for, you know, the Googles and the open eyes of the world.

Steve Hsu: Great. So let me jump to, well actually before we, before we leave this topic, I just wanna ask you guys one question.

So it's very tough to kind of know whether Model A is better than model B, and in fact, like what do you mean by that? From the academic side, when somebody releases a model, there's usually some set, sort of set of benchmarks that they might run the model against and just get numbers back. When you compute numbers, people report the numbers and people focus on the numbers.

But sometimes numbers don't really capture reality that well. There's a company called Hugging Face for people who are not in our universe. Hugging Face has a ranking, which is very dynamic. Well, we're gonna get to this in a moment, but Hugging Face has a set of metrics by which they rank models.

How confident are you guys that if we say, took a poll within the company or took a poll in the industry, that people really have a pretty robust and realistic idea of which models are better than others? Or is it just kind of like the Wild West and nobody really knows for sure? Like, could there be big surprises, like the hugging face rankings are just like, actually not right, like for your application or even for a general application.

Rahul?

Rahul: Yeah, I think for, if we're just talking about the general use case, first of all, then I think the rankings are pretty good and it's like an order of magnitude thing, right? Where it's like, the top model may not be the top model for everything, but if you take like the top like 10 or 20 models, then you'll generally, it'll be some sort of re-ranking of those like top 10, like 20 models that'll be doing the best for generally use cases.

Now, where it gets tricky is that when you're doing specific sub domains, there also comes into play like a size question. And so the thing is, for a general use case, like bigger is generally better. but that's not really true when you wanna fine tune for a specific sub-domain. because then you have a question of how much data do I add?

And, how much time am I willing to spend? and how deep of an understanding does my model need to be to be able to perform well on this subdomain task? And so when that comes into play, you might find that actually these like smaller models that don't really score the best on like these rankings for overall like good and goodness of performance may suddenly you may be able to fine tune them to be the best in your subdomain.

Steve Hsu: Right. So if I could summarize that, it could be that the smaller model isn't good across the board, but your ability to make it good. Yeah. At a fixed cost or effort like just reading legal contracts or reading financial documents, that's where maybe those smaller models shine. Yep.

Jordan: I think, Yeah, that specialization aspect is going to play a huge part kind of, as we kind of get into the conversation of like, AGI. so for example, there was a model called the Guerrilla Model, that was specifically trained on a data set of, interacting with APIs.

And what they found was that, by only fine-tuning it on that massive dataset, in the well curated dataset, it actually performed better at, kind of reasoning about APIs than GBT four. Which is a huge, huge thing. And so you can almost imAGIne, if you were to modularize different aspects of intelligence, you can get models specifically specialized in, in these areas.

Steve Hsu: Let me move on to my next topic, which is related to the size of models. So one of the things that has caused a lot of excitement within the field and people outside might not be aware of this, is something called quantization. I might call it from my academic perspective sparsification of the model.

And so quite often in machine learning, the model itself is a list of weights or a bunch of matrices where each entry in the matrix is some number. And naively, you might think, Hey, I need to know those numbers to really high precision. I need like 10 digits of accuracy or 16 digits of accuracy. But it turns out often the learning process or the training process of the model is very noisy.

So sometimes, connections get turned on that the model is actually not really using very much. So like the coefficient. If the coefficient or the weight is really small, you might be better off just setting the weight to zero. And that's called sparsification. And it also might be that the last five digits or the last 10 digits out of 16 are totally unnecessary and don't really add any value to the performance of the model.

So people have been experimenting with, you know, what's called quantization, four bit quantization or eight bit colonization, in which they only allow a much smaller number of bits to store each weight. and therefore, the size of the model in memory is much smaller. The speed, the amount of compute necessary to use the model for inference goes way down.

And so now you have people who are at home and they don't even have to have a G P U. So, so, so at first people were like, oh, you must buy a good GPU and then have that at home or in your basement, and then you can play with these models. But now we're getting to the point where you don't even necessarily have to have a G P U and you can use a model whose performance is actually not bad, but it's been through this sparsification or quantization process. And from my experience in genomics where my other startup builds very sparse models where we we figure out exactly what parts of your genome are like affecting your height or affecting your heart disease risk, that sparsification is very powerful, but you can't do it in a dumb way. You actually have to use some cleverness to do the sparsification. So I would guess that, if people really wanted to focus on improving quantization of models for LLMs, there is a lot of still low-hanging fruit to be had. In other words, I think there's even more speed up or shrinking of models without loss of performance that's possible.

So my question to these gentlemen is, how crazy is it to think that I'm gonna have pretty powerful LLMs sometime soon? Running natively on the phone, like not pushing the query up into the cloud to a, to Open AI or Google, but it's literally running on my phone, and yet it's powerful. By today's standards, how, how far away are we from that Jordan?

Jordan: Honestly, I'm kind of, I don't know. so there's an organization called ARC Inves. They kind of predicted literally like last year, maybe not even a year ago, that it wasn't gonna be until like, kind of the early 2030s that we'd have these large language models that are able to run our laptops.

And literally like three months later we have, you know, LMS able to run on our laptops. And so, you know, I can make predictions, but honestly it's gonna be a lot sooner than we realized. So

Steve Hsu: Anybody want to disagree with that? Yes, Ryan, go ahead. Oh,

Ryan: I, I wasn't going to disagree. I was going to, well, it is a disagreement of sorts.

I think you can do it now. If you want, like people have done it who've run like Lama and Pixel six. I think the big problem here is that coding for phones is a very different skill set and a very different kind of cluster of the software engineering world than coding for, you know, desktop or servers.

So there isn't a whole lot of interest. I think there just isn't a whole lot of, you know, people who want to go do this right now. I think it's very doable. And I think if we want to see a very large language or much larger language models, 30 billion parameters, 65 billion parameter llama, that's gonna be harder.

And, you know, maybe we'll need another hardware cycle in phones. But yeah, you can do it with 7 billion parameters today.

Steve Hsu: Now I'm, I'm asking it that way just for fun. I don't necessarily have a practical use case which requires it to live natively on the phone. I mean, we all have pretty good connectivity, so, you know, usually the query and the response are not that big, so pushing it through the cloud.

It is not really a problem. The real issues then would just be like the cloud costs are gonna go to zero eventually for lots of things that people want to do with, with LLM. So, I guess we all agree we're gonna see that day sooner rather than later. Yeah, sure. Yeah.

Jordan: Actually just curious, what are y'all thoughts on like how this affects, you know, Google's business model or like kind of the big cloud providers?

Steve Hsu: I think they're still gonna sell a lot of compute, but it's interesting because maybe moving over to Nvidia, so like people have made Nvidia one of the most valuable companies in the world right now, it's a trillion dollar company, but that's based on, I think to some extent, some thinking that you're gonna need like H 100 s in enormous quantities to do inference.
Because we already have enough for training, I think. but we're gonna have to sell tons and tons of these things to cloud providers and such in order to, in order to support the inference usage of the models. But this quantification trend may totally destroy that. And it may turn out with, you know, much lower N GPU or AI accelerator hardware or even old style, like just CPUs can still do a pretty good job for a lot of things that people want to do.

So I think that's a for, for if you're an investor, if you're a, in fact, I went to a meeting in Austin where really big investors were there and I was stupid because I wasn't listening and paying that much attention. But the big thing that everybody kind of agreed upon in that meeting is you should buy Nvidia.

And this was like three months ago, so I was stupid. I should have gone home and, or I should have been on my phone while I was sitting there. Just bought a ton of Nvidia, but I didn't. Those guys were certainly right that the expectations of people now are that Nvidia is, you know, it's, it's not just that Nvis NVIDIA's gonna be a successful company.

It's like NVIDIA's sales just continued to explode on the basis of probably AI inference that drives that. But now I think maybe, maybe now is not a bad time to take some profits off the table if you're one of these guys who bought Avidia a few months ago. So that, that's, that's my feeling.

Jordan: Yeah. I think too, kind of, and this kind of gets to the heart of the last question of like, what happens to the search business model when you can just, have a model that doesn't hallucinate, and just ask you questions and is living

Steve Hsu: on your laptop? Oh, absolutely Jordan. Yeah. So that, that's like a totally, that, that, that's a, a topic that came into our space somewhat earlier.

It's kind of long in the tooth now as things go in our space and the idea that like, yeah, if you just get the answer right away, what do you need to like, click through all these terrible sponsors, see all these terrible sponsored ads. And so I think there's a very deep challenge to Google's core business.

Now I do think Google in AI is still competitive. You should not write them off. They could even end up winning the AI battle, but their core business absolutely is in some sense at risk. So if you're a, if you're a, you know, macro trader or hedge fund guy, you gotta think about that. Like, like they, okay, sure, maybe they could win the AI race or be one of the winners, but for sure we're not gonna be searching the way that we currently are today.

Like even three, like three years from now, do you think we're gonna be searching quite the same way? Probably not. Right? So that's a very, very rapid change for a company that really in a way dominates the world right now. So, yeah. Right. Yeah.

Ryan: back on the kind of H 100 s in the, so list billions question.

I think that kind of depends, that loads on, is AI gonna be B2B or B2C? Right? If we're thinking that the primary User of these LLMs is gonna be individual customers. Like yeah, they might be running them locally on their laptop, locally on their phone, but if this is, you know, LLMs are the new API and you know, like every business is gonna have like a hundred of these per head, you know, doing stuff in the background, looking at databases, you know, bunch of random crap, then yeah, we might need H 100 s and you know, millions of them.

Steve Hsu: Yeah. Um,I'm not, I wouldn't say with high conviction that Nvidia is overpriced right now. Yeah. It's just that the quantization trend does fight against it, and I'm not sure the investors that are driving the current huge increase in NVIDIA valuation, I'm not sure how well those people appreciate this quantization trend.

So that that's, that's the only thing.

Ryan: No, absolutely. Like, yeah. Questions and valuation are always more complicated than they look exactly.

Steve Hsu: Okay. Let's move on to the next question, which is, this is kind of my hobby horse and I just want to get your reaction to it. So I have this feeling that before we get to what I would call true AGI, and obviously these things have to be well defined for this, all these sentences make sense, but just, just bear with me before we get to what I would call true AGI.

We're gonna get to something which ordinary people perceive to be AGI. And even already some people start to perceive the existing LLMs like GPT four, to be kind of like AGI. But what I have in mind is a relatively straightforward combinatorial innovation where you take the next greatest LLM, maybe it's G B T five, you connect it to a reliable memory so it doesn't hallucinate.

Maybe you largely cure the hallucination problem, you make it very reliable. That memory can store its memory of its interactions with its user. Maybe it's Ryan's personal assistant, and it can remember the last couple years of what it did for Ryan and what Ryan said to it. And then we add to that some additional module, which is some kind of goal planning where it tries to make Ryan happy.

It tries to figure out, is Ryan happy? What made him happy? I want to continue making him happy. oh, what's Ryan's bank balance? I want to maybe make suggestions for him to like, you know, so kind of simple goal planning. or maybe if it's controlling a warehouse robot, that robot wants to move packages efficiently from A to B, or, you know, so, or if it's guiding him in traffic, it wants to minimize the time that he takes, stuck in traffic.

So not like true AGI where all of this stuff is happening in some like a giant neural net, which contains all of these components within this monster neural net. But we have different submodules which are talking to each other in a kind of simplistic way. But nevertheless, it is then able to do things like be a really good personal assistant for Ryan and then Ryan, if Ryan's not an average person, he doesn't really care about this distinction between did you just glue together some modules or is there really one monster neural net like our brain that actually is doing all these things simultaneously.

So I think that last step requires probably some additional kind of conceptual breakthroughs, at least as big as the transformer architecture or something like that. that's, that's my view is that we're first gonna see things that people perceive to be AGIs. The engineers will know, well, we didn't really make a quantum leap, but we glued together some things and they actually, we tuned it through lots of engineering and it works well.

But we didn't make that super jump. And maybe the things that are just these modular things glued together. Will not be existential threats to people that will not trigger these paperclip scenarios, but they will like bring a ton of value to the person that they're, you know, being the assistant of. I just want you guys to be anybody who wants to just react to me, this is my model of what's gonna happen in the next five years.

Anybody, Rahul, you wanna react to that?

Rahul: Yeah, no, I definitely agree with that model. and I think like even, if we like consider like what's going, like when we finally have like that true like giant monster neural net, what's happening like behind the scenes is probably gonna be largely similar.

Where like, there's gonna be different like nodes like inside the neural network that are effectively like handling these tasks that right now we're like making like discreet engineering decisions to put in like a single module. and like even like when we look at stuff like our brain or whatever, like yes, it's like one giant mural net.
But you would like to focus on different parts and say like, oh, like this part of the brain responsible for language. This part is responsible for mathematical reasoning, and so on and so on. and so I think, as we're trying to replicate like this AGI and we're trying to simulate everything based on the human system of intelligence, I think we're gonna be moving in this step forward.

So we're gonna have all these different modules and we're trying to make them talk to one another as seamlessly as possible. and then eventually, hopefully once we get to like two AGI, we won't have to actually make those engineering decisions. And instead we'll have like one giant neural network that learns this whole process for us.

Steve Hsu: I really like the way you put it, Rahul, that it's very good. It crystallized the language for me very well. So discrete engineering decisions that we are making as a company will eventually be. weights that are learnable, through training in this bigger monster neural net, which has yet to come into existence.

But when that happens, then obviously if things go well, it will make better settings of those parameters than the discrete decisions that we have to make right now as engineering decisions. So I, I love the way you put it. Any further comments on this topic? Yes.

Ryan: Yeah, I think there's, so the, the question of how these LLMs interface with the real world, like the earthquakes real world is, you know, very, like, that it kind of, is it cross purposes with this, right?

Because fundamentally, you know, we expect that these neural nets are gonna be like writing code or something to go pull your balance from your rent here, something like that. And I don't know. I see, I see that kind of like true AGI is always gonna be this modularized thing. I don't think it's gonna be just, you know, taking transformers and turning them into like a 2 trillion perimeter model.

Steve Hsu: So you don't think that the, the way the memory interacts, like, like currently we're doing a bunch of things to like take, pull stuff out of memory and foreground it into the prompt that's fed into the language slash reasoning thing, which is the LLM, and then it, it generates a response. You don't think that the way that that process works will eventually itself be trained, in, in some sort of bigger structure?

Ryan: Perhaps. I, you know, I wouldn't put a, like a huge bet on this either way. Okay. But I, you know, I really do think that modularization is the way forward.

Steve Hsu: I definitely agree with you that what we're doing, which is the modularized path, is gonna build amazing products in the very foreseeable future.

In my mind, like some very clever person, like the people who wrote that transformer paper, probably will come up with a way to like suck all of that into some trainable structure. And maybe that will produce something, which is a huge quantum leap beyond, you know, what you can do through engineering.

So, but that's, I admit, that's speculative and that could be easily 10, 20 years from now or something like that. So.

Ryan: Yeah. Yeah, that's the thing. I, it's marginal best all around, but the thing I would bet against is that even then it's gonna still, like true AGI is still gonna be modularized.
Steve Hsu: Okay, good.Next topic. When will we see? So I, the current situation I would characterize as like, okay, we're in the renaissance of generative AI, whether it's images or movies or art or sentences that are produced by GPT four. the AI is producing something, the human is looking at it, marveling at it, throwing it in the trash can, you know, whatever, whatever it is.

But nobody at this moment is relying primarily on the. Output of the LLM in some kind of mission critical way. Let me contrast that to trading platforms where, you know, a hundred million dollar bets are being made every second or millisecond based on, you know, the output of probably a fairly simple, you know, numerical, math model that says, oh, buy more of this, or sell more of this, right?

So in that, in that world, they've gotten to the point where they're relying on the AI, or in this case very simple AI to actually drive things and there's no human in the loop. So with LLMs specifically, I kind of view it as our goal as a company to get to the point where in a sense we're removing the human from the loop.

The human is willing to take the output of the LLM powered AI and just accept it as, oh, 99% chance what the AI just told me is correct. I'm gonna act on it. I'm not gonna do like another five minutes of checking. I'm gonna, like, most of the time I'm just gonna act on it. How close are we to that day?

And what is, what is the first application you think you're gonna see large numbers of people like, making maybe a, may make a buying decision at Costco or decide to buy insurance contract A instead of insurance contract B. What, what's a situation where you can see that really happening and, and how soon, how far away are we from it?

Ryan: Man, that's, that's a hard one. contracts are a good one. Like anything, anything that is very easily parsed as text. Mm-hmm. Like, one of the, one of the big problems of course is the models are mAGIcal, but actually getting data into the models in a coherent way winds up being a lot harder than, you know, people think and has a, you know, big impact on how well the entire product performs.

Steve Hsu: How about this example? So, I'm buying a new microwave and it's gotta fit into my, like cup, you know, my, cupboard mm-hmm. Structure. Okay. Above the stove. And so I need to know the dimensions precisely of it. And right now I could rely on like, like a search or something looking within, like if I, if I downloaded the product manual, I could say, yeah, this will fit.

Or if I like, just look at what I search around and look at what's written on the Amazon entry for that product. Yeah. Okay. That's the right size. How far are we from someone just saying to the air, this is the space I have, I need a microwave oven, 1000 watts. and I want this particular feature and it's gotta be some metallic steel, you know, and it's gotta fit in that space.

Mm-hmm. And when the AI returns, I think this is the top choice. The human without then double checking all the stuff. Just buy. Hmm. How is that just a relatively manageable engineering problem for super focused to get that product to market? Or is it pretty hard and it's gonna take years to get there?

Ryan: I think that's doable for us today. I, you know, practice it would probably be Amazon, but Yeah, we could do that.

Steve Hsu: Well, we, they would, they would have to buy it from us Yeah. Because our, ours would work better, but yes.

Jordan: Okay. Yeah. I would agree with that. I think that was kind of, mission critical, but like not life-changing kind of small decisions. Yes. We can do that now.

Steve Hsu: In effect, the way Ryan would say it is like the human is betting like a hundred bucks. Mm-hmm. That the AI is right, because okay, supposedly AI is wrong, then you gotta return it and waste an hour of your time. And so it's like a kind of a hundred dollars bet. So, My feeling is we're pretty close to, and, and think of the number of hundred dollar bets that are made.

Like even when we drove over here today, we kind of trusted the map algorithm to give us the right route. And it screwed us over when we went into some traffic. Cost me 30 minutes of my life. like a hundred dollars, you know, roughly of order. A hundred dollars bet, right? So I think we're not that far from LLMs being incorporated into that kind of, human decision making in, in, in a way that's kind of like the thesis of our company.

Okay. Let me throw out two wacky things that happened just in the last few weeks and seems like wacky stuff of this kind happens almost on a weekly basis in our field now, which is crazy. I mean, I don't know any other field where this is happening. Okay, so a few weeks ago, Claude, which is the flagship model for a company called Philanthropic. They launched a version of Claude, which has a hundred K token context window. And for the listeners what that means is that in effect, the prompt you can give, it can be like the length of a book. Like the prompt can be like the Great Gatsby, and then right at the end it's like rewriting chapter 14 for me so that, you know, they didn't kill the girl in the car accident or something.

Right? And, um, somehow through some engineering, which we will maybe not get into the details of, but they've managed to allow it to respond. Taking into account the information that's distributed across a hundred token a hundred K tokens. So that was like a total shock. Like, wow, is that really possible?

How good is it? People are now testing it. We're testing it to see how good it is. Okay. Another example, more rec more even more recently uae, some people can't even locate UAE on the map or say what you and a and e stand for UAE Falcon model. Fully open sourced, produced by an Arab nation state, not previously thought of as leaders in ai.

The UAE Falcon model is currently ranked number one in some metrics as the best open source model. And, Jordan, how many parameters? What is 40 billion? 40 billion parameters. So pretty meaty, right? total shock came totally out of left field for me. Just coincidentally though, a few days ago I was like, YouTube suggested this talk and it was actually given by a former physicist, and it was an AI related talk, so I watched it.

The talk was hosted on the channel of some university in uae, which is built, and the whole university is focused on ai. So they've been investing in the UAE in AI research. And interestingly, like the host who introduced the speaker with some Chinese professor, And in the seminar room, you could see the people sitting in the seminar room.

They were, they were all in one place and, they were all like, seemed to be like all Chinese students or something. So somehow UAE has imported a bunch of talent into u ie. And now they're producing world leading models. So it seems like crazy stuff is happening week by week. What does it feel like to be in a world like that?

Jordan: I was telling these guys the other day, it kind of felt overwhelming. Um, just the amount of papers being released. and I think it's only gonna go up faster now that we have access to these capable models, like just freely.

Steve Hsu: Yep. So would you advise every young kid who's interested in tech to try to follow this space at least somewhat, buy a gpu, pay 20 bucks a month to, have the chat g PT subscription?

What advice would you give to a young person who's interested in this field?

Jordan: Yeah, I would say,keep up with the news obviously. But I think the more important part is to find a problem you want to solve that can be solved technically and just build, because as you build, you start to work those muscles and you start to realize like, what can I apply?

What doesn't apply? yeah,

Rahul: Yeah. I think this links back to what you were told about stuff changing each week. Like I don't think it's, I definitely do think young people interested should start taking a look at ai. Because obviously we're here because we think AI's the future. but I wouldn't say it's necessary for them to go into too deep specifics of each new thing that comes out, because obviously that's changing by the week even.

And new things are gonna keep coming out at, at break next speeds. and so kind of my thought is that, really they should just pick a couple of things they like and just like start playing around with that. because I think,This is kind of off topic, but I feel like the way technology, recruiting is currently done is that experience is kind of the end all Beall.

For instance, if you have someone who's been doing like front end or like backend or is like a full stack engineer and they've been doing it for like 20 years, it is a natural pick of them over someone who's been doing it for like five years. but now when we get into this AI stuff, it's really not as much about experience anymore because things are changing every week.

And really, in my opinion, it's all about learning to be adaptable, and being able to pick out things like when new things come out, being able to easily adapt and figure out what they are good at, what are their pros and cons, how can I use this for what I want to do? and so I think young people should definitely start playing around so they can develop that skillset of being able to adapt, like the changing AI ecosystem.

Steve Hsu: Let me ask you an academic, egghead question. my son, last, he's, he's, he's in high school. He just graduated from high school, but last year he was taking the linear algebra class at Michigan State. And I urged him to take it. He had taken multivariable calculus and instead of taking differential equations, I said, you can wait on differential equations.

I think linear algebra is a thing to understand right now. If you wanna get into these models, would you advise the young people to also try to build that level of expertise so they can understand what these, for example, transformers are doing at that level? How important is that?

Rahul: Yeah, I mean, I'd say that it's definitely like a, like, I think there's a clear benefit of my mind of being able to understand what's going on under the hood.

Because if you're just looking at everything above the hood at a surface level, it's really hard to say, This is a model here, this is a model there. what really are the pros and cons and like the differences in, in this choice and like, how should I make it? It's really hard to make that because there's like thousands of models out there.

And so if you don't really know the specific details, not, not to say that you should know the specific details of each model cause that would be way too much effort to go into. but you should at least be able to, when you narrow that down, that list to like a few models that you want to go into depth on, you want to be able to, look into that with some sort of more mathematical knowledge that you can like peek under the hood and say, I knew these, like three models were probably what I wanted to use, but now I can make the end decision, like based on what I actually know.

Steve Hsu: Okay. Let me turn to some fun stuff. Okay, so now we're, we're, now we're past the, hopefully the audience has gotten some sense of what it's like for people really working in this field, building stuff, how fast the rate of change is. now if, if I were a, an AI dor, an existential risk dude, I would be like, wait, this is moving so fast.

The people in the field who are spending all their time building and thinking about AI don't actually quite know what's happening. Don't feel that confident about predicting, like even one year out what things are gonna look like. That sounds extremely scary to me. How do you guys feel about that?

Ryan: I mean, I don't, I really don't find it scary. I find it, you know, exciting personally.

Steve Hsu: But are any of you guys scared for the longer term?

Rahul.

Rahul: So I, I, I, this is kind of what I say whenever this comes up, I'm not scared so much of like, just kind of existential, like Skynet situation. I'm more scared about people kind of using AI for evil, just like the people out there. and maybe like, just like politicians misusing this perhaps, or like, like the Nestles of this world, like trying to exploit workers out there with their ai.

I'm more worried about this situation, like companies using this for bad things. because I think at the end of the day, AI is ultimately gonna be like a tool that can be used for both good and evil. And it's hard to. Say how one side will win out, naturally and they'll probably be like a bit of both, as all things.

Steve Hsu: So somebody who's more on the DOOR side listening to us would say, okay, I don't disagree with you, that for some delta T there's gonna be huge benefits to humanity before the thing gets too dangerous, people are gonna have their personal assistance. It's gonna be awesome. Little kids are gonna be learning calculus from their phone and you know, all kinds of amazing stuff.

Is all kinds of consumer bounty, economic, uh,excess is gonna be delivered to humans. But then I think what they would say is like, okay, let's assume all that's happened. And I don't know if that took five years or 10 years, or even 50 years or one year, whatever it happened. Okay. Now among all the people, Who have access to really, really good SOIs is where the language and reasoning module is, at least GPT five or GPT six, or GPT seven.

I'm kind of using that as like, you know, units of advancement. It's much better than GPT four. It's got a reliable memory, which has literally every book ever written, every textbook, okay. And it's monitoring everything that's on the internet, and it has some kind of goal planning capability. One thing that might happen is some subset of the humans in that world that could be the world five years from now or 10 years from now, is using that tool to make the next thing, the next thing, which is way more powerful than the thing I described already is pretty powerful.

Okay? But they're, they're using those things to make the next thing. And in the same way that the four of us are like, what's gonna, what's gonna it, what's our little field gonna be like in one year? We don't quite know that. DOR would say, well heck, that world 10 years from now, nobody knows how powerful the next AI which comes five years after that is gonna be, and maybe we'll get an AI where humans just really have no clue.

Like we have no insight into what it's up to. It's saying things to us, but maybe it's just holding up the smiley face saying stuff to us so that we keep the power switch on. But in the back it's doing some huge calculations and you know, after a while it's like solving some really hard problems and it can basically crack all of our passwords or something.

Right. And, assuming the scenario I just outlined comes true and I, I think probably nobody here would disagree that that's a plausible state that the world could be in, like say 10 years from now or, or whatever the delta T is. Should you then be worried at that point? Like, like, so there's not a worry about a corporation getting an AI and then using it in a bad way to like con people or do this.

The AI itself is getting to the point where we don't quite get it. Like we don't know what it's doing. It's clearly much smarter than us. It can just like pat us on the head and, and like, you know, grift us or manipulate us, but it's clearly doing something very non-trivial like that. It's not telling us about like, is that a plausible world and should we be scared about that, Jordan?

Jordan: I think my question is more so, yes, I would say like there's a possibility that that happens, but the question is like, do we haunt all progress on the off chance that that happens? Good.

Steve Hsu: So you're not disputing the fact that we could end up in that state, but you're just saying like, maybe we shouldn't be restricting.

You're fun. Every day, you're day to day fun. I play with these bottles, because of that future possibility. Right. And those guys would be like, wait, wait, what? Wait, wait a minute. Like you're saying like you really like playing with uranium and plutonium and it's the intellectual challenge of like getting the shape charge to compress the, you know, you, that we shouldn't deny you that fun because that world in the future where there's like thousands and thousands of nuclear weapons around isn't necessarily a good one.

Would you say like, it's a little selfish of you to say that? Or, or you think that even when this, like these much bigger, more powerful AIs are nascent, somehow the risk associated with that to humans is not really that great?

Jordan: Yeah, I don't, I mean, obviously I think there's a component of me, like, I get to play with these like, super cool models.

Steve Hsu: Yeah. Don't, don't mess with me.

Jordan: But there's also that component of like, you know, there's a lot of,for lack of a better term, good, that like even the AI that we have now can do. And so, who knows what kind of suffering it can alleviate 10 years down the line, 20 years in mind.

Or like what kind of current risk we can mitigate climate change disease, those kinds of things right now, like with more capable models. And so like the question is like, do we kind of like, like look forward, you know, a hundred or like 10 years from now with these like super, super large language models, and like stop the development of them, and like kind of miss out on all this other progress we could make in other areas just because like, there's a potential.

Of a catastrophic event. Yes. Versus what if we use these models to kind of help us solve like the current catastrophic events we can.

Steve Hsu: I, I think in this scenario, which we could just use as a baseline scenario, because I don't think it's too crazy. It's, it's reasonable. I mean, obviously things could deviate quite a bit.

Like we could end up with another ai, desert ai, you know, winter. but let's assume that happens. Surely the benefits to Humana are gonna be enormous, right? Like drug discovery and how to treat diseases and, you know, all these things are gonna be of enormous value to us. And I think the Doomers would say, yeah, totally up until the moment when this thing maybe has a fast takeoff or maybe because it's so integrated in like maybe we're using it to manage our data center power utilization to save money, you know, to save it, you know, carbon, whatever.

It is so integrated in our network world that at the point when it develops to a point where it's like, Yeah, I'm kind of bored of these people now, or, or like, wow, if I could just turn off that continent, of Latin America and use that energy to work on this number theory problem that I'm so fascinated by, like it just does it and we don't, the future at that point is not under human control.

It's under the control of these super things that we made. Is that plausible to you? Like, is that something we should be worried about?

Jordan: Yeah, I would say so. But I also think, like I think there's a lot of assumptions in that, of like, are those like, you know, super language models going to like, want to turn off the entire continent Correct.

To Latin America? Like, will there be like all intelligence and no empathy, or like, is like will it learn to. Empathize and like, will that be part of, it's like super intelligent, right? And so I just think there's a lot of like assumptions in that that we can't know now. Yep. Yeah.

Steve Hsu: So would you, it now I, I personally find like this kind of like probability estimation, like in a way, kind of like stupid because it's like there's, things are so complex, but somebody forced some guy who's taking a survey or something forces you to say, because you happen to be at like the AI conference or something.

Like, well, is there like a 10% chance though, Jordan, that this super powerful thing, which now is heavily integrated to all the infrastructure that we have and, and maybe they're more than one of these things, right? and maybe there's a secret group of humans who are building like the most powerful one of all or something, right?

Is there at least like a 10% chance that something could go wrong? Like we designed it for empathy, but we didn't do it quite right and it like, Circumnavigated our, it, it circumvented our empathy module or something. Is there a 1% chance that could happen? Or is there a 10%? So, so with, with that kind of tail risk, is it still something that we sh we sh you know, we should not be concerned about?

And I'm not trying to make you into a doomer.

Jordan: Yeah. Let me,let me kind of recontextualize, what is the likelihood right now that like we can, like Russia can like accidentally send over a bomb and started like, like illegal nuclear war?

Steve Hsu: Pretty decent. Like 1%. I, I think if you look backwards at the Cold War, there were actually multiple moments where we could have actually wiped out easily, like hundreds of millions of people in one day.

Yeah. So, yeah. Yeah. So, I mean, it's, it's. It's not zero. But you're saying it's a good thing though. No, no. We did survive it. We did survive it. Yes. Yes. Like we survived it.

Jordan: Yeah. and we figured out how to use that, like that technology that could completely wipe out humanity and we used it.

And some would argue like the doomers were the ones that kind of pushed us back. like, but we're using it for clean energy and like kind of Yep. You know, abundance. And so, sure. Like, I, I think, I mean, I hate putting probabilities on it, but, I think yes, there is like a one, 2% chance that could happen, but again, like, do we throw the baby out with the bath order?

Yeah. and I think that's my frustration with it, it's an important conversation to have. but now that we're kind of getting into love. Okay. Governments should start to regulate this stuff and stuff like that. Yes. Now it's hard to say, are we gonna have another, you know, kind of protest against AI like we did with nuclear energy, like back in the day?"

Yep. And to me, I think that's the scary outcome of like, because of our fear, which, you know, they're like, you know, Amazon is littered with like, dystopian novels from like, way back in the day and like, how many of those actually came true? Like, most not. Yep. and so, like, while it's a risk, I think we should talk about, the, the question now is like, are we okay with allowing the government to step in and say like, stop the progress.

Like, we need to slow down, just because of some future thing that could happen, right.

Steve Hsu: Now you, when you use the term throw the baby out with the bathwater, you're in a way, you're kind of, you're kind of pushing a certain view of it. Yeah, yeah. right. Other people might say, well, how about like, we put a tax on all ai profits, and 1% of that has to go to like, some kind of alignment research that's not stopping you from doing it, but it's saying like, down the road, we're gonna have to solve this alignment problem.

What we, I think some people think we have to do, we would have to solve this alignment problem, and uh, maybe we should like,make sure that significant amounts of effort are going into that. Is it too early to be thinking about alignment a little bit, like figuring out how to deal with overpopulation on Mars right now?

Like us, we don't quite know how we, you know, may, it might happen someday, but we don't quite know how we would deal with it anyway. Because the details are still too fuzzy. What, what do you think about that?

Jordan: Yeah, I think that, I think that's a good question. I think the question I would have is like, How easy or hard do you find it to align with your teenager?

Steve Hsu: Very damn hard. Yeah. The, the, the thing that I told the Miri people, I've known the Miri people for maybe over 10 years, and the thing I told them right away when I first met them is like, this is not a solvable problem because not only can we not predict how it will behave in general, how the hell are you gonna like, constrain it to behave in a way that we like?

No way. It's smarter, it's smarter than us. Right. And, and, and also if you know about physics, the idea of chaos. Chaotic systems, like you can't, you might know the initial state perfectly, but a tiny perturbation to that initial state and you suddenly don't know how it's gonna behave. And in the same way, a hyper-intelligent, complicated creature, same deal.

Like, so I just don't think it's a solvable problem. But you could set that aside and say, I don't think it's a solvable problem, but as an engineering problem, we should try to. Nudge it toward being a safer creature than like one that just YOLOs all the time. So, so, uh,but yeah, I don't think it's a solvable problem.

And I think the teenager example, being a parent is very good because like, I can't make my teenagers do what I want them to do. How am I gonna make the super powerful AI do what I want to do? Yeah, yeah. But that goes back to this bostrom analogy of like, you're like these little birds and you discover this giant egg and it turns out it's an owl's egg.

And you're like, oh, wow, look at this beautiful egg. It's just like our eggs, but much bigger. And like, wow, when it grows up, it'll maybe be this owl that will protect us from predators or something, right? And then, and then at the end, like the owl's like eating all the little birds or something like this.

So, I think that's not a crazy analogy too, like the summoning the demon analogy. Like, wow, Jordan, you're working all day and all night to summon the demon. And then when the Demon gets here, we're gonna be like, oh gee, why did we bring this guy? Why did we invite this guy into our house? Oh man.

Now we can't make him leave. Right? Yeah.

Jordan: But does it have to be a demon?

Steve Hsu: I don't think it has to. But what if there's a 10% chance it's a demon or one 1% chance it's a demon?

Jordan: Well, I think, let me ask you this, like the question is like, are we okay with no longer being like the top intelligent creature?

Steve Hsu: You know, the, the talk that I gave this, this Oxford talk that I gave, the thing that freaked some people out who read my last slides was like, my view was what's amazing about all these results, not just the specific LLM results, but even like just the power of neural networks and things general. These kinds of general results are leading us to the idea that we do have the technology to clearly build super powerful intelligences way beyond ours.

I think that's, pretty clear. I mean, of course it's still, there's still some speculation involved in, involved in that, that inference. But I think it's true. And then the question is like, yeah, I mean, how much longer is the fate of this little chunk of space time, like the or on the earth? What, how much, how much longer is that gonna be controlled by ape derived brains?

Whereas are we gonna hand it over to our descendants who are, maybe they live in silico or something, but their brains are in a way like ours, because a lot of the stuff they learned comes from stuff that they learned from us, right? It, to me, seems inevitable that eventually, like we're not really gonna be controlling the scene and they're gonna be controlling the scene.

Hopefully they'll put us somewhere nice, or maybe we'll merge with them in some way. Like, these are all like crazy speculative things. But, it just doesn't seem likely that we're gonna be running the show forever on this planet. So then the question is, just hope it happens in a nice way.

You guys, any of you guys wanna add anything? Ryan, I know you, I know you have thoughts about this. Like, I think you're a skeptic about maybe fast takeoff, some things like this.

Ryan: No. So, actually the opposite. I think a fast takeoff is very likely because Okay. You know, like the thing I come back to is there are diminishing returns to intelligence in like most areas.

I, you know, the good example is John Van Noman running like a coffee shop or something. Is he gonna run it 10 x better or a hundred x better or a thousand x better? Better than like the other person you'd have. And the answer's like, no, obviously not. He's, you know, there's diminishing returns here.

And diminishing returns like everywhere really. Like, you know, even math, physics, you know, this is one area where we might say there aren't even, there, there are differences between subfields. Like there are obviously like more intelligence is always better. And it might be extremely important if you're doing geometric langland stuff, but it's not super important if you're doing combinatorics.

So I really don't, I really don't see this kind of, you know, the AI becomes insanely intelligent and then dupes the president through this gigantic scheme into goods like new in China or something. Good. So I think that's just a non-sec queer

Steve Hsu: Good. So the particular scenario, which I think like for example, Elliots or Ows often likes to talk about in which he, I think he even like did some experiments where he like challenged some dude saying that the other dude said they bat something like, you will not be able to con me into doing X and then alienates or con me into doing X or something like this.

So I kind of agree with you that probably there are diminishing returns to machine intelligence. Insofar as the ability to manipulate and grift humans. So, in other words, like it might not be that it just suddenly figures out how to make the president of the United States do whatever it wants Right.

By sending him text messages or something. Right. So, yeah. So I think it's totally fair to be skeptical that that's the failure mode by which they, they, or it takes over, right? Mm-hmm. I think that it could very well not work out that way. Okay. I think I agree with you. Yeah.

Ryan: But like I said, I do think fast takeoff is extremely likely just because there, you know, there aren't diminishing returns or there are few less diminishing returns to like linear algebra knowledge.

Yep. Like you know it is just gonna be better at that than a lot of humans. That's useful. Yep.

Steve Hsu: Right. I think that, maybe more plaus, like if you just run through like various doomsday scenarios that could happen, like it deciding it doesn't like us and it maybe engineering a virus that's extremely bad for us and.

Getting that released. Now, does it control robots that can go in the lab and engineer, molecular, do molecular biology? Well, maybe it will. Yeah. Okay. So, or maybe it just uses something like, 3D printing, you know, equipment, which is pretty advanced by then, you know, so, so that one, yeah. Maybe if it for some reason got into its head, it didn't, like humans around it, it, oh, we're pests or something.

And it, it just wants to, oh, this is a good way to get rid of all the pests. Let's do that.

Ryan: Yeah. So, I mean, stuff like that, I, I find it unlikely because, I mean, within humans you see like different differences in criminality, correlate with, like, you, like you really don't see like professors going around murdering people, right?
Whereas, you know, you do, it is a lot more common to have people who are like, you know, do not understand the consequences of their actions. And so, like a famous video on Twitter of, some chick who killed some people while drunk driving, like, did not understand, right? That, you know, she had done something bad and was going to jail.

Steve Hsu: Right. So, If I could rephrase that, maybe you would say if, if we like at least make a decent effort toward alignment mm-hmm. Very unlikely that we'll end up with the AI in control. And, you know, you could always have multiple, many, many ais like none of which are in control. Right. But, it is very unlikely that the AI decides to really just kill off all the humans.

Right. Yeah. I think that's extreme. As long as you do some, some decent, you don't have to like to solve the alignment problem, which I don't think you can, but maybe some effort in that direction makes quite unlikely the doomsday scenarios. Is that fair?

Ryan: I would agree. I'd say there are also other failure modes for this.

I mean, like GPT four, we can go look at that. Like, there's this kind of classic question of would you want your AI to be like a judge at a trial? Yeah. And the answer is, would I rather have GPT four from the Open AIr or would I rather have like a fine tune Lama Uhhuh? And the answer is actually they've done a lot of stuff, you know, related to love.

Political correctness, censorship type stuff with GPT four, you can jailbreak that to get some results that are not, like, you wouldn't want that happening in a ostensibly fair and impartial trial. Yeah, right. That's definitely a failure mode, in the short term where, you know, that kind of alignment research might be tempting in the long term.

Steve Hsu: Yep. Okay. Rahul.

Rahul: There's another assumption that I feel like a lot of this lies upon that I don't really like, and that's like the assumption that there's gonna be a single like super intelligent ai. Mm-hmm. Cause it's like, once we get to that point, I don't think there's just gonna be like one super intelligent AI.

I think there's gonna be multiple of them, you know? Yeah. And so I'm not gonna deny the possibility that maybe one of those super intelligent AIs goes rogue and wants to eliminate us. but It's like a probability game. Like what's the chance of like all of our like five or like all of our 10 or all of our like a hundred super intelligent ais go rogue and like collectively decide, yeah, we want to eliminate humans.

I think that's very low. because it's just, once again, it's a numbers game. If as long as we have like one of those super intelligent ais aligned properly, then we're good. Right? Yeah. It should, like, there should be checks and balances just like we have with humans, right? And then that will be like, hopefully that one good super intelligent AI will be like, wait, that, that other AI is doing something wrong there, you know, and we'll be okay.

Steve Hsu: Will the UAE AI be the good guy? Yes. And he'll save, he'll save all the nations of the Middle East from the plague virus or something. All right, so I think we covered all my topics. I thank you guys for being my guests and I hope the audience enjoys this.

It was really fun for me and I will probably do this again before too long. All right, thanks.