The Winds of AI Winter (Q2 Four Wars of the AI Stack Recap)

Key Takeaways

[Music] hey everyone welcome to the laden space podcast this is alesio partner and CTO and residents and deible partners and today we're in the Singapore studio with swix hey uh this is our long awaited oneon-one episode uh I don't know how long ago the previous one was do you remember three four months no yeah it's been a it's been a while minute people really enjoyed it it's it's just really I think our travel schedules have been really difficult to get this stuff together and then we also had like a decent backlog of guests for a while I think we've kind of depleted that backlog now and we need to build it up again uh but it's it's been busy and there's been a lot of news so we actually get to do this like sort of rapid fire thing I think some people you know the podcast is grown a lot in the last uh 6 months maybe just reintroducing like what you're up to what I'm up to and um why we're here in Singapore and and stuff like that yeah my first time here in Singapore which has been really nice this country is really amazing I would say first of all everything FS like the busiest part of the city everything is C rers there's like plants in all the buildings or at least in the areas that I've been in which has been awesome and I was at uh one of the offices kind of on the south side and from the 30th floor you can see Indonesia on one side and you can see Malaysia on the other side um so it's uh quite quite small one of the people there said their kid goes to school at the border with Malaysia basically so they could drive to Malaysia every day just to go pick her up from school yeah yeah and we came here we hosted um with you the Sovereign AI Summit Wednesday night we had a a lot of Goldman tamasic g s and we're going to talk about this trend of sovereign AI which maybe we might cover on another episode but basically how do you drive if you're a country how do you drive productivity growth in a time where populations are shrinking the workforce is shrinking and yeahi can kind of supplement a lot of this and then the question is okay should I put all this money in Foundation models should I put it in data C in infrastructure should I put it in gpus should I put it in agents and whatnot so we'll touch on some of these Trends in in the episode but it was a fun event and I did not expect some of the most senior people at the largest financial institution in Singapore ask about stay space models and some of the Alternatives so it's great to see um how advanced the conversation is sometimes yeah I think that that is mostly people trying to listen to jargon that is being floated around as like oh what kill Transformers and then they jump straight there without actually exploring the fundamentals the basics of what they'll actually put to work that's fine it's it's a form to ask questions so you want to ask about the future but I feel like it's not very practical to spend so much time on on those things you know part of the things that I do with l space especially um when I travel is to try to ask questions about what countries that are not the us and not San Francisco can do because everyone feels a bit left out you feel it here as well and I'm trying to promotee Alternatives I think EI engineering is one way that countries can can capitalize on the industry without Building 100 billion dollar cluster which is one5 the GDP of Singapore um and uh and and so you know what my pitch at the summit was that uh we would Singapore with the AER Nation we're also working on bringing um the AER conference to Singapore next year together with iair so yeah we we're just trying my best and you know I'm being looped into various government meetings to to try to make that happen we'll we'll definitely be here next year we'll be I'll be back here very often it's really nice yeah awesome okay well we have a you know a lot of news uh how do you think we should cover maybe just recap since the framework of the forwards of AI is something that came out end of last year um so basically we'll Link in the show notes but the endof year recap for 2023 was basically the for wors ofi uh which we picked GPU Bridge versus GPU poor the data quality Wars the multimodality wars and the rag SL Ops Wars so usually everything falls back under those four categories so I'm pretty happy that seven months later it's something that still matters still kind of holds up yeah most most day IED St from eight months ago it's really not that relevant anymore um and today well we'll try and bucket some of the the recent news on it we haven't done a monthly thing in like three months so three months yeah is a lot of stuff that's mostly because I got busy with the conference yeah um but I I do want to I actually I do want to get get get back on that horse or maybe just do it weekly so that I don't have such a big lift that I don't do it I think the activation energy is is the problem really um so yeah uh I think Frontier Model wise it seems like Claude has really carved out a position persistent space for itself you know for a long time I thop it was kind of like a clear number two to open Ai and with .5 on it at least in like the some of the hard benchmarks on lmis or coding benchmarks on lmis it is the Undisputed number one model in the world even with 40 mini uh and we can talk about 40 mini and benchmarking later on but for Claud to be there and hold that position for uh what is almost what is more than a month now in in AI time is is a big deal there's not much that people know publicly about what they what anthropic did for cloud on it but I think it's it's still a huge achievement it it marks the beginning of a non-open entric world to the point where people on Twitter have cancelled chat gbt that's been a trend that's been going on for a while we talked about the unbundling of chbt but now like new open source projects and tooling they just built for cloud like they don't even use open ey that's a strategic threat to open ey I think a little bit uh obviously open ey is so big that it doesn't really care about that but uh for anthropic is a big win I think like to to see that going in theanthropic differentiating itself and actually implementing research so the rumor is that the scaling monos semanticity paper that they put out um two months ago was a big part of cloud 35 onet I've had off the Record chats with people about that idea and they they they don't agree that it is the only cause um so I I was I was thinking like this the only thing that they did but um you know people people say that uh there's about four or five other tricks that they haven't disclosed yet that went into 35 Sonet but the scaling one of semanticity paper is a very very good read it's very long read but it basically says that you can find control vectors control features now that you can turn on to make it better at code without really retraining it you just train a whole bunch of sparse Auto encoders find a bunch of features and just say like let's up those features and suddenly you're better at code or suddenly you car a lot about the Golden Gate Bridge these are the same things to the model that is a huge huge win for interpretability because up to now we were only doing interpretability on toy models like few million parameters a model of go or chess whatever CLA 3 on it was interpreted and usefully improved using this technique wow no I I think it would be amazing if we could replicate the same on the openen models to then because now we can use Lama 3.1 to generate synthetic data for training and fine-tuning I think obviously anthropic has a lot of compute and a lot of money so once they figure out okay this is what we should make the model better at they can kind of like put a lot of resources I think in open source is probably going to be a more distributed effort you know like I feel like news has held the crown of like the best fine tuning data set owners for a while but at some point that should change hopefully you know like other other groups should should step up and I think if we can apply the same principles to like a model as big as 405b and bring them into like maybe the 7B form factor that would be great but yeah cloud is great I can tell CH gbd a while ago every small podcaster run for Laden space it runs both on Claude and on openi and Claude is definitely better most of the time it's not a benchmark It's Just Vibe but when The Vibes are good The Vibes are good you we run most of the I new summaries on cloud as well um and but and I always run it against open ey sometimes open ey wins um I I do a daily comparison but yeah cloud cloud is very strong at summarization and instruction following which is something I care a lot about so when you talk about Frontier models mlu no longer cut it right like uh we have Rich reach like 92 on mlu it's going to like 95 97 it just means you're memorizing MML like there there's some fundamental irreducible level of mistakes because of M quality uh we talked about this with Clementine on the hugging face episode and so we we need to to see what uh what else what is the next Frontier I think there are 10 directions that I outlined below but we'll talk about that later yeah should we move on to Lama three yeah 3.1 I guess that to make sure they good differentiate between the the models yeah but yeah we have a all episode with uh Thomas Shalom from um the the meta team which was really really good and I'm glad we got the podcast to come out at the same time as the model I think we're the only ones to coordinate for the paper release for the for the big launch the 405 launch zck did a few interviews but we the only ones that did the technical Team interview yeah yeah yeah I mean they were like surfing or something with the Bloomberg person we should get invited to Surf with zck but I I would be down to to the to the audience the the technical breakdown was so so behind the scenes you know uh one for listeners one thing that we have we have attention about is who do we invite because obviously if we get Mark Zuckerberg it'll be a big name then it will cause people to download us more but it will be a less technical interview cuz he's not on the research team he's he's CEO of meta and so like I think this constant back and forth like we want to grow as a podcast but we want to serve a technical audience and we're trying to do that and thread that line because our currency as podcasters is the people that listen to it and we need big names but we also need to serve our audience well and I think um if we if we don't do it well this actually goes all the way back to George Hots when when after he finished recording with us he he said you have two paths in the podcast world either you go be Lex Freedman or you stay you stay small on Niche uh and we we definitely like we like our Niche we think it's a it's a good Niche it's going to grow but at the same time we I still want us to grow I want us to grow on YouTube right and and so uh that's that's always like a meta thing that not to get to meta no not that meta the the other meta yeah L three yeah I think to me the biggest thing is the training gun outputs like every company is just hiding the fact that they've been find tuning and training on gp4 outputs and you cannot technically do it but obviously open ey is not enforcing it I think now for the first time there's like a clear path to how do we make a 7B model good without having to go through gb4 or going to CLA 3 and we'll kind of talk about this later but I think we're seeing maybe the you know not the death but like sellling the Pix and shovels is kind of going away and like building the vertical things is like where most of the value is actually getting capture at least at the early stages so being able to make small models better at specific things through a large model is more important than yet another 7B model that I can try and use but at the end of the day I still need to go through the large labs to find tune so that to me is the most interesting thing you know it's such a large model that like it's obviously amazing but I don't know if a lot of people are switching from gbd4 or Cloud 3.5 to run 405b I also don't know what the hosting hosting options are as far as like scaling you know I don't know if the fireworks works and togethers of the world how much capacity they actually have to serve this model because at the end of the day it's a it's a lot of compute if some of the big products will switch to it and you Cann easily run it yourself so um I don't know but to me the synthetic data piece is definitely the most the most interesting yeah uh I would say that it is not enough now to say that synthetic data is real uh I I actually ship that in the original email and then I changed that in the in the sort of what you see now in the in the podcast description but because there it is so established now that syntic data is real therefore you need to go to the next level which is okay what do you use it for and how do you use it and I think that is what was interesting for Lama 3 for me you you read the paper 90 pages of all filler no killer is something like that this is what the people were saying very very like for once a Frontier Model with proper paper instead of a marketing blog post and uh you know they actually spelled out how they use synthetic data for a few different domains so they have uh synthetic data for code for math for multilinguality for long context for Tool use and then also for ASR and voice generation and I think that yeah okay now you have the license to go dis still Lama 3 405b but well how do you do that that is the sort of the next Frontier now you have the permission to do it how do you do it and uh I think people are you know going to reference L three a lot but then they can use those techniques for for everything else you know in our episode with Thomas he he talked about like I was very focused on synthetic data for pre-training cuz that's that's my context that's my conversations with technium from news and all and um all the other people doing synthetic data for pre-training and fine training but he was talking about post trainining as well and for uh everything here was post training uh in fact I wish we had spent more time with Thomas on on this stuff uh we just didn't have the paper beforehand U but I think like when I call lth three the syth synthetic data model is you have the license for it but then you also have the road map the recipe because it's in the paper and now like now everybody knows how to do this uh and and probably you know obviously like opening eyes probably laughing at us because they they did this like a year ago but now it's in the open yeah I mean they can laugh all they want but they're coming for them I I think I mean that's definitely the biggest Vibe shift right it's like obviously Lama 3.1 is good obviously Claud is good maybe a year and a half ago you didn't get the benefit out the doubt as like an opena competitor to be state-of-the-art you know it was kind of like oh on Tropic yeah those guys are cute over there they're trying to do their thing but it's not open Ai and like llama 2 is great but like it's really not a serious model you know it's like just good enough I think now it's like every time an Tropic releases something people are like okay this is like a serious thing whenever like meta releases something it's like okay they're at the same level and I don't know if o is kind of like sandbagging the yeah I don't and then they kind of you know yesterday or today they announce the search gbt thing behind the weight list this is the Singapore confusion when was it that happen because it happened yesterday us time but today Singapore time thday it's been really confusing but yeah and people are kind of like oh okay open ey I don't know if we can take you seriously well no the one of the uh AI Grant um employees I think hsh tweeted that you know you can skip the weight list just go to perx.com and that was was really really sick burn for the open I search GPT weight list but their implementation we'll have something different they probably like train a dedicated model for that you know like they'll have some Innovation that we hav data licensing obviously uh data licensing yes we're optimistic you know but the the vibe sh is real and I think that's something that it's just worth commenting on and watching and um yeah how the other labs catch up I think what you said there is actually very interesting the the trend of successive releases is very important to watch if things get less and less exciting then it's a red flag for that company and if it's things get more and more exciting it means that they these guys have have a good team they have a good plan good ideas um so yeah like I will call out you know uh the Microsoft f team as well um F1 was kind of widely regarded to be overtrained on benchmarks and f 2 and 5 3 subsequently improved a lot as well I would say also similar for Gemma Gemma 1 and two uh Gemma 2 is currently leading in terms of the uh local llama sort of VI check eval informal straw pole and that's only like a month after release they released that the um the engineer Worlds Fair and um you know like I didn't know what to think about it cuz GMA 1 wasn't like super well received it was just kind of like here's here's like free tier Gemini you know but but now GMA Gemma 2 is actually like a very legitimately widely used uh model by by the open source in local Lama community so that's great until llama 37b came along and so like the and we'll talk about this also like just the The Winds of a winter is also like what is the depreciation schedule on this on this model inference and training cost like it's it's very high yeah I'm curious to get your thought on mrr uh everybody's favorite sparkling weights um company they they just released the you know mrr large enough LGE Mr Large two yeah large two uh so this was one day after llama 3 U presumably because they were speaking at icml which is going on right now um by the way Britney is doing a guest host uh thing for us she's running around the poster sessions doing what I do which is very great cuz I couldn't go cuz my Visa issue I have to be careful what I say here I think because we still want to respect their work uh but Mr allar I would say it's like not as exciting as llama 3 that I think that is very very fair to say it is yes another gpc4 class model released as open weights with a research license not a commercial license but still open weights and that's good for the community but it is a step down in in terms of uh the general excitement around mraw compared to llama I think that would be fair to say and I would say that to mraw themselves so the the general hope is and I cannot say too much it's because I I've had offline conversations with with people close to this the general hope is that they need something more you know of the 10 elements of like what is next in terms of like Frontier Model boundaries MRA needs to make progress there they made progress here with like um instruction following and structured output and multilinguality and all those things but I think to stand out you need to basically pull a stunt you need to be a superlatively good company in one dimension and now unfortunately MCH does not have that Crown as open source Kings you know like a year ago I was saying M are the kings of Open Source AI now meta is they've lost their Crown uh by the way they've also deprecated uh mistr 7B 8X 7B and 8x 22b right so now there's only like the close Source models that are API platform so has MRA basically started becoming more of a Clos model proprietary platform I don't believe that's that's true I believe that they they're still very committed to open source uh but they need to come up with something more that people can use and that's a that's a grind I mean they have what $600 million to do it m right so that's that's still good uh but you know people are waiting for like what's next from from them yeah to me the perception was interesting in the comments of the release everybody was like why do you have a non-commercial license you're not making any money anyway from the inference so I like the AI engineering tier list you know is kind of Shifting in real time and maybe Mr like you said before it was like Hey 10 God for these guys they're saving us in open source they're kind of like speedrunning gbd1 gb2 gbd3 in open source but now it's like they're kind of moving away from that I haven't really heard of that many people using them as skill commercially just from you know discussions um so I'm curious to see what the next step yeah but also you're sort of us- based and maybe they're not focused there right so yeah exactly it's a very big elephant and we only touching pieces of it it's it's blind you know blind leading to Blind uh I I will call out um you know they have some interesting experimentations with the Mamba and mistal Nemo is actually on the efficiency Frontier chart that I drew that is still relevant so don't discount Mr Nemo but Mr Large otherwise like it's it's an update it's a necessary update for M large V1 but other than that they're just kind of holding the the line not really advanc ing the field yet that will be my statement there so those are the frontier big Labs yes and then now we're we're going to shift a little bit towards the the smaller Deployable on device Solutions yeah first of all shout out to our friend Trea uh who released flesh attention 3 flashh attention 2 we kind of did a deep dive on the podcast he came on in the studio back then it's just great to see how small groups can make a big impact on a whole industry just like by making math better so it's just great to see I just want to give Tri a shout out something I mentioned there and something that always comes up even in the Sovereign Summit that we did was what does nvidia's competitors have have any threat to Nvidia you know AMD like madx like um etched and which which caus a lot of noise with their Soo chip as well and just the simple fact is that Nvidia has won the hardware Lottery and people are customizing for NVIDIA like fresh attention 3 only works for NVIDIA only Wass for h100s and like this much work this much scaling this much validation going into this stuff is very difficult to replicate or very expensive to replicate for the other Hardware ecosystems so not impossible I actually heard a really good argument from one of um I think it is Martin cassado from a16z who was saying basically like yeah like absolutely invidious hardware and ecosystem makes sense and obviously that's that's contributed to its like I don't know like it's like the most valuable company in the world right now but current training RS are like 100 million to 200 million in cost but when they go to 500 million when they go to a billion when they go to 1 trillion then you actually start justifying making custom A6 for your run M and if they if they cut your costs by like half then you make your money back in one run yeah yeah yeah Martin has always been a a fan of Uh custom isic I think they wrote a really good post maybe a couple years ago about um Cloud repatriation oh yeah I think a for that but it's becoming more consensus now I think so no shazir blogging again fantastic gift of the world this guy non-stop bangers uh and so he's at character Ai and uh they they he put up post talking about five tricks that they use to serve 20% of Google search traffic as llm inference a lot of people were very shocked by that number but I think you just have to remember that most conversations are multi-turn right like in the span of one Google search I will send like 10 text messages right so obviously there's like a ratio here that uh that matters it's obviously a flex of character ai's traction among the kids because I have tried to use character AI since then and I still cannot for the life of me get it I don't have you tried uh I tried it but yes definitely not yeah they launched voice I Tri to talk to it it was just so stupid I just didn't like it myself but uh this is what come on the podcast know she's here sorry we didn't mean no no no it's because like like I I don't really understand like what the use case is for apart from like the the therapy roleplay homework assistant uh type of type of stuff that is the norm but anyway one of thing most interesting things see detail five tricks um one thing that people talk a lot about is native in8 training I I got it wrong in a Thomas podcast I said fp8 is in8 and I think like that is something that is a easy win like I we should we should basically when we're getting to the point where we're overtraining models 100 s past chinula ratio to optimize for inference the next thing is actually like hey let's stop using so much memory and when when training because we're not we're going to quantize it anyway for inference so like just let's pre let's pre- quantize it in in training so that makes a lot of sense the other thing as well is this concept of global local hybrid architecture which I think is basically going to be the norm right um so he has this formula of 1 to five ratio of global attention to local attention and he says that that is that is uh that works for the long form conversations that character has okay that's great and like simultaneously we have Independence research from other companies about similar hybrid ratios being the best for their research so Nvidia came out with a Mamba Transformer hybrid research thing and their in their estimation you only need 7% Transformers everything else can be St space models Jamba also had something like between like 6 to like 30 to one and uh basically every form of hybrid architecture sees be working at the research stage so I think like if we scale this it makes complete sense that you you just need a a mix of architectures and um it could well be that the Transformer block instead of Transformers being all you need Transformers are the global attention thing and then the the local attention thing can be the Stace models can be the RW KVs can be another Transformer but just uh limited by a sliding window and I I think like we're slowly discovering like the the fundamental building blocks of AI one is Transformers one is something that's local whatever that is and then you know who knows what else is next maybe the other the other stuff is adapters we can talk about that but yeah headline is that gnome maybe he's too confident but I mean I believe him gome thinks that he can do inference at 13 neck cheaper than the fireworks and together right so like there is a lot of room left uh to to improve inference I mean it does make sense right because like otherwise yeah exactly I was like they would be they would be losing a ton of money so um they are rumored to be exploring a sale um so I'm sure money is still an issue for them but I'm also sure they're making a lot of money so I it's very hard to tell because it's not a very public company well I I think that's one of the um things in the market right now too is like hey do you just want to keep building do you want to like just not worry about the money and go build somewhere else kind of like a maybe inflection and adapt and some of these other Nona hires licensing deals and whatnot so I'm curious to see what companies decide to stick with it it's like you uh I think Google or meta should pay $1 billion for Gnome alone right the the purchase price for character is 1 billion mhm which is super which is nothing at their market caps right like meta's market cap right now is 1.15 trillion because they're down 5% 11% in the past month what um yeah so if you pay 1 billion you know that's like 0.01% of your Market cap and they PID they pay 1 billion for WhatsApp and they buy 1% of their market cap on that at the time so yeah that is beyond our pay grade but the the the last piece of the GPU Rich poor Wars so so we're going from the super GPU Rich down to like the medium GPU rich and now now down to the GPU poor is on device models right uh which is something that people are very very excited about so at my conference uh Mozilla AI I think was kind of like the Talk of the Town there on Lama file uh we had Justine T come in and explain explain like some of the optimizations that they did and their their just general vision for on device AI I think that like it's basically the second act of Mozilla like a lot of good with the open source browser and uh obviously then they have since declined because uh it's it's very hard to keep up in in that field and MOA has had some management issues as well but now now that the operating system is moving to the AI layer now they're also like you know promoting open source AI there and also like private AI right like open source is synonymous local private all the good things that people want and I think their vision of like even running this stuff on CPUs at at a very very fast speed by by just like being extremely cracked I think is very understated and uh we should probably try to support it more um and it's just amazing to host these these people and see their progress yeah I think to me the biggest question about on device obviously there's a Gemini Nano which is getting shipped with chrome yeah so so let's survey it right so Lama file is one executable that runs on every architecture y similar for by the way Mojo from from modul which also spoke at the conference and then what else llama CPP mlx those those kinds are also that layer then the next layer up would be the builtin into uh their products by the by the vendor so Google Chrome is building Gemini Nano into the browser the next version of Google Chrome will have Nano inside that you can use like window. a do something and it would just call Nano there there will be no download no latency what whatsoever cuz it runs on your device and there's Apple intelligence as well which is Apple's version which is in in the OS accessible by apps and then there's there's a b longtail of others but like yeah your your comments on those things my biggest question is how much can you differentiate at that model size you know like how big is going to be the performance gap between all these models and like are people going to be aware of what model is running you know right now for the large models we're still pretty aware of like oh is this Sonic 3.5 is this gbd4 is this you know 3.1 405b I think the smaller you get the more it's just going to become like a utility you know so like you're not going to need a model router for like small models you're not going to need any of that like they're all going to converge to like the best possible performance actually Apple intelligence is the model router I think uh they have something like 14 I did a account in in my newsletter like 14 to 20 adapters and So based on your use case they they'll they'll route and load the adapter or they'll route to open AI so there is some routing layer to me I think a lot of people were trying to puzzle out the Strategic moves between open ey and apple here because apple is in a very good position to commoditize open there were some rumors that Google was working with apple to launch it they did not make it for the launch but presumably Apple wants to commoditize open ey right so you know when you when you launch you can choose your preferred external AI provider and it's either I or Google or someone else that puts Apple Apple at the center of the world as with the ability to make routing decisions and um I think that's probably good for privacy probably good for the planet cuz you're you're not running like oversized models on like your your you know your your spell check uh task and I I'm generally pretty positive on it like yeah I I'm not concerned about the capabilities issue it meets their benchmarks Apple put out a whole bunch of proprietary benchmarks cuz they don't like to do anything in the way that everyone else does it so like you know in the Apple intelligence blog post they like I think like all of them were just like their internal human evaluations and the only one of them was the industry standard Benchmark which which is if evl which is good but like you know why why didn't you also release your MML oh cuz you suck on it all right I I I actually think all these models will be good and on the Apple side I'm curious to see what the price tag will be to be the default right now Google pays them 20 billion to be the default search I see I the rumor is zero yeah yeah I mean today even if it was 20 billion I see that's like nothing compared to like you know Nvidia were three trillion so like paying 20 even paying 20 billion to be the default AI provider like would be cheap compared to search given that AI is actually being such a core part of the experience like Google being the default for like Apple's phone experience really doesn't change anything yeah yeah becoming the default AI provider for like the Apple experience would be worth a lot more than this I mean so I can justify it being zero instead of 20 billion is because open air has to foot the inference cost right so that's a lot well yeah Microsoft really is putting it but again Microsoft is for two trillion you know so as someone who this is the web developer coming out uh as someone who is a champion of the open web Apple has been let's just say a roadblock in that in that direction I think Gemini Nano being good is more important than Apple Intelligence being generally capable Apple Intelligence being like uh on device router for Apple apps is but like if you care about the open web you really need Gemini Nano to work and uh we're not sure like right now we have some some demos showing that it's fast enough but we haven't had systematic tests on it along the lines of that research I I would highlight that Apple has also put out data comp LM I actually interviewed data comp at newps last year and they've branched out from just vision and images to language models and Apple has put out a reference implementation of the 7B language model that's built on top of data comp and it is better than fine web which is huge because fine web was the state of the art last month and that's fantastic so so basically like data comp is a open data open weights open model like super everything open so there will be a lot of people optimizing this this kind of model they'll be building on architectures like mobile LM and small LM which basically uh innovate in terms of like shared weights and shared matrices for uh for smaller models so that you you just optimize the the amount of file size and and memory that you take up and um I think the just general trend of on device models like the only way the intelligence to cheap to meter happens is everything happens on device so uh unfortunately that means that open ey is not involved in this like open ey's mission is intelligence too cheap to meter and they're not doing the one thing that needs to happen for that because there's no business plan in monetizing an API for that but by definition none of this is apis I don't know I guess Johnny I and Sam Alman need to figure it out so they can do a their own for opening iPhone I don't know if you would you would buy an opening iPhone I mean I'm very locked into the iOS ecosystem but I will not be the first person to buy it because I don't want to be stuck with like the rabbit equivalent of iPhone but I think it makes a a lot of sense I want they're building a search engine now the next thing is the phone exactly so we'll see we'll see when it comes out weight list we we'll see yeah yeah we'll we'll review it all right so that was GBU Rich GBU 4 maybe we just want to run quickly to the Quality data um there's maybe there mostly drama in this section there's not as much as much research I think there's a lot of news going in the background so like the New York Times lawsuit is still ongoing you know it's just like we won't have specific things to update people on uh there there are specific deals that are happening all the time with stack Overflow making deals with everybody with like Shutterstock making deals with everybody it's just it's hard to make a single news item out of something that is just slowly cooking in the background mhm yeah the on the New York Times think open ai's strategy has been to make the New York Times prove that their content is actually any original or like actually interesting yeah so it's kind of like you know the IR robot meme it's like a can a robot create a beautiful new symphony and the robot is like can you um I think that's the that's what open ai's strategy is yeah I think that the danger with the lawsuit U because this lawsuit is very public cuz opening ey responded including with Ilia showing their emails with New York Times saying that hey we we were doing a deal you were like very close to a deal and then suddenly on the the eve of the deal you called it off uh I don't think New York Times has responded to that one but uh it's very very strange because the New York Times brand is like trying to be like you know they're supposed to be the top newspaper in the country if open I like just and this was my criticism of it at at the point in time like okay we'll just go to the next best paper The Washington Post the financial times all to work with us and then what does New York Times Happ yeah so you just lost out on like $100 million $200 million a year of of uh licensing deals just because he wanted to pick that war which ideologically I think they absolutely right to do that but um you know the other people The Verge did a very good interview with I think the Washington Post I'm going to get the the the the allet wrong the The Verge did an very good interview with a newspaper owner editor on why they did the deal with open I and I think that listening to them on like their thinking through like the the reasoning of like the pros and cons of picking a fight versus partnering I think it's very interesting yeah the I guess the winner in all this is RIT which is making over 200 million just in data licensing to open Ai and some of the other AI providers um I mean 200 million is like more than most AI startups are making so I think it was an IPO play cuz RIT conveniently did this deal before IPO right is it like a one time deal and then you know the stock language is on there I I don't know yeah no well there has done well I guess it's not gone down so in this market they're up 25% I think since IPO but I saw the FTC had open a inquiry into it just to like investigate so I'm curious what the antirust regulations are going to be like when it comes to data obviously Acquisitions are blocked to prevent kind of like stifling competition I wonder if for data it will be similar where hey you cannot actually get all of your data only behind $100 million plus contracts because otherwise you're stopping anyting new company U from building a competing product so yeah I that's a serious overreach of the state there that's how as a as a free market person I want to defend it's weird like I'm free market person and I'm a content creator right so I want to be paid for my content at the same time I I believe that you know people should be able to make their own decisions about uh all these deals but ugc is a weird thing cuz ugc is contributed by volunteers yeah and the other big news about Reddit is that apparently they have added to their robots.txt like only Google shall index us right because we did the deal with Google and that's obviously blocking openi from crawling them anthropic from crawling them you know perplexity from calling them perplexity maybe ignores all robot txt but that's a whole different other issue and then the other thing is I think this is big in the sort of Normie worlds um the actors you know Scarlet Johansson had a very very public Apple notes take down open the ey only Scarlet Johansson can do that to to Sam Alman and then you know I was very proud of my newsletter for for that day I called it Skyfall because the voice of the that voice of sky so I called the Skyfall and but it's true like you there's that that one she can win um and there very well established case law there and the YouTubers in the music industry the Raa like the most litigious uh section of the Creator economy um has gone after yudo and sunno you know Mikey from our our podcast with and uh it's unclear what will happen there but um it's it's going to be a very costly legal battle for sure yeah um I mean music industry and lawsuits name a more iconic duel you know so I think this to expected yeah I think last time we talked about this I I was uh pretty optimistic that this something like this would reach the Supreme Court and with the way that the this Supreme Court is is making rulings like we just need a judgment on whether or not training on on data is transformative use so I think it is literally we're using Transformers to to do transformative use so then it's open season for for AI to do it and comparatively the content creators and owners will lose out they just will yeah cuz right now we're paying their money out of fear of lawsuits if the Supreme Court rules that there are no lawsuits to be had then all that money disappears I think people are price creeping late in space and we're not getting a dime so that's what it is yeah uh no you can you can support with like a $8 a month subscription and that pays for a microphone and and St like that yeah it's you know it's not it's definitely not worth the amount of time we're putting into it but it's a labor of love yeah exactly synthetic data yeah yeah I I guess we talked about it a little bit before with llama um but there was also the alpha proof thing yes just before I came here I was working on that yeah Google trained be almost got gold medal I forget what the yes they're one point short of the gold one point short of the gold medal it's it's a remarkably I wish they had more questions so the the the so the International math Olympiad has six questions and each each question has seven points every single question that the Alpa proof model tried it got full marks on it just failed on two and then the cut off was was like sadly one one point higher than that but still like it was a it's a very big like a lot of people have been looking at IMO as like the next gold prize grand prize in terms of what AI can achieve and betting markets and El Isa owski has has has updated and saying like yeah like we're we're pretty close like we we basically have reached it near gold medal status we definitely reached silver and bronze status and uh we'll probably reach gold medal next year right uh which is good there's also related work from hugging face on the numina math competition so this is on the AI mathematical Olympiad which is an easier version of the the human uh math Olympiad this just all like related research work on search and verifier model assisted exploration of of of mathematical problems so that's super positive I don't really know much else beyond that like it's it's always hard to cover this kind of news CU it's not super practical and it also doesn't generalize so one thing that people are talking about is this a concept of jagged intelligence CU at the same time we're having this discussion about being superhuman you know one of the um IMO questions were solved in 19 seconds after we gave the question to uh Alpha proof at the same time language models cannot determine if 9.9 is smaller than or bigger than 9.11 and part of that is 9.11 is an inside job but it's a funny that's someone else's joke I don't know I really like that joke but it's it's jaged intelligence it is a failure to generalize because of tokenization or because of whatever and what we need is general intelligence we've always been able to train dedicated special models to to win prizes and do stunts but the grand prize is general intelligence that same model does everything is it going to work that way I don't know I think like if you look back a year and a half ago and you would say can one model get to general intelligence most people will be like yeah we can keep scaling I think now it's like is it going to be more of a mix of models you know like can you actually do one model that does it all yeah absolutely I think GPT 5 or Gemini 3 whatever uh would be much more capable at this kind of stuff while it also serves our needs uh with like with everyday things it might be a it might be completely uneconomical like why would you use it giant model to to do normal stuff but it is just a demonstration of proof that we can build super intelligence for sure and and then you know everything else follows from there but right now we're just pursuing super intelligence I always think about this uh you know just reflecting on the GPU Rich por stuff and then now this Alpha geometry stuff I used to say you pursue capability first then you efficient make it more efficient you make Frontier Model then you distill it down to the a7b 70b which is what Lama 3 did and by the way also opening I did it with GPC 40 and then distilled down to 40 mini and then Claude also did it with Opus and then with 35 on it right that suitable recipe I in fact I call it part of the deployment strategy of models you train a base layer you train a large one and then you distill it down you add structured output generation two calling and all that you add the long context you add like this this standard stack of stuff in post training that is growing and growing to the point where now opening has opened a team for Mid training that happens before post training I think like one thing that I realized from this Alpha geometry thing is before you have capability and you have efficiency there's an in between layer of generalization that you need to accomplish you need to do capability in one domain you need to generalize it then you need to efficien size it then you have good models that makes sense I I think like maybe the question is how many things can you make it better for before generalizing you know yeah I don't have a good intuition prod that we'll talk about that in the next uh next thing yeah so we can skip NE neotron is worth looking at if you're interested in synthetic data uh multimodal labeling I I think has happened a lot maybe we jump to multim model now mhm yeah we got a bunch of news well the first news is that 40 voice is still not out even though the the demo was cre I think it they're starting to roll out the bet on the next week so uh I am subscribing I subscrib back to chat GPT plus you give in I gave in CU they're rolling it out next week so you better be on the the cut off or you're not going to get it ni bait I know I said this I said when I talked about uning of trbt is it's basically because they had nothing to offer people that's why people are unsubscribing cuz like why keep paying $20 a month for this right but now they have proprietary models oh yeah I'm back in right like we're so back we're so bad we're so bad I would pay 200 for the Scarlet Johansson voice but you know they'll probably get sued for that um but but yeah the voice voice is coming we had a demo at the Worlds Fair that that was uh that was I think the the second public demo uh Ro Roman I have to really give him a shout out for that we had a few people drop out last minute and he was he say he rescued the conference and and and worked really hard like you know I think off the scenes I think something that people don't understand is opening ey puts a lot of effort into their presentations and if it's not ready they won't launch it like he he was ready to call it off if if we didn't make the AV work for him and I think they they care about their presentation and how they launch things to people those minor polish details really matter um just for the record for for people who don't understand what happened was uh you can first of all you can go see just look for like the GPC 40 talk at the wsare but second of all because it was presented live at a conference with large speakers blaring next to you and it is a real-time voice thing so it's it's listening to his own voice and he needs to distinguish between his own voice and between the the human voice and he needs to ignore his own voice so we had opening Engineers tune that for our stage to make this thing happen which is absurd it was so funny but also like you know shout out to them for for doing that for us and for the community right uh because I think people wanted an update on voice yeah they definitely do care about demos not much to add there yeah lry voice something that maybe is buried among all the Lama 3 news is that Lama 3 is supposed to be a multimotor model it was delayed thanks to the European Union apparently I'm not sure what the the whole story there is I didn't really read that much about it it is coming you llama 3 will be multimodal it uses adap CHS rather than being natively multimodal but I think that it's interesting to see the state of of meta AI research come together because there was this independent threads of voice box and seamless communication these are all projects that meta AI has launched that basically didn't really go anywhere because they were all one-offs but now all that research is being pulled in into llama like llama's just subsuming all of fair all of meta AI into this thing and and yeah you can see uh voice box mentioned in llama 3 voice adapter I was kind of bearish on conformers because I looked at the state of existing conformer research in ICM clear and at new rips and they were far far far behind whisper uh mostly because of scale like that the sheer amount of resources that they're dedicated but meta is is approaching there it's I think it's um they had 230 hours 230,000 hours of speech recordings I think whisper something like 600,000 So Meta just needs to 3x the budget on this thing and they'll do it and we'll have open source voice yeah and then we can hopefully find tune on our voice and then we just need to write this episode instead of actually recording it I should also shout out the other thing from meta which is a very very big deal which is chameleon which is a natively early Fusion Vision and language model so most things are late Fusion basically like you freeze an existing language model you freeze an existing Vision Transformer then you kind of fuse them with a thin adapter layer that that is what uh Lama 3 is also doing but chameleon is slightly different chameleon is interleaving in the same way that ID fix U the sort of um data set was was doing interleaving natively for image generation and uh vision and and and text understanding and I think like once that is better understood that it's going to be better that that is the more deep learning pilled version of this uh the more GPU Rich version of of doing all this I asked eay this question about comedian in his in his episode he did not confirm or deny but I think he uh he would agree that that is the right way to do multimodality and now that we have we're proving out that multimodality is valuable to people basically all this half ass measures around adapters is going to flip to natively multimodal to me that's what GPC 40 represents that is the train from scratch y fully omn modal model with which is early Fusion so if you want to read that you should read the comedian paper basically that's what is my is my whole point and there was some of the chameleon drama because the open model doesn't have image generation yeah and then there were fine tuning recipe and then the leads were like no do not follow these instructions to find T generation that's it's really funny I don't know what the the I I okay so yeah whenever image generation is concerned obviously because the Gemini issue you know it's it's very tricky for for large companies to to release that but they can remove it say that they remove it point out exactly where they remove it and let the open source Community put it back in so the last piece I had which I kind of deleted was uh just a special mention honorable mention of Gemma again with P Gemma which is one of the smaller releases from Google IO I think you went right so pal Gemma was was mentioned in there uh I don't know one yeah yeah one of the work very small release um but uh kopali Jemma now is being talked a lot about as a as a late Fusion model for extracting structured text out of PDFs very very important for business work work workhorses yes uh so apparently it is doing better than Amazon textract and all the other state-ofthe-art and it's just it's a tiny tiny model that does this and it's really interesting it's a combination of the Omar kab's coar sort of retrieval approach on top of a vision model which I was severely underestimating polya when it came out but like it's continues to come up like there's a lot of Trends and again this is making a lot of progress here just in terms of their their applications in in real world use cases like these are small model but they're very very capable and they're very good basis to build things like KAG J yeah no Google has been it's been doing great I think maybe a lot of people initially wrote them off but between um you know some of the Gemini Nel stuff like uh Gemma too P Gemma what talk about some of the KV cach and context caching yeah yeah that's so there's a a lot to like in our friend Logan is over there now so he's excited about everything they got going on so I think there's a little bit of a fight between AI studio and vertex and what Logan represents is is so he's moved from Deval to pm and he was PM for the Gemma 2 launch vertex has this reputation of being extremely hard to use it's one one reason why gcp has kind of fallen behind a little bit and and so AI Studio represents like the developer friendly version of this like the the the netfi or versel to to the AWS right and I think it's Google's chance to reinvent itself for this audience for the AI engineer audience that doesn't want like five levels of off IDs and org IDs and policy permissions just to get something going true true um yeah we want to jump into rag Ops Wars I what to say here I I think that what rag Ops Wars are to me like the the tooling arounds the ecosystem and I might need to actually rename this war war renaming alert what are we calling it uh lmos atos because it it it used to be when you when the only job for AIS to do was was chat Bots mhm then rag matters then Ops matters but now we need AI to also write code we also need AIS to work with other agents right that's not reflected in any any of the other Wars so I think that just the the whole point is what does an llm plug into with the the broader ecosystem to be more capable than an LM can be on its own y i just announced it but this is something I've been thinking about a lot I it's a blog post I've been working on uh basically basically my tip to other people is if you want to see where things are going you go open up the chat GPT GPT Creator every single button on the GPT Creator is a potential startup M um EXA uh is is for search the knowledge rag thing is for rag the Cod inter yeah congrats is that announced I don't know if well it's announced now by the time this goes out it'll be briefly what is EB uh so e2b is basically a code interpreter SDK as a service so you can add code interpreter to any model uh they partner with mrra to add that in then this open source CLA artifacts clone using ITB it's a I mean the amount of like traction that they've been getting in open source has been amazing I think they went in like four months from like 10K to a million containers spun up um on the cloud so I mean you told me this maybe like nine months ago 12 months ago something like that uh you were like uh what you literally just said every CH GPT plug in can be can be a business startup um and I think now it's more clear than ever that and the chap Bots are just kind of like the Band-Aid solution you know before we build more more comprehensive systems and um yeah EXA just raac a series a um from light speed so I tried to get you in on that one as well yeah yeah I'm trying to be a scout man I don't know um so yeah this is a giving as a VC early stage VC like giving capabilities to the models is like way more important than the actual LL Ops you know the observability and like all these things like those are nice but like that way you build real value for a lot of the customers is like how can this model do more than just chat with me so running code doing analysis doing web search H I might disagree with you I think it's comp they're all valuable they're all valuable they're all valuable so I want disagree with you just on like it I find Ops my number one problem right now building Small Talk building an building anything I do and I I don't think I'm happy with all the op Solutions I've explored there are some 80 something op startups right I nearly you know started one of them but we'll briefly talk about this op thing then we'll go back to go back to rag the central way I I explain this thing to people is that all the model Labs view their job as stopping by serving you their model over an API right that is unfortunately not everything that you need in order to productionize this API so obviously there's all these startups they're like we are Ops guys we've we've done this for 30 years we will now is for AI and 80 of them show up and they all raise money and the the question is like what do you actually need as as like sort of AI native Ops layer versus what do you just plug into Data dong right uh I don't know if you have you have dealt with that because I'm not like a super Ops person but I appreciate the importance of this thing I think there's three broad categories which is Frameworks gateways and monitoring or tracing we've talked to like I interviewed human human Loop in London in you've talked to a fair share of them I've talked a fair share of them so the Frameworks would be uh honestly I won't name the startup but basically what this Frame company was doing was charging me $49 a month to store my prompt template and every time I make an inference it would FST string call the prom template on some variables that I Supply and it's charging $49 a month for unlimited storage of that it's absurd but like people want prompt management tools they want to interoperate between pm and developer there's some value there I don't know what the r price is there some price I was at I'm sure I can share this I was at the Grab office um and they also treat prompts as code but they build their own thing to I promps into my code base as a developer right but maybe do do you want it outside I it's like how do you well you can have it in the code base but like what's like the prompt file what's like you know it's not just a string and model and config exactly how do you pass these things uh I think like the problem with building Frameworks is like Frameworks generalize things that we know work and like right now we don't really know what works yeah but some people have to try you know the whole point of early stage is you try it before you know it works yeah but I think like the the past if you see the most successful open source Frameworks that became successful businesses are Frameworks that were built inside companies and then were kind of spun out as project so uh I think very vertical vertical pill instead of horizontal pill I mean we try to be horizontal pill right and it's like where are all the horizontal startups so there are a lot of them they're just not that they're not going to win by by by themselves by themselves I think some of them will win by sh excellent execution and and then but like the market won't pull them they will have to pull the market well but that's the thing it's like you know take like Julius right it's like hey what are you guys doing Julius like the same as code interpreter and yet they're pretty successful a lot of people use it because they're like solving a problem and then they're more dedicated to it than code interpreter exactly so it's like I think just take it more seriously then chbt you you win I think people underestimate how important it is to be very good at doing something versus trying to serve everybody uh with some of these things so yeah I think that's a learning that a lot of Founders are having yes okay so Dr out the Ops worlds so it's a it's a three Circle vend diagram right it's Frameworks it's gateways so the only job of Gateway is to just be one endpoint that uh proxies all the other endpoints right and and it it normalizes the apis mostly to Open the Eyes API just because most people started opening Ai and then lastly it's monitoring and tracing right so logging those things understanding the latency like P99 or whatever and like the number of steps that you take so Lang Smith is obviously very very early on to this stuff uh but so it's l fuse so is my God like there's so many like I'm sure like data dog has some like uh biases um you know there it's very hard to for me to to choose between all those things so I as a as a small team developer wants one tool that does all these things and my Discovery has been that there's so much specialization here there like everyone is like oh yeah we we do this but we don't do that for the other stuff we recommend these these two other friends of ours and I'm like why am I integrating four tools when I just one that they all they all the same thing mhm um that that is my current frustration the obvious frustration solution is I build my own right which is you know we had 14 standards now we have 15 so it's just a very messy place to be in I I wish there was a better solution to recommend to people because right now I cannot clearly recommend things yeah I think the biggest change in this market is like latency is actually not that important anymore like we lived in the past 10 years in a world where like 10 15 20 milliseconds made a big difference I think today people will be happy to trade 50 milliseconds to get higher quality output from model so but still all the traing is all like how long did it take like what's the thing instead of saying is this quality good for this output like should you use another model like we're just kind of taking what we did with cloud and putting it in llms instead of saying what actually matters when it comes to llms what you should actually monitor like I don't really care what my P99 is if the model is crap right it's like also like I don't own most of the models so it's like this is the gb4 API performance it's like okay I'm I going M it's like I can't do anything about it you know so I think that's maybe why the value is not there like you know I'm I supposed to pay 100K a year like I pay to data dog or whatever to tell me for have you tell me that gb4 is slow it's like you know and just not uh I don't know I agree it's it's challenging there um okay so the last piece I I'll mention is briefly mlops is still real and I think llm Ops or whatever you call this a engineer Ops the Ops ler on top of the LM layer uh might follow the same Evolution path as the mlops layer and so the most impressive thing I've seen from the mlops layer is from Apple when they announc Apple intelligence they also announced Taria which is their internal MLS tool which where you can profile the performance of each layer of a transformer and you can ab test like a 100 different variations of different quantizations and stuff and pick the best performance and I could see a straight line from there to like okay I want this but for my my AI engineering Ops like uh I want this level of clarity on like what I do and uh there's a lot of internal engineering within within these big companies who take their ml training very seriously and I see that also happening for AI engineering as well let's Bally talk about Rag and context caching maybe unless you have other like lmos stuff that you're excited about um a little mo stuff I'm excited about no I think I think that's really a lot of it is like move Beyond being observability or like help for like making the prom call and like actually being an llm OS you know I think today it's mostly like llm rails you know like there's no OS but I think like actually helping people build thing that's why you know if you look at x2b it's like that's the OS you know those are kind of like the OS Primitives that you need around it yeah okay so I'll mention a couple things then one layer I'm I've been excited about publicly by haven't talked about it on this podcast is memory databases memory layers of on top of vector databases the Vogue thing of last year was Vector databases right everybody had a vector database uh company and I think the Insight is that Vector databas is are too low level like they're they're not very useful out of the box they do cosign similar similarity matching and retrieval and that's about it we briefly maybe mention here bm42 which uh was this whole debate between vesa and who else uh quadrants cudr and um I think a couple other companies also chipped in but it was mainly a very very public and ugly Twitter battle between uh benchmarking for for databases and the history of benchmarking for databases goes as far back as Larry Ellison and Oracle and and all that it's just very cute to see it happening in the vector database space uh nothing some things don't change but on top of that like I think one of the reasons I put Vector databases inside of these wars is in order to grow the vector data bases have to become more Frameworks in order to grow the Ops companies have to become more Frameworks right and then the framework companies have to become Ops companies which is what L chain is so one element of the vector databas is growing I've been looking for what the next direction of vector databas is growing is is memory long conversation memory I have on me this um uh B which is one of the personal AI wearables I'm also getting the Limitless uh personal a wearable which just like I I just wanted to record my whole conversation and just repeat back to me or let me let me find augment my memory as a as a I'm sure character AI has some version of this like everyone has conversation memory that is different from factual memory and right now Vector database is very oriented to us factual memory document retrieval knowledge based retrieval but it's not the same thing as conversation retrieval where I need to know what I've said to you what I said to you yesterday what I said to you a year ago or 3 years ago and it's there's a different nature of retrieval right um so there's a at at the conference that we ran graph rag was a was a lot of uh Focus for people the marriage and Doge GRS and rag I think that this is commonly a trap in ml that people are like they discover that graphs are a thing for the first time they're like oh yeah everything's a graph like the future is graphs and then nothing happens very very common this happened like through four times in in the industry's past as well but maybe this time is different maybe unless unless unless so this the fun this is why I'm not an investor like you have to get the time that the this time is different because no ideas are really truly new MH but some sometimes this time is different maybe and so memory databases are one form of that where like they're focused on the problem of long form memory for uh for agents for assistants for chatbots and all that I I definitely see that coming there were some funding rounds that I can't really talk about in this sector and I've seen that happen a lot um yeah I have one more K in LM West but any comments on yeah no I think that makes sense to me that moving away from just semantic similarity I think is the most important because people use the same word with very different meanings especially when talking you know when writing is different but yeah yeah the other direction that Vector databases gone into which um Lance DB presented at my conference was multimodel so L character uses lb for multimodal embeddings that's just a minor difference I don't think that's like a Quantum Leap in terms of what the vector database does for you the other thing that I see in LM world is is mostly um the evolution of like uh just the the ecosystem of Agents right the the the agents talking to other agents and coordinating with other agents so I interview Graham nuig at at ice I clear and he since announced that they are pivoting open open Deon or broadening open Devon into all hands AI I'm not sure about that name but uh it is the it is one of the three llm OS startups that got funded in the past two months uh that I know about and maybe you know more they they're all building like this ecosystem of Agents talk working with other agents and all this all this tooling U for for agents to me makes more sense it is probably the biggest thing I missed in in doing the four Wars the need for startups to build this ecosystem thing up right so the big categories have been taken search done code interpreter done there's long tale with others right so memory is emerging then there's like other stuff uh and so they're they're focusing on that like to to me uh browser is slightly different from search and browser base is is another company invested in that is focused on that but they're not the only one in that category by by by any means I used to tell people go to the Devon demo and look at the four things that they offer and say each of each of those things do a startup Devon since then they spoke at the conference as well Scott uh was super nice to me and then uh actually gave me some personal time as well they have an updated chart of their plans look at their plans they have like 16 things each of those things is a potential startup now mhm and that is the lmos everyone's buildings where that direction because they need it to do what they need to do as as an agent if you believe in the agent's future you need all these things yeah you think the agent OS is its own company do you think it's a open standard do you think I would love it to be open standard the reality is that people want to own that standard so we have we actually wound down the a engineer foundation with with the with the first Pro was the agent protocol which e2b actually donated to the foundation because no one's interested everyone wants to be VC back when they want to own it right so there's just it's too early to be open source this the people will keep this proprietary and more power to them they need to make it work they need to make Revenue before all the other stuff can happen yeah I'm really curious you know we're investors in a bunch of agent companies none of them really care about how to communicate with other agents they're so focused internally you know but I think in the future you know talk about this you're talking about agent to other external agents yeah so I think I'm not talking about that yeah I wonder when like because that's where the future is going right so today it's like intra agent I see connectivity you know at some point it's like well it's not like somebody I'm selling into a company and the company already uses Agent X for that job I need to talk to that agent you know but I think nobody really cares about that today so I think that's usually yeah so uh I think that that layer right now is open API just give me a restful protocol I can I can interoperate with that restful protocol only does request response so then the next layer is something I have worked on which is long running request response which is workflows which is what temporal was supposed to do before let's just say management issues yeah but like you know RPC or some you know I think the the dream is uh and this is one of the the the my problems with the lmos concept is that do we really need to rewrite every single thing for AI native use cases should the AI just use the these things these tools the same way as humans use them reality is for now yes they they need specialized apis in in the distant future when these things cost nothing then they can use it the same way as humans does but right now they need specialized interfaces the layer between agents ideally should just be English you know like the same way that we we talk but like it English is too under specified unstructured to to make that happen so so it's interesting because we talked to each other in English but then we both use tools to do things to then get the response back for those people who want to dive in a little bit more I think autogen I I would definitely recommend looking at that crew AI there are established Frameworks now that are working on interagent communication layers to coordinate them and not necessarily externally from company to company just internally as well between if you have multiple agents farming out work to do do different things you're going to need this anyway and uh I don't think it's that hard it's it's they are using English they're using some some mix of structured output and yeah if you have a better idea than that let us know yeah we're listening so that's the four Wars discussion I think I want to leave some discussion time open for miscellaneous trends that are happening in the industry that don't exactly fit in the four Wars or or are layer above the four Wars so the first one to me is just this trend of Open Source obviously this overlaps a lot with the GPU Port thing but I want to really call out this depreciation thing that I've been working on like I I I do think it's probably one of the the bigger thesis that I've come that I've had in the past month which is that we now have a rough idea of the deprecation schedule of uh this this little of model spent and yeah basically drew a chart I'll link it in the show notes but I drew a chart of the price efficiency Frontier of as of March April 2024 and then I had listed all the models that list that sit within that Frontier Haiku was was the best cost per intelligence at that point in time and then I did the same chart in July two days ago and the whole thing has moved and mraw was like deprecating their old models that they used to be in the old Frontier it is so shocking how predictive and tight this band is very very tight band and the whole industry is moving the same way and it's roughly one order of magnitude drop in cost for the same level of intelligence every 4 months my previous number for this was one order manage to Dro in cost every 12 months M but the timeline is accelerated because gbt 3 took about a year to to drop order magnitude but now GPT 4 it's really crazy I don't know what to say about that but I you think gbt next and Cloud 4 push it back down because they're coming out with higher intelligence higher cost or is it maybe like the timeline is going down because New Frontier models are not really coming out at the same rate interesting I don't know that's a that's a really good question wow I'm stumped I don't have you're like wow you got a good question I don't have I don't I don't I don't have an answer no I mean you have question but like I thought I had solved this and then now you came along with the first response is something I haven't thought about uh yeah yeah so there's there's two directions here right when when the cost of Frontier of models are going up potentially like SB 1047 is going to make it illegal to train even larger models for it where I I I think the opposition is increase enough that it's not going to be a real real concern for people but I think every lab basically needs a small medium large play and like we said in the the sort of model deployment framework first you choose you pursue capability then you pursue generalization they pursue efficiency and and what we're talking about here is is efficiency like that now we care about efficiency the this definitely one the emerg stories of the year that has happened is efficiency matters for 40 40 mini and 35 Sonet in a way that in January nobody was talking about mhm and that's great yeah regardless of GPT next and Cloud or whatever or Gemini 2 we will still have efficiency Frontiers to to pursue and it seems like doing the higher capable thing creates the synthetic data for us to do the efficient efficient thing and that means lifting up the like I had this difference chart between llama 3.0 8B llama 3.0 7tb versus their 3.1 differences and the 8B had the most uh uplift across all the benchmarks right it makes sense you're training from the 405b you're distilling from there and and it's going to have the the biggest lift up so the best way to train more efficient models is to train the large model right yeah yeah and then you can distill down to the rest it so this is fascinating on from the investor point of view you're like okay you're worried about pix and shovels you're worried about investing in Foundation model Labs um and that's a that's a matter of opinion I do think that s Foundation model labs are worth investing in because they they do pay back very quickly I think for engineers the question is what do you do when you know that your basee cost is going down in order of magnitude every 4 months how do you how do you make those assumptions and I don't know the answer to that I'm just posing the question I'm calling attention to it because I think that cognition burning like rumors is I don't know nothing from Scott I haven't talked to him at all about this even though he's he's very friendly but they did that they got the immedia attention and now the the cost of intelligence is going down and it will be economically viable tomorrow in the meantime they have a crap ton of value uh from from user data and a c on of value from a media exposure I think that the correct stun to pull is to pull is like make economically nonviable startups now and then wait yeah but honestly basically I'm basically advocating for people to burn VC money yeah no they they can burn my money all they want if they're building something useful I think the big problem not a problem but the price of the model comes out and then people build on it and then there's really no the model providers don't really have a lot of Leverage on like keeping the price High you know they just have to bring it down because the people Downstream of them are not making that much money with them you know and I wonder what's going to be the model where it's like this model is so good I'm not putting the price down you know like if gbd 40 was like amazing and was actually solving a lot of like creating a lot of value Downstream people would be happy to pay I think people today are not that happy with the models you know like they're good but like I'm not paying that much because I'm not really getting that much out of it like we have um this AI Center of Excellence with a lot of the Fortune 500 groups and there are people saving 10 20 million a year like with these models doing boring stuff you know like document translation things like that but nobody's making 100 million nobody's making 150 million so like the prices just have to go down to match but maybe that will change yeah at some point I always mentioned temperature two use cases right like where those are temperature zero use cases where you need Precision you need creativity what are the cases where Hallucination is a feature not a bug right so we the first podcast to interview web Sim and pretty I'm still pretty positive about the generative part of AI like we we took generative Ai and we used it to do rag you know like we have an infinite creativity engine let's go let's go do do more of that uh you so we'll hopefully do more episodes there uh you have some stuff on agents you want to yeah no I think this is something that we talked a lot about and um you know we wrote this post months and months ago about a shifting from software to service to Services of software and that's only more true now I think like most companies that are buying it tooling they want the AI to do some sort of labor for them and that's why the pick and shovels kind of thisin maybe comes from a little bit most companies do not want to buy tools to build AI they want the AI and they also do not want to pay a lot of money for something that makes employees more productive because the productivity gains are not acing to the companies they're just acing to the employees you know people work less they have longer lunch breaks because they get things done faster but most companies are not making a lot more money by making employees productive that's not true for startups so if you look at most startups today in AI like the much smaller teams compared to before versus agents we have companies like you know Bri wve which we had on the podcast you're selling labor which is something that people are used to paying on a certain pay scale so when you're doing that you know if you spry wve they don't have a public but like they charge a lot of money more than you would expect because hedge funds and like Investment Banking investment advisers they're used to pay a lot of money for research it's like the labor they don't even care that you sayi they just want labor to be done I'll mention one one push back but as a hitch fund we used to pay for analyst research out of our brokerage cost M and not read them and that that to me that's that's my risk of of bright wave but but you know no but I I think that as a consumer of research like if we want to go down the the rabbit hole there's a lot of pressure on funds for like a opx efficiency so there's not really capture uh researchers anymore at most funds and like even the sside research is like not that good so taking them from inhouse to external thing yeah makes sense um so yeah with you know we have drops on the the security analysis same people are used to paying for managed security or like Outsource stock analysts they don't want to buy a AI tool to make the security team more productive so okay and what specifically does drops on do they do uh sock analysis so not sock like the compliance but it's like when you have security alerts how do you investigate them so large Enterprises they get like thousands of fishing email and then they forward them to it and it's it or security person the the tiered zero has to go in and say that's a fishing email that isn't that isn't so they have an agent that does that so the cost to do like for a human to do the analysis at the rate that they get paid that's like $35 per alert drops on is like $6 per alert so it's it's a very basic economic analysis for the company whether or not they want to buy it it's not about is my analyst going to have more free time like is it more productive so selling the labor is like the story of the market right now my of this is I should start a consulting services today mhm and then slowly automate myself my my employees out of a job right mhm is that fundable is that fundable that's a good question I think whether or not depends how big you want it to make this is a Services Company basically yeah that's I mean that's what I know now now it's maybe not as good of an example but cross strike started as a security research um yeah I mean it's still one of the most successful companies of all time yeah interesting model I'm always checking my my he there um anything else on the the agent uh side of things no that's really something that people should spend more time on it's like what's the end labor that I'm building because you know sometimes when you're being too generic and you want to help people build things like adapt like ad you know David was on the podcast and he said they were sold out of things but they're kind of like and then he sold out himself it's like they're working they're working with each company and the company has to invest the time to build with them need more hands off exactly um so and that's more verticalized how I'll shout out here Jason Lou he was also on a podcast and spoke at the conference uh he has this idea like it's it's reports not rag you want to you want things to produce reports because reports can actually get consumed uh rag is still too much work still too much chat Bing I'll briefly mention that new benchmarks I'm thinking about um I think you need to have um every everyone in studying AI research understanding the progress of AI and and Foundation models needs to have in mind what is next after MML I have 10 proposals most of them half of them come from the huging face episode so everyone's loving Clementine I want her back on and she she she was amazing and very very charismatic even though she made us take down the YouTube but uh musr for multi reasoning math for math I EV for instruction following big bench hard and it uh code we're now getting to the area that the hugging face leaderboard does not have and I'm I'm considering making my own because I I I care about this so much so mbpp is the current one that is post-human eval because human eval is Wily known to be saturated and pycode is like the the newest one that I would Point people to context utilization we had mark from gradient on talk about ruler but also zero sces in infinite bench with the two that Lama 3 used use instead of ruler but basically something that's a little bit more rigorous than needle in a hay stack U that is something that that people need then you have function calling here I think gorilla API Bank Nexus pretty pretty consensus I I've got nothing there apart from yeah like all models need need something like this Vision now is like multi like multimodal the vision is the most important um I think like Vib Val is actually the the State ofthe art here I you know open to open to being corrected and then multilinguality so basically like these are the 10 directions right uh post MML here are the frontier capabilities if you're developing models or if you're if you're encountering a new model evaluate them on all these elements and then you have a good sense of how state of the art they are and what you need them for in terms of applying them to your use case so I just want to get that out there yep and we had the rcgi thing how do you think about benchmarking for you know everyday thing or like benchmarking for something that is maybe like a hardto reach goal yeah this has been a debate uh for uh that's obviously very important and probably more important for product usage right here I'm talking about benchmarking for General model evals and then there's a there's a Schism in the AI engineering community or criticism of engineering community they did not care about enough about product EVS so ham Hussein uh LED that and I I had a bit of disagreement with him but I I acknowledge that I I think that is important there was an oversight in my original AI engineer post uh so the job of the AI engineer is to produce products specific evals for your use case and there's no way that these General academic benchmarks are going to do that because they don't know your use case it's not important they will correlate with your use case and that is a good sign right these are very very rigorous and thought through so you want to look for correlates then you want to look for specifics and that's something that only you can do so yeah rgi Will correlate with IQ it's an IQ test right how well does IQ test correlate of job performance 5% 10% not nothing but not everything and so it's important anything else super intelligence we we can you know we we try not to talk about safety uh my my favorite safety joke from our dinner is that you know if you're worried about agents taking over the worlds and you need a button to take them down just install crowd strike on all every agent and you have a button that has just been proved at the largest scale in the world to disable all agents right so that save super intelligence you should just install cross trk that's what should do that's funny except for the CR strike people awesome man this was great I'm glad we did it I'm sure we'll do it more regular now out of Visa jail yeah yeah I think uh you know a news is surprisingly helpful for doing this like uh yeah yeah I I I had no idea when I started I I just I I needed a thing to summarize discords but now it's becoming a proper Media company like yeah thousand people sign up every month uh it's growing cool thank you all for listening yeah see you next time right [Music]