AI Engineer Summit 2023 — DAY 1 Livestream

Key Takeaways

feeling chasing the know stop and I won't let it [Music] go let it like Fe up we all true belever we're looking on the world I can see on the horizon all our we can [Music] feel you true belever looking on the world I can see on the horiz all we [Music] can ladies and gentlemen the opening kyot presentations will commence in the ballroom starting in 15 minutes please make your way to the ballroom and find your seats thank you [Music] [Music] [Music] Tak one more breath beside [Music] you so I could find strength to divide us give it we got it I know we did the best we could if I could go back UND do the mess I would memorize your face before I go but this is how we go got to give it up sometimes as go KN when to kill your PR there's no one to blame nothing really stays the same this is we go sometimes we hold on to let [Music] go we hold on to let go there is nothing lost between [Music] us and I know you have your reasons some days I'm a mess but I know there's a row over all the past your head on my shoulder but I know we better on oh but this is how we go got to give it up sometimes it's go knowing when to kill you Pride there's no one to blame nothing really stays the same this is how we grow ladies and gentlemen the opening keynote presentations will commence in the ballroom starting in 10 minutes please make your way to the ballroom and find your seats [Music] [Music] here [Music] [Music] you you would you it would you get it would you get it [Music] you you [Music] you you [Music] you you you you what you what you would you would [Music] [Music] [Music] you [Music] me [Music] you you you you you you [Music] you you you what you [Music] [Music] we got an insomniac with eyes WI sh and we got everything we need and then a little too much I know that you're starving for something you can't touch but you be honest with me right now there's something in theur I can feel it coming up don't you want to feel it taking over your senses don't you ever Fe it Technologic fces baby escape with me I'll come sweep you up for your feet don't you want to feel it don't you want to don't you think there's something in my B that's weighing me down it's just [Music] the ladies and gentlemen the opening keynote presentations will commence in the ballroom starting in 5 minutes please make your way to the ballroom and find your seats thank you [Music] top and I'm ready to my breath and I'm to go I catch and I'm ready [Music] to we are a sto feeling you IGN do you ever St to feel it caught in the after glow I'll come back to your door know that [Music] [Applause] you all that I want you know we got it all and hold my breath and I'm ready to go I'm right in and I'm ready to go I found what I want and I on to and breath IAT [Music] and we light it up again the sky and our silou dancing on the pavement in a perfect Stone you those eyes again when I least expected said you're all that I want we know together we got it all holding my bre ready to go I'm falling right in and I'm ready to go I found what I want and I know that we're on top so and I'm breath to IAT and I'm ready [Music] to would better places if I find myself at your door gra the keys let's go I you show up my I follow you up let go holding my breath and I'm ready to I'm falling right in and I'm ready to go I found what I want and I know to so and I'm ready to my breath and to go I [Music] cat and I'm ready to hold breath ready to right I'm ready to and Bre ladies and gentlemen the opening keynote presentations are starting now please find your seats [Music] [Music] [Music] [Music] St [Music] [Music] [Music] [Music] [Music] oh [Music] [Music] [Music] [Music] [Music] w [Music] launch control we have a go Roger [Music] [Applause] [Music] ladies and Gentlemen please welcome to the stage the co-founder of the AI engineer Summit the managing partner of software 3.0 LLC and your host Benjamin [Music] dunfey Engineers Founders friends sponsors wow now I got my slides Engineers Founders sponsors Partners colleagues and Friends welcome to the AI engineer Summit 2023 I am yeah yeah I am deeply honored to have the opportunity to host you all at this event with my co-host swix and I'm especially delighted to kick off the presentation portion of the summit we've curated two days of stage content for you from some of the top Founders and engineers in this versioning new industry of AI Engineering in just a few moments swix is going to come on stage to help set the context for those talks but for my time on stage I really want to know who is here so Auto GPT is here a few months ago they were just an open source project mind you the fastest growing open source project in history but just an open source project now they're the presenting sponsor of the AI engineer Summit so needless to say we can expect some big announcements when Torin takes the stage in a few moments let's see who else is here super base super base is here the social media experts who make product launches fun and who make datab is both easy and trustworthy uh given that they're open source they're a diamond sponsored this year and we're honored that they broke their no event sponsorships policy for this event so Paul a uh let me know how that policy change works out for you fixie is here where's fixie I'm looking forward to asking their Booth sidekick if they can see what I'm wearing yet and if they can recommend any any updates to my wardrobe we'll call this sidekick Derek uh and if vision is not in the cards yet Matt and Ben uh I can make an intro to Logan Kilpatrick and perhaps we can get a preview of that Vision API Microsoft is here the company who helped to commercialize this Movement by pumping $13 billion into open Ai and many of you might be thinking well how can I get some of that gravy for my startup well head over to their booth in the Carmel room just past the elevators behind you and have a chat they have representatives from Microsoft for startups who can help to open those doors for you and if you have any questions for the co-pilot team well Microsoft's Booth is right next to GitHub oh and happy belated birthday Cloud Flair who is here Round of Applause they have uh Cloud Flair you are now 13 so that means you're going to be going through some changes but don't worry there's plenty of folks who will be delighted to talk to you about those changes at your booth just next to Microsoft and Carmel across the elevators alesio Finelli is here the VC half of the Laten Spate podcast but no less technical than swix perhaps after watching some of today's talks we can convince him to come back to the founder World a little bit easier right but this time alesio let's get you a soft start up and leave the hardware to The Experts some of you got that joke and who else is here well of course you you're one of 500 people who were selected to attend the inaugural AI engineer Summit so give yourselves a round of applause this means that you're not only an experienced software engineer but that you're actively experimenting with shipping to production or have founded an AI enhanced or AI native apps and companies so I want you to remember this when you're interacting with one another but we also want to recognize that many people who are not here today but are watching the live stream we thank you and appreciate you and we hope that you enjoy the content on the live stream and be sure to watch tomorrow's opening keynote address for some exciting announcements for 2024 which will expand our participation limits but for those of you who are here we hope that you've been enjoying your conversations with other attendees so far one of the reasons that I really love conferences is that it brings together experts from around the world who are passionate about a singular Niche subject and so most of your conversations are already high signal but is there a way we can actually improve this many of you have downloaded our conference mobile app Network and have have been enjoying its use it includes the conference schedule a full list of all of our Expo Partners recommended restaurants bars and cafes within walking distance of the venue and other event information but in addition to these features we're pleased to announce AI enhanced matching using our generative matching algorithm which is a fancy way of saying we use llms to match you with the right people we can match you with people who help to solve your stated problems we all have problems we need to solve that's why in your network profile you can literally tell us what problem you're looking to have solved and our GMA will connect you with the right people we call this app Network you can download the app today at ai. engineer Network the matching is current Curr ly limited to in-person attendees of this Summit only but we have one more big announcement that we'd like to make we are pleased to announce that we are open- sourcing Network both the event app and the matching algorithm our goal is to provide a simple yet powerful mobile event experience for attendees at any event around the world and we hope that the matching algorithm can better assist to connect people at those events and we i' like to thank our lead engineer Simon stermer for the mobile app and swet Teller for ID generative matching algorithm and our infrastructure partners descope for o and superbase for database and PG Vector so go talk to them about PG Vector outside so a round of applause for all of our developers [Music] partners for this app so you can access the repo today at ai. engineer Network or github.com a. engineer Network so with that I'd love to welcome our first speaker to the stage he's the co-host of the Laten space podcast and the co-founder of this very Summit please join me and welcome to the stage [Applause] [Music] swix hello everyone hello helloo is this working no okay um it sounds like it's working okay um uh give it up for Ben this this conference would not be happening without him like [Applause] um um a few logistical things one I'm I'm carrying a magic Trackpad because everyone has clickers what if we had multiple Dimensions so we're going to experiment with this today and uh and two I'm also I'm using like AI like fancy knew everything right like so this is to um and we're going to go two dimensional with our uh slides as well so I'm here to talk about the AI engineer you're all here because you believe that there's some value to this idea um and then I just put like a ridiculous 1,000x on this um but I do think there is some meaning towards thinking about highers higher orders of magnitude towards raising your Ambitions and that's what I would like all of you to do today and to do with your friends back home so um and obviously a lot of AI generated art because I mean it's an AI conference we got to do we got to do that um first of all I want to congratulate you on being here um not just I'm not talking about here location wise physically I'm talking about here in terms of the point in time uh imagine if you were a mathematician when was the best time to be born uh I I will propose uh around about 600 AD this dude Brahma Gupta he invented zero pretty pretty pretty novel invention that took us only 4,000 years to do that um but there's a there's there's certain times where like if you're in that field you you have to be there that's the thing if you're alive during that time you have to be doing that thing uh physics when was the best time to be born there's the right answer 1905 1927 um and this conference kind of is um inspired by the salv conference um that's Albert Einstein Mar cury and a lot of people that you just saw in the opener movie um same thing if you if you made cars there's the right time 1900 to 1930 if you made personal Computing products 1980 20110 uh if you ever get this like if you have Millennial if you're very online you ever get these memes like you're born too late to explore the earth born too early to explore the stars um you're not too late we are here uh this is based on demographics and history the approximate timeline of all Humanity um we know that we're roughly about 73% of all concurrent intelligences if we don't expand our own intelligences or go to other planets um so my argument and my message to you today is that you are just in time and the timing is right to 1000x um I think a lot a lot of my technology and industrial organization thinking is informed by kada Perez one of the most influential thinkers on Tech revolutions so she wrote this book about the insulation and deployment periods of tech cycles and we're definitely going through one today uh a lot of you on your mind here you're I know you're here but also me you're back home thinking how much of this is a fat how much of this is web 3 again um and we've seen this over and over uh the the great historians greater than us have explored this um over the Industrial Revolution the age of Railways age of heavy engineering and steel oil uh and most recently Tech Revolution uh funny enough I I recently put a I put a put a it's it's they all roughly span between 50 to 70 years and like if you're around in that time that's the field to be pursuing um so when did the AI Revolution start um we're very lucky it's very hard historically to place a start point on something that changes human civilization we have a moment 2012 Alex we're roughly 10 years on um and we can put numbers to it right so the most of the time these these curves is of theoretical they're just kind of like like y- AIS is just um here we can actually just put the amount of compute we're put we using to training models there's a huge inflection that's Alex right on the the blue dot over there that's a huge inflection uh where we realized gradually realized it took too long to realize but scale is starting to work um and if you actually take this out um I a lot of people have been taking this out and I want you to take scaling seriously there's three reasons why six is a magic number um there's a very famous investor who I shall not name says imagine roughly six words of magnitude and moreal compute by the end of the decade and plann for that um so there there is more of this coming linear projection wise um and you can plan on a lot more investment in language models uh John carac says there's six key insights towards AGI and lastly uh George Hots has these these like really nice analogies gpc3 took about one person year of compute gp4 took about 100 person years to compute you stretched it out to GPT 10 the difference between GPT 4 and gpt1 uh again under the six-fold increments in gbt advancements and that would be more compute than the equivalent compute of every ever whoever lived um so just being in the right moment you will get to U live on top of these mega mega trends that is greater than any single one of us um and I think you're all here thinking about the AI engineer and I put it in a very very small sort of local context of hey what's the or chart and where do the ml engineer SE sit where the ml researchers sit where the software engineer sit and what's the Gap that's opening it's CI engineer um it's a very much of a demand and Supply argument um there's something like 100,000 card carrying data science machine learning engineers and GitHub claims to have 100 million uh registered developers uh I don't know how I don't know what the real number is you can debate 40 to 50 million to 100 million it's orders of magnitude more um so we are we think there's going to be much more AI Engineers than ml Engineers there's all these reasons why same reasons that I mentioned in the blog post that you've all read um and also why engineering and not just prompting is because um LMS themselves are not agis yet right like we actually have to coordinate them in in systems of software we have to write code around them and orchestrate them with code uh in order to do do something useful and we already know how to do code so I want to spread it out a little bit more I think that the the conversation on AI engineer um has like sort of a vague uh discrepancy and and I want to basically split it out into three areas of AI engineer software engineer enhanced by AI tooling like co-pilot software software engineer building AI products like mid Journey AI product that replaces human engineer potentially like autog gbt and maybe rap ghost Ghost Rider um so let's give these guys a name uh and in case you're wondering like enhanced by versus replaces I think about it vers in very much like the South driving car terms like level two level three uh there's a difference between whether humans in the loop where there or humans as to fall back so let's name it uh three major types of AI engineer the AI enhanced engineer for people who are enhanced by AI uh people who build AI products a products engineer and then the AI engineer agent who is not human um and naturally of course if you're interested in sort of progressing up the career ladder there's AI enhanc engineer then products engineer engineer agent um so this talk was really inspired by actually amjad who's speaking next uh where he did did a recent talk on uh with the HC podcast and Sam Alman who actually sees um thousand X engineers in open the eye every day um and it's really a set of stackable 10 by 10 by 10 improvements over the course of the next two days I think you'll be seeing a lot of the speakers will be working on different parts of this stack um so I really encourage you to think about where in where in your life this AI movement um can improve and uh increase your productivity um I'm very very honored to have uh drawn um from from you know all over the world um the leading Lights of the AI engineering movement um we are a very small room today um I do think we can 100 and 1,000 extra here um and it's not just about tools and speakers it's also about you so I highly encourage you to take part in all the opportunities that we have for you to mix and mingle with each other with the speakers um and with the sponsors as well so there's that the final word um I do want to offer you um is effectively what I think in terms of non technical terms the a the the 1000x engineer could offer um my favorite advice for what a tenic engineer could look like is a an engineer that teaches 10 other people what they know right um that's that's not a technical term but it is uh very useful um and there's all these scaling laws for networks which I which I really keep in mind so uh you can go from o of n to O N2 to O2 to the power n but really um what o n is is you attending all the talks and Lear consuming all the content and letting people in with your Pacman rule um o squ is helping others learn um my very first blog post was at exactly a conference like this where I was encouraged to write something and of course it was on machine learning uh and finally going home and then building your own uh networks uh of AI engineers and helping uh to grow networks of of of learning as well so I hope you take that with you uh in your AI engineer journey I hope that over the next few days you get a sense of what it's like to be at the start of an industry and I'm just glad to be here with you thanks so much all right excited to be here I agree with uh swix and Ben that it feels like it feels like a moment it feels like a historical moment here um my name is amjad I'm the co-founder of repet where we aspire to be the fastest way to get from an idea to a deployed software that you can scale so I'm going to take you back a little bit not like swix to the 6 600 AD but perhaps to the start of computing if my clicker works it does not work so next slide we're going to get AGI before we get uh good presentation software all right here we go all right so um very early computers the eniac was the first you're in complete uh you know programmable uh vum and machine computer the way you programmed it is like you literally punched cards um not physically but you had a machine that sort of punched these cars these are sort of binary code for the machine to interpret it was really hard there wasn't really a software industry because this was really difficult it automated some tasks that human computers did at the time but it wasn't this it didn't create the software industry yet but then we moved to text from from punchcards and um we had First Assembly and then we had compilers and higher level languages such as C and then someone invented JavaScript and it's all been downhill since then but Tex editors were really or like tax based programming was a minimum a 10ax Improvement if not not 100x Improvement in programming so we've had these moments where we've had orders of magnitude improvements in programming before and then you know the IDE became a thing because you know we had large scale software this is a screenshot from like 20 17 or 18 when we added LSP uh to every programming environment on repet so anyone with an account can get intelligence and we're really proud about that at the time we're burning a lot of CPU doing sort of inference and you know if you've run typescript server that's like a lot of ram but we were really proud that we're giving everyone in the world tools to create professional grade software about 3 four years ago we started kind of thinking about how AI could change software it actually started much sooner than that but with the with gpt2 you know you could sort of kind of you know give it some code and kind of complete part of it you're like okay this this thing is actually happening and we we better be part of it and so we started building and we built this uh product called uh Ghost Rider uh which does autocomplete chat and all sorts of things inside the IDE and in just those two years I mean the the pace of progress across the industry the tools basically AI you know was deployed um and a lot of different Engineers were using it the AI enhanced engineer as swix kind of called it everyone is sort of using these tools and so uh we have a world now where a lot of people are gaining huge amount of productivity Improvement I don't think we're at a mold magnitude Improvement yet we're probably in the 50 80 perhaps 100% Improvement for some people but we're still at the start of this and we think that's going to going to be 10x 100x perhaps a th thousand X over the next decade the problem however repas Mission has always been about access our mission is to uh empower the next billion developers and so we really didn't want to create this world where some people have access to Ghost Rider and other people don't have access to it and we started thinking about okay what is it if if you really take into heart everything that the AI engineer conference is about that we're at a moment where software is changing where AI is going to be part of the software stack then you have to really step back a little bit and try to rethink how programming changes so our view is these programming add-ons such as co-pilot and coding ghost rer and all these things we're giving them cute names we think that's not the way forward we think that AI needs to be really infused in every programming interaction that you have and it needs to be part of the default experience of repet and I'm sure other products in the future that's why we're announcing today that we're giving AI for our millions of users that are coding on repet and so we think this is going to be the the biggest deployment of um AI enhanced uh coding in the world uh we're going to be burning as much GPU as we're burning CPU so pray for us uh we have people all over the world coding on all sorts of devices we have people coding on Android phones uh and and they're all going to get AI now so they're all going to be AI enhanced Engineers but you know as T showed it's it's not just about AI enhanced engineering there's also product so AI being part of the software creation stack makes sense but AI part of the call stack is also where a lot of value is created so um uh so that's why we're also we have this new product called Model farm and model Farm basically gives gives you access to um uh models right into your IDE so all it takes is three lines of code to start doing inference we launched with Google Cloud uh llms but we're adding uh llama uh pretty soon we're adding uh stable diffusion and if you if you're an LM provider and want to work with us and provide this on our platform would love to talk to you but basically um everyone will get uh there's some free tier here everyone will get free access at least and till the end of the year uh to model farm so you can start doing inference and start building AI based products so uh next up I'm going to bring up uh my colleague the head of aiet Mel kasta to talk about how we train our own AI models and we have uh one more announcement for you coming [Music] [Applause] up [Music] do you need that click fine thank you all right hi everyone so today I'm going to be talking about how we're training llm for code at ret and I will explain why this weird title uh if you've been around Twitter I think a bit more than a month ago you might you must have read this study from semi analysis and their point was it's meaningless to work on small models train on you know limited amount of gpus and that came as a shock to us because we had a very good success story back in May where we started to train our models from scratch and then you know ham Jad and I and the team started to think how we really wasting our time here um I'm going to try to convince you it's actually is not the case so our code completion feature or RIT is powered by our own bespoke larger language model we training open source code both published on GitHub and also developed by the rapit user base it's a very low latency feature so we try to find a difference with spot compared to what you might be used with other plugins we try to keep our P95 latency below 250 milliseconds such as the developer experience is almost instantaneous you don't even have to think about it and the code is going to be completed for you at the model size that we were using we have been St of the art across the past few months and let's do a show hands who has heard about our V1 model back in May all right that feels good for a second I feel like an AI star uh jokes aside so we released rapid code V 13B uh back in may we got a lot of adoption a lot of love and it's a lot of contribution and that's one of the key reasons why we decided to give it back uh rapid history has been built on the on the shoulders of giants of all the people contributing to the open source space so we thought we should do exactly the same same year we should give back our model and today I'm going to be announcing rapid code V 1.5 3B so the evolution of the model that we we released back in May let's go in detail as amjad was saying so the next 10 minutes we're going to do a technical Deep dive and I'm going to tell you how we built it and why it's so powerful so first of all we follow a slightly different recipe compared to the last time if you recall back uh in May a Wei W was a llama style uh code model which means we follow a lot of the best recipes that meta pioneered uh now we went you know one level up and we are training up to 300 tokens per parameter so if you have been following a bit history of llms even in you know two years ago most of the models were under train pardon me for the for the word it's not exactly you know technically speaking it's not correct but the truth is you know uh mid 2022 the chinchila paper from De mind came out and it was like a bit warning for the old field basically what the paper tells us is that we are under training our models we should give them way more high quality data and in exchange we could train smaller models so in a sense we're amortizing training time for inference time spending more compute to train a smaller more powerful model and then at inference time the latency will be lower and that's the key inside that we're going to be carrying along you know this whole keynote today now conver differently from the V1 this time we also double the amount of high quality data so we train it up to 1 trillion tokens of code it's the data mixture is roughly 200 billion tokens across five EPO plus a linear cool down at the end that really allows us to squeeze the best possible performance for the model and Rapid code v1.5 this time supports 30 programming languages and we also added a mixture coming from Stock Exchange post that are oriented towards developers so questions about coding questions about software engineering and so forth so this is the basis of our data now let's go and take a look inside at the data set that we used so we started from the stack which is an initiative led by big code it's a group you know under the huging face umbrella uh very uh grateful about the work that these people have been doing basically they have built a big pipeline getting data from GitHub selecting top repositories cleaning up part of the data and then especially living only code that is licensed under permissive licenses such as MIT uh BSD Apachi 2 and so forth out of this mixture we selected 30 top languages and then really the key secret ingredient here is how much time we spent on working on the data you must have been hearing this again and again and every time you go to an llm talk there is a g stage saying you should pay attention about the data quality uh I'm here to tell you exactly the same once again that's probably the most important thing that you could be spending your time on especially because the model I'm talking about today is stained from scratch so this is not a fine tuning all the models that we release have been trained from the very first token prepared by us so it's extremely important to have high uh data quality so we we took inspiration from the initial quality pipelines built by codex by the pound paper and then we applied way more tics there so we're filtering for code that is being autogenerated minified non-arable basically all the code that you wouldn't want your model to recommend back to you because it's not something that you will be writing your self we also remove toxic content and all this pipeline have been built on spark so I'm trying to encourage you to also think of working on your own models because pretty much a lot of the base components are out there available open source so you you could really build the whole pipeline to train and serve an llm with a lot of Open Source components and as Wix was saying you have seen this CRA acceleration in the last 9 months if you wanted to do this in 2022 good luck with that uh it feels like we're a decade ahead compared to last year so it's pretty amazing and I didn't even expect in myself the speed to move this fast the other inside that we kind of pioneer for the our V1 model and turns out to be very powerful also for this new one so when we release the V1 few weeks after coincidentally a very interesting paper has been published called scaling data constrain language models and I high recommend it it's a it's a great read and it's probably one of the most interesting results in LM in opinion and this intuition allowed us to basically train the model to completion rather than making tradeoffs on the data quality it allowed us to select a small high quality subset of data and then repeat it several times the key finding of this paper is basically in these two plots I'm going to be sharing the slides so you know you can go and check the links and the idea is your loss curve after you repeat data four or five times is going to be comparable to training on a novel data set okay now not only this is very useful because it allows to work only on high quality data it also allowed us to work with data that is exclusively released under permissive license therefore once again for our 1.5 model we're going to be releasing it open source and it's going to be released with a permiss commercially permissive license so you can use it there we go just shoot as an email when you use it because I'm very curious if you're having a good time so details about the training we change a few things here and there slightly larger model it's a 3.3 B uh it's 4K context the old one was a 2K we train a new domain specific vocabulary 32k so a small one uh it it helps us to achieve even higher compression on the data uh if youve been reading again about llms you know that in a simplist from a simplistic point of view there are data compressors losty data compressor so if your vocabulary allows you to get to pack even more data on fewer tokens then your B basically bringing more signal to the model while you're training and with this new vocabulary we're squeezing you know a few percents extra and it's a better vocabulary for code compared to what star coder or code Lam are using today we trained 128 h180 gigs gpus which are as rare as you know gold at this point we have been on the Mosaic platform for a week and to our knowledge this is the first model officially announced to be train on h100s and release open source so we're very excited about it and we follow a list of llm best practices so of course we support flesh attention we have group cure attention which allow us to achieve better inference performance alabi position embedding latest optimizers in the game and that you know is really the reason why at the end you will see very exciting numbers that I don't want to spoil right away so let's start from the base model and then there is surprise coming um this is the evaluation pel one on human eval for those of you who never heard about it umal is a benchmark list back in 2021 if I recall correctly the format is the following you have a natural language description of a task in English and then expect the model to generate a self a self-contained python snippet that then is going to be tested with a test harness so you generate code and then you execute it and you see if the if the values in output are exactly what you expect now an interesting evolution in the last few months in the field is we were not content on on benchmarking exclusively on python so we're also doing that across several different programming languages and this is coming from the multilingual code eval Arness again built by big code and they also maintain a very interesting leaderboard so what they do is they take models across you know several companies and several open source contributors the Run devals themselves and then they compile this very interesting leaderboard so you will find us there I guess uh in a few days so uh from the left column we have star coder Tri which as of yesterday was the state-of-the-art model at the 3B parameter size across languages and today our we 1.5 is basically optimal across every single language that you see on the list but what gets me excited is not that much know the fact that we are know more powerful than star coder which has been released a few months ago what got me hyped you know when we were training it is that we're very very close to col Lama 7B so as a reminder col Lama 7B is a Lama model from meta the 7B version which has been trained on two trillion tokens of natural language and then it has an additional pre-training phrase of 500 billion tokens exclusively on code so it's a model that is twice the size it's 2.5x more data way more GPU compute so you see where I'm going you know we're getting very close how do we surpass Cod Lama here is the trick this is the the other model that we've been training in parallel and this is the rle tune version and it means the following we further pre-trained it on 200 billion tokens of code this time coming from our home developers so on repet when you create a public rep hole it's automatically published under IM license so we use this code to further pre-train our model and we extract again 30 billion tokens of code same languages same data filtering pipeline to retain only the top quality ones we do this three EPO we do also linear cool down and we are using basically the languages that are predominantly popular for repit users so not the same list as we saw before if you go and rep it I would say 95% of the people are mostly writing Python and JavaScript these are the cool cool languages of today another key Insight is our cut off for this model is literally a few weeks ago so if there is a cool new library that everyone is you know writing software for in the last month month our model is going to be capable of generating code that you know follows that Library so and we're going to keep basically these models up to date so that we can follow the trends and we can make our developers more happy here is the table that I love so we're back to this uh back toback comparison uh on the very left we have our base model uh we didn't add star coder here for a sake of space and also the mod the Bas model is already uh topping it every other language so it didn't make sense now we have col in between and you can see why uh we are on pretty much every language substantially better so we have 36% on the open AI human eval Benchmark as a reminder um I when I was working on pal coder for example that was our ped one result uh that we published uh in early 2022 and that model was at 530 billion tokens so almost 200x larger than this model and it achieves exactly the same you our pass one performance same Cod D inci 001 if you go back to the paper is getting exactly 36% so we were pretty much amazed when this happened now why do we go through all this struggle of training our models not only because it's cool you know uh we love to do this stuff but there is a there is a rational behind it so we really want to go as fast as possible with the most powerful small model we could train and the reason is all of our models are actually optimized for inference rather than for being awesome at benchmarks the fact that that happens gives us a lot of Pride and also makes us feel good when we do have Vibe check with the model and it performs as we expect or even better but it turns out that our key result is on a single model with no batching we're generating know above 200 tokens per second and we tune the architecture for Speed in every possible way we're training a smallab as I was saying before we're using flesh attention with a traton kernel we're using the latest gqa so every single aspect is there to make sure that we can go as fast as we can and we optimize basically for the usage on the Tron inference server and acceleration framework such as tensor or tlm they really squeeze you know the last drop for MB the gpus now the other very interesting Insight is we work very hard also to make the model deployment go much faster so if you ever know had the bad luck to work with kubernetes in your life you know you know thankfully can get you know to get your pod and you download all the dependencies and build it and yada yada you know so the very first time we brought this infrastructure up it took 18 minutes to go you know from clicking until the model was deployed now if you want to you know adapt to the load that the application is receiving 18 minutes you know looks like an eternity like if if there is a traffic Spike good luck with that um so one of our awesome Engineers Bradley you're going to find him at the boots later today brought this number from 18 minutes to just two minutes there is a laundry list of tricks that he used I'm not going to go through them just talk to Brad uh the cool inside here is the fact now whenever we get more load we can react very quickly and that's how we serve a very large user base so the moment that I'm J announc AI for all literally 10 minutes ago we fli the switch and now Co comption is in front of all our users and that's the way we made this happen now I've been asked several times guys why are you releasing your model open source you put so much effort maybe not that's that's an advantage for a company um it turns out that the moment we did it we got a lot of adoption and apart from a lot of laog which you know always feels good and it feels good you know to chat with other people in AI uh that are using what we build uh we also started to get fine tune versions instruct tune versions of that and we have seen a lot of people using our small model deployed in local say with gml which goes super fast on Apple silicon and they built their own custom privacy aware like GitHub copal alternative with rapid we1 so we expect the same to happen with uh v1.5 in the next few days as we speak also if you go on Ain phase the model is available uh we're working on the read me come to to with mava the boot is The Mastermind behind it so it's going to tell you every single detail on how to make get running production we're going to be here until tonight so more than happy to play with the model together now in the last minute I've left I want to give you like a teaser of what we're going to be doing in the next weeks so we aligned a few very exciting collaborations the first one is with glaive AI and it's a company that is building synthetic data sets and we're working on on an if version of our models on instruct F tune version over 2,000 uh 210,000 uh coding instruction we're already seeing very exciting results we want to trle check them and you know follow our twitters and the moment that we're sure that this is performing as we expect is going to be out there and you're going to be able to play with it second announcement we're also collaborating with more flabs I think Jesse is here today and he's going to run a session later explaining you exactly what uh this new format does I'm going to give you a teaser and then you know go to Jesse talk and he's going to explain you all the details so we are design pars on the fist format which is f in the syntax three you might have heard of feel in the middle this concept where you can take your file split it in a half and then basically if you're WR writing code in between you can tell the llm that the top of the file is a prefix the bottom of the file is your suffix and you give this context to the model so that it knows which part it should fill now we found that this format is even more powerful is aware of the abstract syntax 3 underlying the source code where very uh promising results already and again this will be out you know just a matter of like a few days or weeks last thing we have collaboration with the perplexity AI guys you might have used their Labs so it's a place where the host models incredibly fast and the rapid B 1.5 will appear there and you can start to play with it and get a VI check by tonight thanks [Applause] everyone [Music] ladies and Gentlemen please welcome to the stage the inventor of Auto GPT and his team torrin Bruce [Music] Richards [Music] thank you San Francisco with a warm welcome I'm torren the creator of Auto GPT and I'm excited to show you all what the Brilliant Minds that aut gbt have been working on over the past months I'm going to hand off the stage now to senen one of our founding AI Engineers thank you there you go thank you Toren um I want to talk about something that I think not many of you realize I didn't realize this for a long time we're not achieving the peak of our potentials we can all work faster we can work better and we can do more with less time and less and less stress let's take this fretetet for example I don't know about you but I've stared at this interface for hours on end and um I'm sick of it how would you go about filling out this spreadsheet all right it's the lead generating name of the company you got the links uh what you probably do is you go on Google you search copy paste maybe go on LinkedIn copy paste back to Google over and over and over again for hours going back to the same interface going back to the same websites but what if instead of all that you can just chat and you get the same end result a filled out spreadsheet with all the leads all right let me give you another example we all have unread messages right not because we're lazy allegedly but because we're overwhelmed now how would you go about cleaning out your inbox you'd sit there for hours and hours sending the same variation of the same email but what if you could just chat last example I promise actually uh these emails will now be leads in your inbox uh instead of uh just unre emails last example say you're a company or a developer you spend millions of dollars developing apps that take weeks months sometimes even years sitting there copy pasting anyways cuz you're probably using Chachi PE I know I am copy pasting but what if instead of all that effort you just chat I think you get the point there's a reason you've heard of autog GPT autog GPT inspired the minds of millions it gave hope to what a world could look like where we all reach our full potential the light at the end of the tunnel you could see the Sparks of digital artificial intelligence and in this world everyone goes from using their minds mostly to execute menial tasks with only 10% of their brains being used for Creative work to becoming creative masterminds or crating the peak potential of their lives and in this world we're all AI Engineers whether you know it or not and people have noticed Auto GPT was the fastest repository to 100,000 Stars every major new news network picked up on this everyone understands what the potential of this is and it kicked off a whole new field of development a whole new paradigm of augmenting humans to give them time back and live a more stress-free life and even the major players in the space all realized how big of a deal it is and work and now work on these agents and so I want to hand off to the primary open source developer at autog GPT to to talk a little bit more about the open source repo thank you slen thank you slen and hello San Francisco I think all of us being here is a Real Testament to the power of Open Source and on that note we have some really exciting news to share because just last week our open source repo auto gbt hit 150,000 stars on GitHub of course metrics are fun but to me it is so much more than just a number it is the 150,000 people who took an interest in what we're doing and decided to click that button so if you start our repo then thank you it is also the 460 Plus contributors who took their time and effort submitting thousands of PO requests and issues in the process and to all of them as well thank you so much it is also the 47,000 members of our online community and all of the interesting and insightful interactions that they've given us it's been a wild ride at times but it has allowed us to do and learn so much in the past six months and I'm extremely excited for what is to come based on that now I've already said it but we could not have done this without our community and Community matters so we are committed to fostering to growing and to empower this community and to build the future together and I'll head it back to slend to tell you what that means thank you puts and we haven't stayed stagnant since the open source agent originally came out we've continued to work on it and we've continued to improve its capabilities and implement the latest cut Cutting Edge research but we've also been working on some other things to show our commitment to the agent space and the open source ecosystem we' built Forge which is a template for any agent for any agent Creator to have a better time to develop their agents with a standardized template we also build a Dev tool UI to easily interact with and iter iteratively improve your agent using an intuitive interface all of these tools are built on top of the agent protocol from the AI engineer foundation and other industry standards to maximize compatibility and interoperability anyone who implements this protocol can use our Benchmark friend end Dev tool and other offerings built on top of this protocol and while this Dev tool template is in beta or in Alpha uh it has served uh our participants of the current hackathon we're running where we have $30,000 in cash on the line and we've learned a lot from this we've received a lot of great feedback we received a lot of bug fixes uh and insights that we're going to take into the future one of those insights is that code is King we've realized that coding agents are the fundamental agents uh of the world let me tell you the digital fabric the fundamental digital fabric is code our goal is to build a general generalist agent yes but code is this is the stepping stone to AGI a motivated coder can get anything done uh except for get a bed frame another thing that we've learned over time is that without a compass you don't know where you're going you know at the start of the repo we were getting thousands of pull requests and that's a poll request every 2 hours we had no way to know whether the poll requests were good and how do we even test these poll requests we didn't have a real Direction it took time to test these and it was unnecessarily costly and so we created a compass we created a benchmark to direct the development of the open source repo and quantitatively know if we were improving it's an easy way to know if your agents are improving down different categories and people are currently benefiting from this for the virtual hackathon and this is just cool we've been running this in our CI pipeline for the past couple months on different open source uh agents within the ecosystem and what the tests have shown is that we're on the brink of something special these agents have showed continual Improvement and don't worry I wouldn't put this in a research paper the it's very noisy it's very messy but there is a continuous trend from 35 55% uh this is just a graph of the success rate on The Benchmark over time over the month of August another thing that we're committed to is safety as the ecosystem grows and as the capabilities of Agents increase there's always questions of trust and reliability and these are problems that autog GPT is committed to one of these problems is prompt injection which will always be there OAS uh one of the big security organizations uh has talked about this and said this is one of the big problems that not just language models face but also agents it's essentially when agents visit a website what all agents need to do and uh the website has something malicious um and then the llm is like all right I need to be doing that now and you can see that there uh in this example then there's this other uh category that I like to call uh innocently malicious where uh agents are just bad sometimes it's the truth and in this example behind me uh this person asked uh an open source agent to delete all the Json files within a directory a specific one and the agent ended up deleting all the Json files on a laptop and this is going to continue to be a problem if we want agents to do the things that humans can do they will need root access and So within Auto GPT we're committed to and think about these problems extensively and we've been working on a research paper to solve some of these issues um and in order for agents to be commercially viable and trusted these safety problems need to be solved you can't have a 99% success rate it has to be 100% that one email that sent could be a lost contract or our lost lead and so this is fundamental not just to the development of the open source agent but to all agents out there after all our end goal is a digital AGI to augment all of humanity and and I'm going to invite Craig to announce some exciting news regarding some developments with auto GPT hello so it's been a wild Journey from zero to here in six months and we keep stressing this because it's so important to us we're only here because of our community because of that shared passion in pushing the frontier of what AI agents can do so we're really excited to announce that red Point Ventures has invested $12 million in turning this Vision to a reality now this isn't just funding this is the them showing their deep belief in our mission and their dedication to open source s that's why we went with them because they are so dedicated to staying open source and that's really important to every single one of us working on this project now this is where we need you with this funding we want to grow our team and add more passionate individuals so join us message us join our discore community and let's all help make the world's best open-source generalist agen together we can redefine the future of [Applause] work thank [Music] you ladies and Gentlemen please welcome to the stage stage our next speakers applied AI engineer at open AI Simone Fishman and member of developer relations at open AI Logan [Music] [Applause] [Music] [Applause] [Music] Kilpatrick cool how's it going good hey everyone um a little bit about us too uh so you can think of opa as a as a product and research company uh we build aome models and then we think about what are some of the best ways to apply them to solve the biggest problems that Humanity faces and so there's this uh deployment pipeline Logan and I said at the end of this deployment pipeline we work with uh people in the real world that are using open eyes models we spend our time thinking about what are some of the best ways to use our models what are some of the hardest problems that haven't been solved yet and how can we apply uh open eye technology to solve these um I'm on the apply team and I'm a engineer yeah and my name is l Patrick and I do developer relations stuff so helping people build fun and exciting uh products and services using our using our API so yeah folks saw from the title of the talk uh we'll we'll talk about multimodal stuff but I think it's important to start off with with where we are today and I think you know as as we all know people who have been building in the AI space for the last 6 12 18 months um 2023 has really been the year of chat Bots and I think it's uh it's been incredible to see how much people have actually been able to do how much value you can create in the world with like just a simple chatbot um and it's it still blows my mind to think about how rudimentary these systems are and how much more value there's going to be created um in the next year in the next decade um and that's why I'm excited for 2024 which I think is is really going to be the I don't know if I can trademark this but the year of multimodal models um it's a tongue twister but also hopefully the domain is available year multim models.com um no don't don't buy it if it's available uh yeah so I'm I'm excited we you open AI has a ton of multimodal capabilities that are that are in the works um some folks might have already tried some of these in chat gbt and the IOS app or the the web app today things like uh Vision taking in images describing them um we'll we'll show that later on um also the ability to generate images we've had this historically with with Dolly 2 but uh doly 3 really if if folks have tried it it it takes things to the next level so excited to to show some of that today as well cool so if you if you think of uh the way that multimodal capabilities are working right now a it's a little bit of a of a setup of islands where we have di that takes text and generates images we have whisper that takes in audio and generates text transcripts uh we have gptv with vision capabilities GT with vision capabilities that takes images and text and can resen over both at the same time um but right now this these are all very disperate things uh a how however you can think of text as a connective tissue between all of these models and there's a lot of interesting things that that uh we can build right now uh using that Paradigm but what we're actually really excited for is uh a future in which there's Unity between all these modalities and a and this is where we're going this is not where we're today uh but you can you can think of models in the same way that that like GPT can consume uh images and text simultaneously uh maybe in the future we'll consume May more modalities and we'll uput even more modalities and we be able toon about them in the at the same time however we're not there yet and so uh today Logan are going to show you just like some some uh architecture patterns and some ways in which you can uh mimic this kind of situation with what we have available today and and and some of the patterns that you can start to think about as we move towards this future in which models can a reason Way Beyond text as Simone and I were were making these demos today um waiting till the the last minute as as always it it was really interesting to see that like really much of the work of making multimodal systems today is like how do you hook everything up together and connect the different modalities and again as Simone said using text as sort of the the bridge between different modalities um but it's going to be super interesting to see like how much developer efficiency gains there are when you no longer have to do that and you really just have like a single model that can do text in text out video at some point you know speech in speech out at some point um so it'll be super cool to to see when that's possible and uh make make making demos even uh even easier and simpler all right well uh we'll show you guys two demos today uh and we'll talk about like some some high level ideas and some high level Concepts uh and hopefully at the end of it you you'll be inspired to think about like what what are some of the things that uh maybe you're not able to build today but you'll be able to build uh 6 months a year from now and how you should start think about your products uh as they are able to incorporate more modalities cool so on to demo number one uh this is a it's a it's a very very simple Del Vision Loop um yeah so yeah sorry um excited to to look at this demo so Simone we we'll pull up the demo and I'll I'll sort of just walk through it but the basic idea is let's take a real image um let's use gbtv or gbt 4 with with image inputs to essentially create a nice human uh readable understandable description of that image um and then we'll put that into Dolly 3 and actually go and generate a synthetic version of that image um so this whole pipeline takes a little bit to run because uh it's not a production um system at the moment uh but the nice part is uh we've got a couple of examples ready and we can you want kick one off live as well we can let it right the background so very very this is a a fun simple idea but uh the this is a photo that I took in the lobby downstairs uh just when you walk into the the hotel uh there are these uh kind like a Halloween themed Painted Ladies um and so what we did here is that we asked uh gb4 with vision to describe this image in detail and then we asked it to uh generate a description for Del to uh generate a new image based on this um you can see it does an okay job here's the description of the image here's the prompt that uses uh it it picks up on a lot of details like the the RP and the tombstone and the old dogs uh welcome uh think here and then it generates a whole new image but there a lot of details are off you know like the the um the marble is black and the uh the SP are wide and so what we do next is that we pass the yeah it's close enough uh but we give the two images to GPT with vision again um and we asked it to compare them and see what are some of the differences and uh it it picks up on a lot of the the different details and then we ask it to create a new a new image based on these uh differences and it goes ahead New Image you see uh all the black marble is gone the spider is now larg and black uh but you know it it matches something closely and I think this this is just to illustrate I think there's a long way to go but this is to illustrate the idea that there are plenty of tasks that we do right now in AI where we we need the human in the loop to be able to evaluate a visual output that a model produces compare it with something else then like iterate on the instructions pass that again to another model and so that that's a pipeline where we like thought that humans were very essential and that we're probably going to essential for some time and now that's something that the models can do by themselves um and there's a couple of of uh interesting uh patterns here I think I think one of them is describing images that's powerful because now you have an image now you have text and you can resent about that text you can do a lot of things with that text but another really powerful element is a uh comparing images um and and and spotting differences like having like a final destination they want to get to and like a current destination and and that pattern of comparing things you can apply to a lot of things so imagine to and I Logan and I were just chatting about like some other like way that you can apply this and and and Logan's idea was imagine you are uh creating uh your room and you're you just moved to a new place and you're in Instagram you find some images of like a vibe that you like and and like maybe some object and and then you can grab that image you can give that to gp4 with vision and you can tell like okay now like like cwl through Amazon and find like all the lamps that match this vibe that I want for my room um I want this so badly this I can't do interior design so it's like I I would love to be able to just be like get me all the stuff that matches this specific Vibe it's it's a a hard problem right now yeah um and a couple Simone can I make one other quick comment which is just I I think also you know folks were were laughing you know in in good Gest when this when this third image came up came up I think it's important to know that there's there's like no prompt engineering or anything like that that's happening this is like the the rawest output that you can get this is a a 1hour demo version so people can uh will hopefully go wild with this once it's available through the API and like ideally get much better results than than we're seeing today um yeah probably using a bunch of techniques that other people talked about at the conference so far so this is the the very basic version of of this demo yeah and and we wanted to keep it simple minimal just to illustrate the the the power of the models this is as raw as you can get when it comes to the models that like there is almost all the completion output it's going straight into the model and and I think there like 50 lines of code so like the majority of the powerlifting is being done by the models here um um another quick example that I'll show you guys and then I'll try to do one live uh which will probably uh be tragic but uh um so this is this is the backstage right here I just took this photo right before walking on stage uh you can see that uh GPT with vision does a really good job actually of describing that uh the there's a monitors and there's boxes and there cables and there's what not uh and then this is the image that Del generates Del 3 uh so you can see blue carpet cables boxes allot the elements and then it goes on to spot the differences and it notices for example that in this image there are all these vertical lights that are not present in the first image says that here uh lighting like all this like vertical lights on the walls and ceilings which adds H but then it rewrites The Prompt and it gets rid of all the vertical lights and it get and and it adds the uh the curtain in the back which wasn't present here but it's present in the the black curtain here um so just little interesting things it's still a long way to go but like this this new this whole new this opens a whole new box of interaction patterns the the fact that now you can reason visually um cool and and let's give a shot to a a live example so this uh this was a a trail run that that I did over the weekend uh up in pisma woods um and so I was going to do it from scratch um hope that it works want to go to another there you go uh cool so the image dep picts Syrian ambitous Woodland setting uh the focus of the imagees a wooden Boardwalk or a fot bridge that winds through the dense forest uh very Det description light filters through a trees uh and I'm just passing that raw just straight to Del yeah and if if folks have seen what happens in the the DI uh mode in the chbt IOS app for example it's actually doing a little bit I I don't uh know off the top of my head like what the prompt is for that but it's it's doing some amount of prompt engineering like if folks have actually tried to use like our Labs product before to make Dolly images you have to do that prompt engineering yourself um and I think that's been one of the limitations like if people used mid Journey or or other um other image models in the past like it's just kind of hard to make good prompts that work well for these systems so it's nice that the uh the model can can take a stab at doing it for you it's telling us a lot of how the the second image is a lot more beautiful and more detailed which checks out it's it's also interesting to see uh just for folks to to think about it's interesting to see that like it's still of these um image models like the main limitation as we're seeing this demo in real time is actually [Music] no of course we're going back to the slides next go back slides time I'm Gonna Leave it running and then at the time if we if we have time it'll probably work the second run it work the three times before this um cool okay for the second demo um a we're going to take it a little bit further and we're going to do something uh with video a and the idea here is that there's a lot of video summarization demos out there that we've seen the majority of them just take a transcript and then uh ask gp4 to summarize this transcript however videos have a lot of uh information in them that is conveyed visually and so what we're doing here is that we're taking frames from the video um and then we're asking gp4 with vision to described all the frames and then we are asking whisper to transcribe the video and now we have this long textual representation of the video that not only includes all the audio information but also includes visual information for the video and then we're doing some exciting like mixes on that uh that Logan will tell you about yeah I'm ready for the next slide um yeah so for for this demo we're literally just taking the gbd4 introduction uh video If folks have seen on YouTube it's a good video If you haven't seen it before um so taking the video raw from YouTube uh taking the video raw from YouTube again like Simone said cutting up those uh the different frames from the video putting those into to GPT for with image input getting the summaries which you can see and I know it's really hard um but literally just like actually saying what's the these are simple images so it's easy to capture the the depth of what's shown here um taking those images and then going to the next piece which is essentially a big another another wonderful J Dolly image but a big description of uh of the transcript and then all the image essentially like image embeddings is the is the easiest way of thinking about it so if you want to actually see the results of this QR code bottom right hand corner is real um you can scan it and see the resulting article it's it's pretty it's pretty good um it does a good job and I think for for me you know why this is exciting is cuz you can sort of capture the again capture the dev of uh of what happens in a video so a dolly image to start and then a bunch of actual frames that like match up with the contextual representation of what's being talked about in the blog post um and again there's no hand I I couldn't open source the code because it has a bunch of unreleased apis but no no sort of magic behind the scenes stuff that's happening this is like a raw crappy prompt um to generate this uh this blog post which I think is again I think is really cool and um takes videos and and makes them more accessible in the in the text form so I Like It Cool let's see if this finished no oh well um cool oh yeah sure cool okay uh so some some uh concluding takeaways um a start thinking multimodal uh that's that's something net new that's that's happening these days and and if you have any crazy ideas that you think wow it would be really cool if if technology could do this uh we'll probably be able together and and the products that you'll be able to build 6 months from from now a year from now are going to be incredible so start having this in mind as as as people who are building AI products and people who are building companies um think of text as a as a connecting tissue right now and and I think this is a very powerful concept and that's going to continue to be the case for the near future uh and there are many powerful patterns that are yet to be explored when it comes to multimodal stuff especially when it comes to to uh doing things with images uh so really excited to uh soon get in the hands of all of you guys and and to see why you all build with this I think it's uh it's really exciting uh to see a AI start to venture into the visual world yeah agents with image input is going to be sick I can't wait feel like so much of the internet is requires that yeah and we're excited I think there's there's a lot of stuff that's going to happen in the in the near future and um I think it's cool to be able to hopefully get a glimpse of of what some of those use cases look like so anything else you want to say say Sone that's good all right this is wonderful thank you all thank [Music] [Music] you and now please welcome the founder and CEO of Lindy Flo crello [Music] Tak one more breath beside you so I could find strength to divide [Music] us thanks everyone I'm going to talk about the future that awaits us and not the Super distant future either like I'm talking about 5 10 years so only within our lifetimes the future that awaits us once agents have fully realized their potential if I had to describe it in one sentence i' say that it's a world where a 25y old can have the same or more business impact as the Coca-Cola Company it sounds insane when you say it this way but there's actually a precedent to it it happened before with media consider what Oprah had to do to build her media Empire right you had to go and pitch a bunch of a bunch of CNN Executives in some stuffy room and raise money hire your crew find cameramen and obviously the internet and apps like YouTube have brought that friction down to zero and he brought that fiction down so much it actually sounds like a joke consider the top YouTuber Mr Beast right it's got a much greater reach than Oprah he he actually has more overreach than the Super Bowl and it's just him and his laptop is how he got started so it happened before even weirder Ryan's world he studied when he was 3 years old making videos on YouTube of him unboxing and reviewing toys today he's 12 years old and he's worth $100 million my point is that once you bring the friction down to zero and once you remove The Gatekeepers you don't just get the same kind of content except cheaper the nature of the content changes when you remove The Gatekeepers right look at opra look at the Super Bowl that makes sense it's like a talk show it's like a sports game Ryan's world and Mr be is just weird and so my point is that we're about to see the exact same transformation happen to the world of business at large we're going to take the friction down to zero and as a result we're going to see much weirder more creative ideas come to life you know Lind is my second startup before it I had another one called team flow and I remember when I started it I had a perhaps naive understanding of what starting a business entail I thought it was all about building a cool product and bringing it to Market and then I found out that actually the fun part right before you get there people I see the the fers laughing in the audience uh before you get there you've got to meet with lawyers and incorporate and meet with bankers and open the bank account and meet with vs and raise money and meet with recruiters and hire a team and it goes on and on and I mean you guys know once you have a business it's not much easier just keeps going so when that wave of generative AI came about all these amazing products that we seeing that generate copyrighting for you generate images I was like that's awesome but it doesn't solve my problem of it's just too darn painful to start a business and also the GTP isn't made of copywriters or illustrators it's made of work and actions so that's when I got interested in agentic ai ai that can actually automate the Minal parts of your life does this amazing movie of his space by the same guy who made silicon Val highly recommend and there this awful depressing character in The named Milton Milton he spends his life in some basement doing God knows what they call it filing TPS reports and in the end Milton does the most productive thing of his career which is that he burns the building to the ground that's what we're going to do fig figuratively the cops are coming um you know I I take this as a symbol that no one is happy with this status quo people are always worried about oh robots are stealing people's jobs I think it's people who've been stealing robots jobs do you want to be Milton and it's a huge problem when you look at the data the average manager in the US spends 15 hours every week on this kind of admin ative task that's $459 billion every year just in the US that's more than the GDP of no way so that's where we start we build an AI employee and the first thing it does very well is it acts as your personal assistant we've Called It Lindy the good news is as we've dug into that problem space we found out there's three big time wasters and and you know the ones no surprise there this is where you spend your life at work and you hate it so calendar your email your meetings so the product we built those are actual screenshots of the product you can ask it to schedule your meetings for you this example here is actually pretty cool because it demonstrates another ability of Lindy which is she continuously learns from her interaction with you so here I was like help me find half an hour every week with Eric and she called it FL Eric because previously I had asked her find stud minutes with Eric tomorrow and she did that but she named the meeting meeting with Flo and all my meetings or meeting with Flo so it's not very helpful and I was like no I give her a little bit of feedback call it Flo Eric and she did so she renamed um meing and and she saved the preference for future instances and generally I can give any arbitrary preference of any complexity that I want to Lindy and she'll remember them and honor them I can CC Lindy to my emails so that she helps me schedule them and when you use Lindy she can pre-draft your replies for you in your inbox in your voice for each individual recipient because you don't talk the same way to your partner as you do to your investors hopefully so every morning I wake up I open my Gmail and I just have all the drafts ready for me to review Lindy prepares me for my meetings so I've asked her hey five minutes before every meeting send me the zoom link the LinkedIn of the people I'm meeting with and the summary of my last few emails with them she just does that now the really crazy thing is that we ourselves didn't actually build any of these features what we did is we built a universal framework allowing an AI to pursue any arbitrary goal using any arbitrary tool and some very complex and sophisticated behaviors come out of that as we'll see later now my pet peeve every time people go on stage and they talk about their AI products they always talk about the good part it always works it's very Cherry Picked and so I'm going to break that pattern a little bit today and I'm going to talk about the time when it didn't work a few weeks ago I asked Indy to help me work on my vocabulary and every morning to send me a new interesting world and so she does that every morning I wake up I have a new world in my inbox that's great I start to use them until one morning I received this word p liquidity a captivating term denoting the act of me entering through a conversation with no fixed Direction and I passed for a minute on that one I was like pure liity so I was like I never heard of this one before I just Googled it and sure enough it doesn't exist so then I went back and reg Googled every world that she sent me and none of them existed so if i' if I've used any whe that doesn't exist today that's why she's been poisoning my brain but when it works it works great and what it means for you when it works is that your Computing experience of the future isn't one when you're when you're in the basement failing TPS report all day it's not one where you're working on your computer it's one where you're having a conversation with your computer you're in Flow State you just focus on what you uniquely do best and all the minial awful parts of your work that you hate arrange themselves automatically for you now I don't know about you guys I think this is all the awesome I cannot wait for this to fully come to fruition but it doesn't yet get you to the stage that I was talking about for a 25y old has more business impact than the cooca the Coca-Cola Company in order to get there you have to go one step further and instead of having just one Lindy work for you as your assistant you can have an entire Society of lindes working together on your business to pursue your goals if you want to realize how powerful is that can be consider the fact that every single item around you in the room right now was made by a group of people even the simplest of items not one person can do it in fact there's this guy who ran this project called the toaster project he wanted to see can a single human make a very simple item like a toaster he spent six months on it it cost him $2,000 probably more like 50k if you include the value of his time and this is what heend the in the end or it could have gone to Amazon and bought a perfectly fine toaster for 25 bucks I think this contrast is a good illustration of the difference in abilities between one person six months 50k pretty badl looking toaster and a group of people 25 bucks perfectly fine toaster I think the same thing happens to llms you go to GPT you ask it to do something like hey build an entire IOS app for me subut design it publish it to the App Store do everything it can't and then people conclude oh gp4 can't do it right you got to wait for GPT 5 GPT 6 gpt7 I think it's the same thing as if you went to some guy and you asked him make a rocket for me and he can't and then you're like oh humans can't make rockets and obviously they can just got to let them work together so that's exactly what we built is a framework for multiple agents to work together in pursuit of your goals this is what it looks like the most awesome example that I know of is that we have created a society of Lindy for Lindy to build herself we need to build a lot of Integrations for Lindy to work well with Slack twio Google Sheets and so on and so forth instead we are building this Society of lindies at the top level there's this tool creation Lindy that Tes an instruction like hey build a slack integration talks to this Lindy that goes online and finds the open API spec and the online web documentation talks to this Lindy that's a manager Lindy just splits up the task across many Engineers the engineers work on the task there's a specific engineer for the authentication code because there's a few goas here and then they pass the work to QA engineering Lindy that does the work and if it doesn't work sends it back to the software engineer if it works submit a PR this for the record is 70 or 80% of the way there but I think it points to the Future so this is how you get to that future where a 25-year-old in his San Francisco Studio can have more of a business impact than the Coca-Cola company I think this is going to be the greatest equalizer of human history today the best CMO in the world probably works for apple or Nike or Coca-Cola not too long from now the best CMO in the world is going to be an AI CMO same goes for the best designer in the world the best engineer in the world they're all going to be a designers AI Engineers they're going to work for you we're all going to have the same lever of Infinite Strength to make change happen in the world and the only question is going to be can you use that lever that's the only skill that's going to matter in the future imagine if you weren't constrained anymore by Time by money by your team by your network imagine if you could build anything and it was just you your your laptop and your lindies thank [Applause] [Music] you [Music] [Music] n [Music] hell I don't actually know am I audible now I am um that was day one Keynotes everyone how do you feel yeah give it out forone who [Applause] presented very very uh excited for everything that's uh already happened and about to come um there's obviously so much going on in the AI field and AI engineering Fields um but I feel like often times I've always wanted to do this at a conference and like now I help to run one so let's do it uh I always wanted to do one of these um Marvel's a little bit passes du but I think the model is kind of work working and um most of you are familiar with the first two um that we have um and uh any any lat space listers in the house yeah a little bit little bit um highly recommend uh we have a few other things that are planned and we'll be talking about them over over over uh the next few days um I think what I think what's interesting about the I field is so much stuff is being built um some stuff is much more much further along and some stuff is an infancy and we always want to try to encourage uh people to join in like it's not too late that's a lot of my message you're just in time it's not too late so you can still 1,000 x from here um so with that in mind I want to introduce um the the next set of speakers and and um the Marvel analogy is kind of blown up by ant uh CTO superbase who's in here in the audience somewhere cuz he said like we're going to make something called phone level announcement um and I really idolize you know the way that he presented it uh things in threes um so we're going to present three small things they're just in our infancy so let's talk about it um so first up we have bar talking about the state of AI engineering give it [Applause] [Music] out can people hear me awesome okay hi everyone um oh I see some some friends in the audience and a lot of new faces so I'm bar for those who I haven't met I'm an investor at amplify Partners where we invest in very technical Founders and platforms and I focus a lot on data and Ai and so we're going to talk about the state of AI engineering so where are we right now we're at the AI engineer Summit and even from today you see how quickly the field is moving I definitely don't need to tell this audience that everything from new state-ofthe-art models to changing and very rapidly changing tooling and so we had a conversation and thought it would be helpful to take a step backwards and say what is actually happening how do we get a good sense of the state of AI engineering how do we get a good sense of what tools people are using and how these things change over time uh especially given the really really really rapid advancements so we put out a survey it's still open and you all are going to get the very first Alpha View at the results of the survey 841 people have filled it out about how they're using AI at work as far as I'm aware that's the largest survey of AI Engineers that is out there and this is the very first one but after the inaugural one we can track these over time so the survey covers a lot and I don't have time to cover all the things but we go over everything from demographics to use cases to what are people actually using in their stack to some questions that people care about that are fun rapid fire to who who should we really celebrate in the community that is doing a really good job bringing people together and educating them through newsletters and podcasts on the demographics front before I was an investor I was a data SCI scientist for most of my career then I worked a little bit in data infrastructure are there any other data scientists in the house by show of hands not okay I see some in the corner how about uh software Engineers as your formal job title okay how about AI Engineers as your formal job title okay yeah so uh software Engineers actually beat out AI Engineers these were the top five roles but we're at an AI engineering conference we think that we're going to see a lot more more of the title AI engineering but it's both a job function and a skill set that we see across a bunch of different functions so swix has talked about AI becoming more ubiquitous of the folks that we talked to who have over 10 years of software experience 20% of them have less than one year of AIML experience but they're getting into the field now if I remember correctly something like 40% have less than 3 years of AIML experience so we're seeing this flood into combination of AI skill set and the AI engineering role uh there are a lot of use cases we could talk about but just to give some highlights most people are using llms for more than one use case for internal and external customer facing tools I think if we did this survey a few months ago there would not have been as many folks who are using external products so congrats to everyone that's doing the hard work to make that happen um accuracy and cost are the most important when choosing a model and serving cost and evaluation are what people said are the most challenging there's a whole section on evaluation and reading through the comments are pretty funny because um while folks had the opportunity to vote on human review and and uh academic benchmarks there were some write-ins like I evaluate based on Vibes or based on my eyeballs and so uh it's just a commentary on how far we have to go there and finally uh open AI models are the most popular good job uh the open aiers in in the crowd but 80% are experimenting with more than one provider and I'm including open source in that all right we're not going to go through all of this uh but we will share a survey that goes through all of this I'm just going to share some fun ones I don't know what happened there but as the Cliffhanger most AI Engineers are using a vector database and uh what I'm saying here that you can't really see is that there's a near even split between folks using third party and and self-hosted for Vector DB so we'll be interesting to see where this goes over time for prompt management I thought this comment was hilarious is prompt management a thing I prototype using open AI playground and then hard code prompts into source code apparently I missed something um and so we're seeing a combination of folks using external tools building internal tools that's the most popular thing and using internal spreadsheets but I think this is also an interesting question around kind of who in the stock owns this and does it become more important or less important over time as the models get better but as we're doing more Ai and and potentially have more prompts and finally we have a whole section of pretty fun questions around is the future open source or third party it was actually pretty even with open source narrowly winning and there aren't too many doomers in the AI engineering crowd [Music] um [Applause] so I I think uh 12% of folks confidently said there's a 0% chance of uh there 0% probability of Doom um but you see that there there's an interesting distribution there awesome and I just want to shout out these were the top 10 of each category of newsletters podcasts and communities and um the way that folks voted on this was if they felt like they've learned something from from one of these in the past months so major shout out to this yeah it's getting a lot of photos uh ma major shout out to the folks who are putting in a lot of work to educate and help build uh learning and space for AI engineering all right if you want to take a look at all of this more in depth got a QR code for you and uh a link you're the first people to see some early results of the AI engineering survey I'm always open to feedback to discussion to questions you want to see next some things like there were more people pre-training models than we expected are going to dictate what we continue to look at and what we continue to survey and share out but I hope you get value out of the transparency and don't be a stranger awesome to Sasha thanks so much yeah so that's the first small launch that we're going to do which is definitive industry survey um the next thing I think about when building an industry is open source communi so with that Sasha Shang everyone Hi [Music] everyone so today I'm very excited to stand here here in front of you to announce that we are starting a new organization called the AI engineer Foundation before before I start I just want to note that I have been an engineer for the last 10 years so standing in front of this audience is deeply uncomfortable for me uh so um I'll try my best to explain what it is um so um um for engineer Foundation we exist to solve problems for AI Engineers very on brand and today we're going to enumerate three problems uh to start with first every project is Reinventing slightly different interfaces interfaces to popular llm um apologize interface to popular llms uh have been implemented differently by different libraries and um and this is a problem for AI Engineers because we're going to have to learn uh each interfaces independently secondly uh tooling Gap development and monitoring tools lack interoperability this this is what I mean without standards people build end to-end apps that integrate with limited set of tools um this creates an ecosystem which is basically a bunch of verticalized silos that encourages turn however on the other hand with mutually agreed upon standards we find common points of shared interest so that AI Engineers can find the best-in-class tools with familiar interfaces and this creates a ecosystem as well that um this creates an ecosystem that essentially encourages a more stable uh infrastructure as well as more modularized framework as well as uh more collaboration problem three Venture backed open source lockin uh as well as uh we're preventing Venture backed open source lockin as well as um having to navigate RE relicensing challenges uh so some of you guys might know in August uh we have um this announcement by Hari Corp um after 9 years of terraform being open source they were suddenly getting relicensed to be a non-op source compliant um project and uh this created oh this created a lot of um panic in the industry and um luckily we have the Linux Foundation who acted very quickly to create open tofu uh to ensure that terraform stays open source so who we are as um AI engineer Foundation is um everything we do is open source and we are a nonprofit neutral body and we are building a strong AI engineer community our first project is called Agent protocol it's a simple API spec and as of yesterday it's start uh about 300 on GitHub and it's a unified interface uh for AI agent developers um it's currently a simple rest API and by rest API I really mean these uh nine endpoints as well as a well-defined schema for data types and you can check out agent protocol at agent protocol. with agent Protocol New tools are s cly uh Avail available uh so the autog GPT team recently launched uh the AR Arena hacks hackathon that is built on top of agent protocol so now we have evaluation benchmarks that if you're an agent developer you should totally still participate in this hackathon it's still ongoing um to submit your agent to this leaderboard to see how they perform there are three ways uh for for you to stay engaged with with us if you are a open source if you have an open source project and then you would like for other AI Engineers to benefit from you can submit your project to us and if you're a de if you are a developer you can stay engaged with us through our Discord Community um and um if lastly if you really resonate with the problems that we're trying to solve and if you are financially able to do so please sponsor us um more information can be found on uh the website a.f Foundation May the source be open thank you I don't know which we're using now but I'm I'm just going to stick to this for now um so yeah that's the foundation everyone very very early um I I do think that um we have to set these things in motion so that there's a place for you if you want to come join and collaborate in open source Community um a lot of people ask me about my stuff uh what am I doing I'm just you know moving people around or or you know suggesting projects and promoting people uh I am working on a thing um I uh am not like super ready to like talk about it but you know like there's no other place to talk about it um so it's work uh it's small AI uh Tim if you want to roll the clip um and and basically that's a cute logo thanks to Candy Co for that um and essentially uh it is what I've what I've been pitching um with um an API Gateway that actually uh helps AI Engineers to code faster uh make their codebase simpler and um do a lot of things that like would would otherwise take a lot of specialist knowledge right I think a lot of a engineering is it's about access uh from machine learning to product if you have been on the and so that's the website small eyes just went live today um uh if if you have been on the conference website yesterday uh thanks is Sean Oliver for coding this up you actually have seen uh the summit a iBot which actually presents the information about the conference in a in a better way that's more personal to me than uh the website because I sometimes I just want to see details about the speakers the talks whatever um and the reason like why don't we do this more often right like the open ey docs don't have a chat bot the Lang chain docs they kind of launch the chat bot is on a different domain if you have to find it somewhere um it's not ubiquitous because it's a lot of work and a lot of code and to to write um what we have found is that we've been able to fine-tune data into models on production uh traffic um and that's that's effectively the kind of stuff I've been working on that's that's a that's a screenshot of the fine tuning UI that opena has launched and uh this is an example of the kind of stuff that a fine tune smaller model can do uh that would eliminate a whole raft of uh sort of glue code that most people would be writing um so that's what I'm working on uh that's smalli um but I don't want to make it about me this this conference is definitely not like you know like the the small conference um each of these things are new projects in their infancy that we're presenting alongside of the Keynotes and the the the image I want to give is that this is a very new field uh there is room for you still and please join us please promote your projects this is a permission to space as well you don't need our permission I never asked permission from open the ey to to get started on this crazy journey and neither do you so go forth and build thanks everyone [Music] please welcome back to the stage the co-founders and hosts of The aiae Summit swix and Benjamin duny I still don't have a mic though there we go all right almost perfect all right how are we feeling I am so excited by so many things but particularly I'm especially excited about the announcement of the AI engineer Foundation you know when swix and I were first talking about making this conference back in February I initially proposed potentially we could do a foundation because you know this AI engineering phenomenon is going to change engineering it's going to change it we think for the better but a lot of lot of people are going to struggle with it and we want to help out as much as possible this is inevitable we're going to make this happen and we're going to help make it a much smoother transition so I'm extremely excited about that if you like the mission that was just announced um you can support it by getting a special edition te I don't know if we can get a zoom in on that on the camera or not this is the special edition te it is now set up just outside behind you you can scan a QR code have this as well no no no this is what everyone gets that's what everyone gets everyone's got one of those those are cool this is cooler but there are only so many and there's not that many so here's what you need to do there's a QR code out there you scan it and that takes you to a stripe checkout link you donate 50 bucks and you get a t-shirt simple I know but there's a lot of you and there's I think there's only like a 100 t-shirts or something so play nice um but yeah so we're going to a break now about 40 minutes and then when that break is then we'll come back for our second block of opening keynote presentations um and then after that we're going to have the topic tables we did move the food and beverage I know it says on the schedule food and beverage happening about now we move that to 7 I think it makes more sense since we just had lunch at like 1 um so food will be coming just not until about 700 7:30 after the uh the next block of sessions what else did I want to say this is just a preview of things to come I mean we had so many amazing announcements on the stage so many amazing speakers um and we have a lot more packed in for you so can we hear for all the speakers so far you have anything else swix do you want to say a few a few words before we break um no let's eat I mean I've sure these people are hungry and thirsty and want to chat well 40 minutes from now yeah or a few hours from now yeah all right I I haven't had lunch so sorry there's coffee I I don't have a mic [Music] [Music] anymore [Music] taking one more breath beside you so I could find strength to divide us we got it I know we did the best we could if I could go back under the mess would memorize your face before I go but this is how we grow got to give it up sometimes it's go know it when to kill your pride there's no one to blame nothing really stays the same this is how we go sometimes we hold on to let [Music] [Applause] of trust additionally users now that we all have access to chat gbt and can really easily access these models we have very high expectations when we're using AI Futures inside of products we expect outputs to be crisp exactly what we wanted we expect to see never see hallucinations and in general it should be fast and accurate and so I want to go over three easy to implement tactics to get better and safer responses and like I said these can be used in your everyday when you're just using chat GPT or if you're integrating AI onto your product these will help go a long way to making sure that your outputs are better and that users are happier the first are called multi Persona prompting this comes out of a research study from the University of Illinois essentially what this method does is it calls on various agents to work on a specific task when you prompted and those agents are designed for that specific task so for example if I was to prompt a model to help me write a book multi- Persona prompting would lead the model to get a publicist an author um maybe the intended target audience of my book and they would work hand inand in kind of a brainstorm mechanism with the AI leading this brainstorm they'd go back and forth throwing ideas off the wall collaborating till they came to a final answer and this prompting method is really cool it's because you get to see the whole collaboration process and so it's very helpful in cases where you have complex task at hand or requires additional logic I personally like using it for generative tasks next up is the according to Method what this does is it grounds prompts to a specific source so instead of just asking you know what part of the digestive tube do you expect uh starch to be digested you can say that and then just add to the end according to Wikipedia so adding according to specified Source will increase the chance that the model goes to that specific source to retrieve the information this can help reduce hallucinations by up to 20% so this is really good if you have a fine tuned model or a general model that you know that you're reaching to a very uh consistent data source for your answers this is out of Johns Hopkins University was published very recently and last up and arguably my favorite is called the motion prompt this was done by Microsoft and a few other universities and what it basically looked at was how llms would react to emotional stimuli at the end of prompts so for example if your boss tells you that this product is really important for your career for for a big client you're probably going to take it much more ser seriously and this prompting method tries to tie into that cognitive behavior of humans and it's really simple all you have to do is add one of these emotional stimuli to the end of your normal prompt and I'm sure you'll actually get better outputs I've seen it done time and time again from everything from cover letters to generating change logs the output just seem to get better and more accurate and the experiments show that this can lead to anywhere from an 8% increase to 115% increase depending on the task at hand and so those are three really quick easy hit methods that you can use in chat gbt or in the um AI features in your product we have all these available as templates um in promptu you can just go there and copy them um it's promp hub. us um you can use them there run them through our playground sham with your team or you can have them via the links and so thanks for taking the time to watch this I hope they've walked away with a couple new me method that you can try out in your everyday if you have any questions feel free to reach out and be happy to chat about this stuff thanks hi everyone I'm presenting Storyteller an app for generating short audio stories for preschool kids Storyteller is implemented using typescript and model Fusion in AI orchestration library that I've been developing it generates audio stories that are about 2 minutes long s and all it needs is a voice input here's an example of the kind of story it generates to give you an idea one day while they were playing Benny noticed something strange the forest wasn't as vibrant as before the leaves were turning brown and the Animals seemed less cheerful worried Benny asked his friends what was wrong friends why do the trees look so sad and why are you all so quiet today Benny the forest is in trouble the trees are dying and we don't know what to do how does this work let's dive into the details of the Storyteller application Storyteller is a client server application the client is written using react and the server as a custom fastify implementation the main challenges were responsiveness meaning getting results to the user as quickly as possible uh quality and consistency so when you start Storyteller it's just a small screen that has a record topic button and once you start pressing it it it starts recording um the audio when you release gets sent to the server as a buffer and there we transcribe it for transcription I'm using open AI whisper um it is really quick for a short topic 1.5 seconds and once it becomes available an event goes back to the client so the client server communication Works through an event stream server sent events that are being sent back the event arrives on the client and the react State updates updating the screen okay so then the user know something is going on in parallel I start generating the story outline for this I use gpt3 turbo instruct which I found to be very fast so it can generate a story outline in about 4 seconds and once we have that we can start a bunch of other tasks in parallel generating the title generating the image and generating and narrating the audio story all happen in parallel I'll go through those one by one now first the title is generated for this open AI gpt3 turbo instruct is used again giving a really quick result once the title is available it's being sent to the client again as an event and rendered there in parallel the image generation runs first uh there needs to be a prompt to actually generate the image and here consistency is important so we pass in the whole story into a gp4 prompt that then extracts relevant representative keywords for an image prompt from the story that image prompt is passed into stability AI stable diffusion Xcel where an image is generated the generated image is stored as a virtual file in the server and then an event is sent to the client with a path to that file the client can then through a regular URL request just retrieve the image as part of an image tag and it shows up in the UI generating the full audio story is the most timec consuming piece of the puzzle here we have a complex prompt that takes in the story and creates a structure with dialogue and speakers and extends the story we use gp4 here with a low temperature to retain the story and the problem is it takes one and a half minutes which is unacceptably long for an interactive client so how can this be solved the key idea is streaming the structure that's a little bit more difficult than just streaming characters token by token um we need to always partially pass the structure and then determine if there is a new passage that we can actually uh narrate and uh synthesiz Speech 4 model Fusion takes care of the partial parsing and returns an ital over fragments of partially P results but the application needs to decide what to do with them here we determine which Story part is finished so we can actually narrate it so we narrate each story part as it's getting finished for each story part we need to determine which voice uh we use to narrated the narrator has a predefined voice and for all the speakers where we already have voices we can immediately proceed however when there's a new speaker we need to figure out which voice to give it the first step for this is to generate a voice description for the speaker here's the GPD 335 turbo Pro form that gives us a structured result with gender and the voice description and we then use that um for retrieval where we beforehand embedded all the voices based on their descriptions and now can just retrieve them filtered by gender um then a voice is selected making sure there are no speakers with the same voice and finally we can generate the audio here for speech synthesis element and 11 labs are supported based on the voes that have been chosen one of those providers is picked and the audio synthesized similar to the images we generate an audio file and we store it virtually in the server and then send the path to the client which reconstructs the URL and just retre it as a media element once the first audio is completed the client can then start playing and while this is ongoing in the background you're listening and in the background the server continues to generate more and more parts and that's it so let's recap how the main challenge of responsiveness is addressed here we have a loading state that has multiple parts that are updated as more results become available we use streaming and parallel processing in the back end to make results available as quickly as possible and you can start listening while the processing is still going on and finally models are being chosen such that the processing time for like the generation say the story is minimized cool I hope you enjoyed my talk thank you for listening and if you want to find out more you can find Storyteller and also model Fusion on GitHub at github.com lrl Storyteller and github.com lrl model Fusion hi everyone I'm Jeff show and I want to share with you an interesting generative AI project that I recently did not too long ago I made a game with 100% AI generated content it's a simple game where you're wandering around lost in the forest and you go from scene to scene having encounters that impact your Vigor and your courage and the idea is that you want to find your home before you run out of Courage there's 16 scenes in a 4x4 grid and so if you play a few times you will have seen them all now my favorite part of making this game was generating each scene and just seeing what AI would come up with and I thought wouldn't it be cool to share that experience with the player what if every time they went to a new Scene It was generated fresh for them and every game would be unique and different this way it would be a game of infinite exploration that sounded so cool that I wanted to try to do it now the first thing that I would need to do is to generate each scene and have a consistent way of doing that my scene definitions are Json objects that describe what the scene is is when you first find it as well as when you come back to it later and how that impacts your stats so I started out by using open ai's completion endpoint and doing some prompt engineering this is the prompt that I used this is a very detailed prompt it's rather long but it worked really well most of the time I I would get scenes that had the right Json format and the content was good it was fitting it was varied it was interesting so I was happy with this but I wanted to make it even more reliable and I decided to fine-tune a model I use open ai's fine-tuning endpoint and they recommend 50 to 100 examples I ated 50 examples just like these and use them to fine tune now the key is I shortened The Prompt I simplified it I took out any of the Json and just generally described what I wanted hoping that that information would be embedded in the training data and I tried this out I wasn't sure if it would work and I tried it it only cost about a dollar or two that includes generating all the examples and doing the fine tuning and when I tried it I was very happy to find that it worked perfectly even though I didn't mention the Json at all it came out perfect because of what was in the examples and that meant I had less tokens in the prompt which is faster and cheaper and just easier to work with so I was really pleased with how this worked the next step was to make the images now I used a tool called Leonardo Leonardo not only lets you generate images they also let you create your own image models and this is great for a game because it means that you can have stylistically consistent images which is exactly what I needed so I spent a while using all the different parameters that Leonardo offers and working with the prompt to try and find an image that looked right and that I liked it turned out that using the description directly from the scene as the prompt made nice pictures which I was surprised about since it had like second person and said things other than what was in there but it worked out great now the tricky part with fine-tuning an image model is that you need consistent images that have like the parts that should be the same are the same in all of your training data but the parts that you want to vary need to be varied otherwise it will overfit and all of your images will look the same but if you don't have that consistency between them then it won't really know what you want and you won't get that good stylistic consistency this was really tricky especially in my case I needed the perspective and the scale to be consistent from scene to scene obviously I needed them all to be set in the forest and I wanted to have this overall tone and texture that look the same some of my scenes have people in them some have animals some have buildings some have nothing and so it was hard to get that variety I ended up having to train a couple couple of models with different parameters different sets of images but I eventually found one that worked out and to test it out I generated a lot of images I mean a whole bunch and you can see they all have similar features like the zigzag path down the middle obviously the trees and the look and everything looks the same and yet there's plenty of variety each one is unique and different but still feels cohesive which I am very pleased about so now I had everything I needed to put it together and make the game I made a simple asset server that had an AI pipeline starting by requesting a new scene from open AI endpoint using my custom model once I get that I validate the Json to make sure that it's got all the keys it needs if it's good I take the description and I send that to Leonardo Leonardo makes an image from my custom model gives it back to me I put it all together and send it off now did this work well let me show you here is an example scene that was created and I'm very happy with it I made a simple preview server so that I could scroll through a bunch of these scenes that I generated to make sure they worked and it looked good so I made some changes to the game to request images each time the player went to a new scene now there was a problem here it takes 10 20 sometimes 30 seconds to do this and that wouldn't be good for the play experience so what I did is I added some caching I prefill a bunch of these scenes and then as scenes are taken out of it I fill it back up again once it gets below a certain threshold and that way there's always a scene that's ready to go with that the game was ready and I'm going to share it with you right now now keep in mind everything that we see has never been seen before and will never be seen again so this is the game you always start out at this lamp poost and you have to wander around and find your way home your stats are in the bottom left corner as your Vigor goes down your speed goes down as well and as the courage goes down the viewport will get smaller and smaller let's look around and explore we're going to move down and here's the first generated scene this looks really cool this is like a uh you encounter a soft blue pulsating light coming from the organic formation scattered around the Glade your fear and tiredness lift and you feel rejuvenated and the Viger goes up but I'm already at full so that's really cool let's head off in this direction now I won't read all of these but this looks like a cool like campfire scene which is really neat and I'm going to head down and what have we got here there's a a large dark cave over here at the end of the path somewhere and it's it's daunting so my courage is going down let's head this way instead and now we've gotten into some fog foggy trees and hard to see let's go back uh this is like a really windy road that we're going through let's head down oh I'm back where I started well this is the game and it would continue on and on and on until you found your way home and then you can just play again and it would be different every time that's great I just have a few closing thoughts one thing is that these images are low resolution they're 52 pixels and I could make them a higher resolution by adding an AI upscaler to my pipeline it would add more time so it's a trade-off also I could get more creative with adding something to the prompt to make a scene for example I could let the user select a theme or maybe even get the time of day or the current weather at the location of where the user is set and then the scenes could be generated to match where they are for a very immersive experience and of course I can use this same process on other projects that's all I hope that you found this interesting and enjoyed watching it as much as I enjoyed putting it all together thank you so much hello and welcome to my talk on how we're thinking about the levels of code AI my name is OT cookit and I am the director ctor of devil at sourcegraph at sourcegraph we're building Cody the only AI coding assistant that knows your entire code base to help educate our customers and users as well as shape our thinking of code AI we've been using a concept that we call levels of code AI internally these levels have really resonated with our community so we wanted to publicize them and start a conversation with the broader developer community and we're better to do it than at the AI engineer Summit when we talk about code AI we refer to software that builds software today 92% of developers are using code aai tools whereas this number was just 1% a year ago our founder and CEO Quinn slack has shared a bold prediction that in 5 years 99% of code will be written by AI while we await that Future Let's talk about how we see the levels of code AI today we see six distinct levels across three different categories human initiated where humans are the primary coders AI initiated where AI starts to take a proactive role in software development an AI Le code where AI has full autonomy over a code base we'll contrast these levels of code with the SAE levels of autonomy for vehicles let's dive in at level zero the developer writes all code manually without any AI assistance the developer is responsible for writing testing and debugging a codebase AI does not generate or modify any part of the code base but IDE features like symbol name completion can provide a bit of assistance this level reflects the traditional software development process before introducing any AI assistance into the develop vment workflow a vehicle operating at level zero is fully reliant on the human driver for acceleration steering braking and everything in between at level one the developer begins to use AI that can generate single lines or whole blocks of code based on developer intent for example a developer might write the signature of a function and the AI will infert the context and generate the implementation details for set function at level one the AI assistant has been trained on millions of lines of open- source code and can leverage this to provide Superior completions based on the developers guidance SAE level one Vehicles still require the full attention of the human driver but offer features such as cruise control or Lane centering that make driving an easier safer and more comfortable experience at level two the AI coding assistant has Superior understanding and context of the codebase it is interacting with where at level one the context is Broad and general a level two AI coding assistant has specific context about the code base that it is working in this allows the AI assistant to make better suggestions for code completions for example if you were working in a node.js code base and were using the axom library to handle HTTP requests a level two AI assistant would provide autocomplete suggestions based on the axium library as opposed to a different node HTTP Library like fetch or super agent at SAE level two we get partial automation the human driver is still in control and can override anything the car does at any time but features such as traffic aware cruise control or automatic Lane changes can make driving a much smoother experience at level three the developer provides high level requirements and the AI assistant delivers a code-based solution the AI coding assistant goes beyond generating singular Snippets of code to building out full components and even Integrations with other pieces of software rather than writing the code themselves a developer could instruct a level three code a assistant to add a user authentication to an application that they are building and the coding assistant would generate all of the code required the coding assistant could then explain to the devel Vel ER decoded wrad how it works and how it integrates with the rest of the application SAE level 3 is also the first level where the vehicle itself takes on the primary role of driving with the human driver being a fallback in case the vehicle cannot drive itself safely the vehicle can perform most of the driving tasks but may encounter situations where it cannot adequately perform these tasks so it's forced to give control back to the human driver at level four the code AI assistant can proactively handle coding tasks without developer oversight let's imagine a few scenarios where a level four code AI assistant would play a role a level four capable code AI assistant could continuously monitor your code changes and autonomously submit PRS to ensure your documentation stays up to date even better the coding assistant could monitor bug reports from customers and submit PRS to fix the issues the human developer could then simply review the pull requests and merge them level four SAE Vehicles can perform virtually all driving tasks under specific conditions for example weo operates a fleet of fully automated self-driving taxis in cities where they have high quality mapping data and can provide a safe driving experience for passengers without human drivers a customer simply hails a wayo taxi using a mobile app provides a dist destination and the vehicle is responsible for taking the passenger to their final destination without any additional human input at level five the AI assistant requires minimal human guidance on code generation and is capable of handling the entire software development life cycle the developer provides high level requirements and specifications the AI then designs the architecture writes production quality code handles deployment and continuously improves the code base the developer's role is to validate that the end product meets the stated requirements but the developer does not necessarily look at the generated code the code AI assistant has complete autonomy to take code from concept to production a self-driving car capable of level five driving automation can perform all driving tasks under all conditions humans optional the car is responsible for for making all the decisions at this level a steering wheel or any ability for a human to override the car is unnecessary so there you have it the six levels of code AI are at least how we're thinking about them at source graph do you agree disagree we'd love to hear your thoughts find us at Booth G5 and let's chat and if you'd like to try Cody for yourself get it for your IDE of choice at cody. deev thank you and I'll see you on the show floor hey I'm MAA and I will show you how we created a GPT powered full stack web generator and how it was used to create over 10,000 applications in one month so first we'll see what it is and then secondly we'll check out how it works under the hood so let's get started so everything happens on this web page and it's super simple first we have to enter the name of our application let's say we are building a simple to-do app second part is describe how it works in a couple of sentences so we have a simple to-do app with one page listing all the tasks user can create tasks change them toggle them edit them creativity level corresponds to GPT temperature so we can go on the safe side and get less features or we can go a little bit crazy but also have more mistakes so I will stick with this balanced one and the last thing left to do is just to hit this generate button B here we can see the result of the generation so we got a full stack app in react node.js Prisma and it's all glued together with a full stack framework wasp so the secret of wasp is that it relies on this single configuration file which describes your app in a high level declarative manner so here for example we can see our o in just a couple of lines our routes Pages our data model so everything is here and still here we can see our client code for example here's our react and here we have our node.js function which are being executed on the back end so the last thing to do is just to download this app locally and run it with vosp so let's do that so I downloaded the app locally and now we just have to run it via V start and there we have it we can log in with usern and password now let's create a couple of tasks Mage become an AI engineer Market is done and now let's check it out to the database so we have a database inspector that also comes with wasp and here we can see two rows one for each task and both are completed he's done is true so let's try adding another task have fun and let's check it out in the data datase again we see it's not completed it's done is false but if we complete it we can see now it's true so there it is we got a fully working full stack web app in react node.js and wasp you can also now deploy this app with a single CLI command it's a regular react at no JS app so you can deploy it virtually anywhere but we have a CLI helper in vosp that makes it super easy to deploy to fly.io and more provid are coming soon this is one of my favorite features when we got Mage out it was hardly the first AI coding agent but it was among the first ones that could generate a full stack web app with almost no errors when we released this and people started using it we were getting two main questions how come this works so well and secondly how can you offer it for free is it that cheap so let's answer them one by one there are three main reasons for M's performance first it is specialized only for full stack web apps and nothing else only react node.js and wasp that allows us to assume a lot up front and makes everything easier and faster second it makes use of a high level web framework wasp that takes away a ton of boiler plate and makes it much easier for GPT to do its job and lastly Mage fixes the errors before it gives you the final result again because of the two points I mentioned previously this is also a simpler problem than for the general AI coding agents let's dive a bit deeper let's go back to our generated app since Mage knows we are building a full stack web app and it's using V for it we can produce a lot of code up front without even touching the open AI API and asking GPT any questions for example some of the config files then also some of the authentification logic which we can see right here and Global CSS and similar so we call this Step Zero only then the code agent takes over the code agent's work consists of three main phases planning generating the code and fixing the errors so let's expand the generation log and explore each of the cases here following the Step Zero we can see the planning phase given our app description Mage devic needs to generate the following queries and actions entities or data models and one page after that the generation step happens Mage is actually implementing everything in planned for above and finally here comes the error fixing phase Mage can detect some of the common errors and fix it for itself here it failed to fix so it had to try again and fin finally when it cannot detect any more errors we are done we can also see that all this took about 27,000 tokens the cool thing is that while developing Mage we identify the most common errors it consistently kept making like mixing up the default and named Imports some of them we even ended up fixing it with a simple her ristic without involving GPT that took care of 90% of all errors again V framework with its high level configuration was of great help here since it removed the ton of code and reduced the space for errors significantly now let's take a look at another question we had how much does it all cost typical app we created with Mage took about two to three minutes and 25 to 60,000 tokens which comes to about 10 to 20 cent but there is one trick we used we use GPT 3.5 and GPT 4 interchangeably for different stages and that reduce the bill a lot if we use only GPT 4 for everything the cost would have been 10x more so $1 to $2 per app what we did is we used GPT 4 only for the planning stage which is the most complex step and one that requires the most creativity for the actual implementation we could comfortably use GPT 3. .5 which is both faster and cheaper and that were great again the key here is that he provided a highly guided environment for decoding agent given vp's web abstractions and that's why this approach worked this is also the main difference between Mage and the other coding agents we tried another popular agent that uses the more free approach and relies more on the GPT itself and the cost to make a similar app as we did with Mage was between 80 cents and $10 so what should you use Mage for and what should you expect is it going to magically produce any app you image or do you still have to put some work in at current stage Mage serves as a really good and highly customized cross starter for full stack web apps at that level it can operate with almost no or very little errors that you can easily detect and fix most of people that tried it found it as a super easy way to get her app kickstarted with the mainstream pieces of Stack such as react node and tailwind and that's how Mage got its popularity I personally believe this is what the future of s starters look like looks like tailor to your app instead of starting out with a generic boiler plate as you would expect the more you push it more errors it starts making on the other hand not giving enough information and just saying something like make Facebook but yellow can also be counter effective so what comes next we created Mage as an experiment to see how well it can produce use full stack web apps with fosp and it works surprisingly well the current main limitation of Mage comes from its Simplicity and the fact there is no interaction with the user beyond the initial prompt so that's something we are looking to add next a live debugging mode where you can while still on the web page interact with the agent and request changes and error fixes another thing that would be interesting to explore would be using an llm that is fine tuned for vosp and web development although that would also make it more expensive also since vosp has such simple and humor readable syntax it's hard to predict how much benefit would fine tuning bring still it would be a cool thing to try out and that's it we saw what Mage was how it works and what is the secret source that made it both fast and affordable to create PB apps so thanks so much for watching I had a lot of fun making this video with my helper and I hope you also found it interesting please give Mage a try and let us know how it went we are the same team that created which is a fully open- Source web framework that makes it super easy to develop with react and no GS also check out our repo and join our Discord for any questions and comments thank you [Music] [Music] oh [Music] I was watching you watch the sun come up vintage t-shirt and warn through High Times these nights taste like gold sweet with Obsession show me something as each morning comes we we out the night like we wear our clothes dancing right through the fire while we watch it close singing r as we give up our go as a new morning comes through the windows riding all newes through the P ladies and gentlemen we are starting now please take your seats clo like super M like super heroes it's coming over now it's away CR down a Harmony of peace that only way can hear super Crush you want to feel like it's star in America under your influence a full moon wax in now I couldn't see it until you showed me how feels like we're insane we blame it all on love so saturated so we can't get enough we were out the night like we wear our clothes dancing right through the fire while we watcho singing our Anthem we get up our gos as a new evening comes through the window it's coming over it's w down a Harmony of that only we can hear super CR you want to feel like us it's St forever so in America it's coming over me electric SYM every night on fire I KN on master a super sonic CR you want to feel it it's forever you're in [Music] America don't hold back tonight is all we have the sky is going black so come with us don't hold back tonight is all we have the sky is going black so come with [Music] us don't hold back tonight is all we have the sky is going so come with us don't hold back tonight is always we ladies and Gentlemen please welcome to the stage the CEO and co-founder of Lang chain Harrison [Applause] Chase thank you guys for having me and and thank you guys for being here this is this is maybe one of the most famous uh screens of of 2023 and and yet I believe and I think we all believe and that's why we're all here that this is just the beginning of a lot of amazing things that we're all going to create because as good as chat GPT is and as good as the language models that underly them are by themselves they're just the start by themselves they don't know about current events they cannot run the code that you write and they don't remember their previous interactions with you in order to get to a future where we have truly personalized and actually helpful AI assistants we're going to need to take these language models and use them as one part of a larger system and that and that's what I think a lot of us in here are are trying to do these systems will be able to produce seemingly you know amazing and magical experiences they'll understand the appropriate context and they'll be able to reason about it and respond appropriately at Lang chain we're trying to help teams close that gap between these magical experiences and and and the work that's actually required to get there and we believe that behind all of these seemingly magical product moments there is an extraordinary feat of engineering and that's why it's awesome to be here at the AI engineering Summit and so I'm going to talk a little bit about some of the approaches that we see work for developers when they're building these context aware reasoning applications that are going to power the future so first I'm going to talk about context and when I say context I mean bringing relevant context to the language model so that it can reason about what to do and bringing that context is really really important because if you don't provide that context no matter how good the language model is it's not going to be able to to figure out what to do and and and so the first type of context and probably the most common type of context that we see people bringing to the language model we see them bringing through this instruction prompting type of approach where they basically tell the language model how to respond to specific scenarios or specific inputs um this this is pretty straightforward and I think the way to think about it is if you have a new employee who shows up on the first day of work you give them an employee handbook and it tells them how they should behave in certain scenarios and and equivalate that to kind of like this instruction prompting technique it's um you know it's pretty straightforward I think that's why people start with it and as the models get better and better this zero shot type of prompting is going to be able to carry a lot of the relevant context for how you expect the language model to behave there's some cases where telling the language model is actually quite hard and it becomes better to give it some few shot examples it becomes better to give it examples where you show the language model how to behave rather than just tell it how to behave and so I think a few concrete places where this works is where it's actually a little bit difficult to describe how exactly the language model should respond so tone I think is a good use case for this and then also structured output is a good use case for this you can give examples of the of the structured output format you can give examples of the output tone a little bit more easily than you could describe in language my particular tone the structured output is a little bit you can describe structured output but I think as it starts to get more and more complicated giving these really specific examples can help the next type of context is maybe the most um you know it pops to the Mind most when you hear of context and when you hear about bringing context to the language model contrasting this with the first two retrieval augmented generation uses context not to decide uh how to respond but to decide kind of like what to base its response in so the kind of like canonical thing is you have a user question you do some retrieval strategy you get back some context you pass that to the language model and you say answer this question based on the context that's provided to you um and so this is a little bit different from the instructions it's it's maybe the same as asking someone to take a test with uh like an open book test you can look at the book you can look at the answers and in this case the answers are the text that you pass in to this context and and then the fourth way that we see people providing context to language models is through fine tuning so updating the actual weights of the language model um this is still kind of like in its infancy and I think we're starting to figure out how best to do this and what scenarios this is this is good to do in um one of the things that we've seen is that this is good for the same use cases where F shot examples are kind of good it takes it to another extreme um and so for tone and structured data uh parsing these are two use cases where we've seen it pretty beneficial to start doing some fine tuning and and the idea here is that yeah it can be helpful to have three examples of how your model should respond and what the tone there should be but what if you could give it 10,000 examples and it updates its weight accordingly and so I think for those where where the output is in a a a specific format and again you need more examples you need to show it a lot more than you can tell it this is where we see fine tuning starting to become helpful and I think we'll see that grow more and more over time so we've talked about Conta um and now I want to talk a little bit about the reasoning bit and I think this is the most exciting and the most new bit of it as well and so we've tried to think and categorize some of the approaches that we've seen to allow uh these applications to do this reasoning component and and so we've listed a few of them out here and tried to tried to discern a few different axes along which they kind of vary so if we think about kind of like just plain old code this is kind of like the way things were you know in like a year ago so a long long time ago um and so so in code you kind of like you you it's all there it's declared if it says what to run um it says what the outputs are what what steps to take things like that we start adding in a language model call and so this is like the the simplest form of of these reasoning applications and here you're using the language model to determine what the output should be but that's it you're not you're not using it to take actions yet nothing fancy you're just using it to determine what the output should be and it's just a single language model call so you're providing the context and and then you're uh returning the output to the user if we take it up a little bit then we start to get into a chain of of language model calls or a chain of language model call to API back to language model and so this can be us this is again used to uh decide the the the steps of the output um and here there's uh multiple calls that are happening and this can be used to break down um more complex task into individual components it can be used to insert knowledge dynamically in the middle of kind of like one language model call then you go fetch some knowledge based on that language model call and then you do another one um but importantly here the steps are known you do this and then you do this and then you do this and so it's a chain of events and that starts to change a little bit when you use a router um and and so in here you're Now using the language model call to start determining which steps to take so that's the big difference here it's no longer just determining the output of the system but it's determining which steps to take and so you can use it to determine uh which prompts to use so route between a prompt that's really good at math problems versus a prompt that's really good at at writing English say you can use it to route between uh language models so one model might be better than another you might want to use CLA because of its long context window or you might want to use GPD 4 because it's really good at reasoning and so having the language model look at the question and decide whether it needs to reason or whether it wants to respond in a long form fashion you can determine which branches to go down or I think uh another common use case is using it to determine which of several tools to take so do I want to call this tool or do I want to call this to and what what should the input to that tools be and so we have this router here and I think um before going on to the next step the the main thing here that distinguishes it from that step is there's no kind of like Cycles you don't kind of get these these Loops um you're just choosing kind of like which branch to go down once you start adding in these Loops this is where we see uh a lot more complex applications um this is this is these are things that we often see being called agents kind of like out out in the wild and it's essentially kind of like a w Loop um and then in that Loop you're doing a series of steps um and the language model is determining which steps to do and then at some point there's a point where it can choose whether to end the loop or not um and if it ends the loop then you finish and return to the user otherwise you go back and and and continue the loop and so here you get the language model deciding what the outputs are it decides what steps to take and you do have these these Cycles um the last thing um and I think the the this is this is largely what we would describe as kind of like what Auto GPT did that took the World by storm is this idea of an agent um where you kind of like remove a lot of the um the the kind of like guard rails around what steps to take so here the sequences of steps that are available are are almost like determined by the llm and what I mean by this is that here's where you can start doing things like adding in tools that the that the language model can take so if you guys are familiar with the Voyager paper it it starts adding in tools and building up a skill set of tools over time and so some of the actions that the language model can take are are dynamically created um and then I think the other big thing here is that you remove some of the scaffolding from the state machines um so some of the uh if I go back a little bit so a lot of the these kind of like cycles that we see in the while break things down into discret States the most common one that we see are kind of like plan execute and validate so you ask the language model to plan what to do it then goes do it and then you validate it often with a language model call or something like that and I think the big difference between that and then the autonomous agent style thing is that here you're implicitly asking the agent to do all of those things in one go it should know when it should plan it should know when it should uh validate and it should know when it should kind of like determine what action to take and you're asking at all to do that implicitly you don't have these kind of like distinct sequences of steps laid out in the code and so this is a little bit about how we're thinking about it I think the the thing to uh the thing that I like to say when saying this as well which goes back to the beginning is that the main thing that we think is it's still just extremely early on in the space we still think it's the beginning and this could you know in in 3 months be kind of irrelevant as the space progresses so I would just keep that in mind if we think about kind of like some of the magical experiences like this where it can reason over the relevant context what is it going to take to kind of like build it under the hood what is the engineering that's going to go in to all these seemingly magical experiences and so this is an example of of what could be going under the hood of something like this it's going to be a challenging experience to build these complex systems and that's why we're building some of the tooling like like this what you see here to help debug understand and iterate on these systems of the future and so what exactly are the challenges associated with building these complex context aware reasoning applications the first is kind of just the orchestration layer so figuring out which of the different reasoning kind of like cognitive architectures you should be using should you be using a simple chain should you be using a router a more complex agent and I think the thing to remember here is that it's not necessarily that one is better than the other or superior to the other they all have kind of like their pros and cons and strengths and weaknesses so chains are really good because you have more control over the sequence of steps that are taken agents are better because they can more dynamically react to unexpected inputs and handle edge cases and so being able to choose the the right cognitive architecture that you want and being able to quickly experiment with a bunch of other ones are part of what inspired the initial release of Lang chain and and and kind of how we aim to help people uh prototype these these types of applications and then Lang Smith which is this thing here provides a lot of visibility into what is actually is going on as these uh as these applications start to get more and more complex understanding what exact sequences of of tools are being used what what exact sequences of language model calls are being made becomes increasingly important another big thing that we see people struggling with and spending a lot of time on is good old fashioned data engineering so a lot of this comes down to providing the right context to language models and the right context is often data and so you need to have ways to load that data you need to have ways to transform that data to transport that data and then you often want to have observability into what exact data is getting passed around and where and so laying chain itself has a lot of Open Source kind of like modules for loading that data and transforming that data and then Lang Smith we often see being really useful for debugging what exactly does that data look like by the time it's getting to the language model have you extracted the right documents from your vector store have you transformed them and formatted in the right way where it's clear to the language model what's actually in them these are all things that you're going to want to be able to to debug so there's no little small errors or small issues that pop up and then the the third thing that we see a lot of people spending time on when building these applications is good old fashioned prompt engineering um so the main new thing here is language models and the main way of interacting with language models is through prompts and so being able to understand what exactly does the fully formatted prompt look like by the time it's going into the language model is is really important what like how are you combining the system instructions with maybe the few shot examples um any retrieved context the chat history that you've got going on any previous steps that the agent took what does this all look by the time it gets to the language model and and and what does this look like in the middle of this complex application so it's easy enough to to kind of like test and debug this if it's the first call the first part of the system but after it's already done three of these steps if you want to kind of like debug what that prompt looks like what that fully formatted prompt looks like being able to do that um it becomes increasingly difficult as the systems kind of like scale up in in their entagled and so we've tried to make it really easy to to hop into any kind of like particular language model call at any point in time open it up in a playground like this so you can edit it directly and and and experiment with that prompt engineering and and and go kind of like uh change some of the instructions and see how it responds or swap out model providers so that you can see if another model provider does better another big challenge with these language model applications and is probably worth a talk on its own is evaluation of them and so I think evaluation is really hard for a few reasons I think the two primary ones are a lack of data and a lack of good metrics so comparing to traditional kind of like data science and machine learning with those you generally started with a data set you needed that to build your model and so then when it came time to evaluate it you at least had those data points that you could look at and evaluate on and I think that's a little bit different with a lot of these um llm applications because these these models are fantastic zero shot kind of like Learners that's the that's kind of like the the whole exciting bit of them and so you can get to a working MVP without building up kind of like any data set at all and that's awesome but that does make it a little bit of a challenge when it comes to evaluating them because you don't have these data points and so one of the things that we often encourage a lot of people to do and try to help them do as well is build up these data these data sets and iterate on those and those can come from either labeling data points by hand or looking at production traffic and pulling things in um or autogenerating things with with llms the second um Big Challenge in evaluation is lack of metrics um I think most traditional kind of like quantitative metrics don't perform Super well for for large unstructured outputs a lot of what we see people doing is still doing a kind of like Vibe check to kind of like see how the model is performing um and and as unsatisfying as that as that is I still think that's probably the best way to gain uh kind of like um intuition as to what's going on and so a lot of what we try to do is make it really easy to observe the outputs and the inputs of the language model so that you can build up that intuition um in terms of more quantitative and systematic metrics we're we're very bullish on llm assisted evaluation so using llms to evaluate the outputs um and and then I think maybe the the biggest thing that we see people doing in production is just keeping track of feedback um whether it be direct or indirect feedback so do they leave kind of like a thumbs up or a thumbs down on your application that's an example of direct feedback where you're Gathering that an example of indirect feedback might be if they click on a link or that that might be a good thing that you provided a good suggestion or if they if they respond really confused to your chatbot that might be a good indication that your uh jabot actually did not perform well and so tracking these over time and and doing AB testing with that using kind of like uh traditional AB testing software can be can be pretty impactful for Gathering a sense online of how of how your model is doing and then the last interesting thing that we're spending a lot of time thinking about is collaboration so as these systems get bigger and bigger they're doubtless going to be a collaboration among a lot of people um and so who exactly is is working on these systems um is it is it all AI Engineers as we're here today is it a combination of AI engineers and data engineers and data scientists and and product managers and I think one of the interesting trends that we're seeing is it's still a little bit unclear what the best skill sets for this new AI engineer type role is and then there could very well be a bunch of different skill sets that are valuable so going back to kind of like the two things that we see making up a lot of these applications the context awareness and the reasoning bit the context awareness is bringing the right context to these applications you often need kind of like a data engineering team to get in there and assist with that the reasoning bit is often done through prompting and often times that's done by non-technical people who can really outline kind of like the exact specification of the app that they're building whether they be product managers or subject matter experts and so how do you enable collaboration between these two different types of folks and and what exactly does that look like I don't think that's something that anyone kind of like knows or definitely hasn't solved but I think that's a really interesting Trend that that we're thinking a lot about going forward and so I think like the the the main thing that I want to leave you all with is that the big thing that we believe is that it's still really really early on in this journey it's just the beginning as crazy as things have been over the past year they're hopefully going to get even crazier you saw an incredible demo of of GPT 4V things like that are going to change it and so we think behind all these things it's going to take a lot of engineering and we're trying to build some of the tooling to help enable that and I think you guys are all on the right track towards becoming those types of engineers by being at a conference like this so thank you swix for having me thank you guys for being here have a good rest of your [Applause] day please welcome our next speaker the founder of 567 Jason [Music] Li hey guys so I didn't know I was going to be one of the keynote speakers so this is probably going to be the most reduced scope talk of today I'm talking about typ hints and in particular I'm talking about how pantic might be all you need to build with language models in particular I want to talk about structured prompting which is the idea that we can use object to Define what we want back out rather than kind of praying to the llm gods that the comma is in the right place and the bracket was closed so everyone here basically kind of knows or at least agrees that large language models are kind of eating software but what this really means in production is 90% of the applications you build are just ones when you're asking the language model to Output Json or some structured output that you're parsing with a regular expression and that experience is pretty terrible and the reason this is the case is because we really want language models to be backwards compatible with the existing software that we have you know code gen works but a lot of the systems we have today are systems that we can't change and so yeah the idea is that although language models were introduced to us to chat gbt most of us are actually Building Systems and not chat Bots we want to process input data integrate with existing systems via apis or schemas that we might not have control over and so the goal for today is effectively introduce open AI function calling introduce pantic then introduce instructor and Marvin as a library to make using pantic to prompt language models much easier and what this gets us is uh you know better validation makes your code a little bit cleaner and then afterwards I'll talk over some design patterns that I've uncovered and some of the applications that we have um this is basically almost everyone's experience here right like you know Ry Goodside had a tweet about asking to get Json out of Bard and the only way you could do it was to threaten to take a human life and that's not code I really want to commit into my repos and then when you do ask for Json you know maybe it works today but maybe tomorrow instead of getting Json you're going to get like okay here you go here's some Json and then again you kind of pray that the Json parsed correctly and I don't know if you noticed but here user is a key for one query and username is a key for another and you would not really notice this unless you had like good logging in place but really this just not happening to begin with right like you shouldn't have to like read the logs to figure out that the passwords didn't match when you're signing for an account and so what this means is our prompts and our schemas and our outputs are all strings we're kind of writing code and text edit rather than IDE where you could you know get linting or typechecking or syntax highlighting and so open a function calls somewhat fix this right we get to Define Json schema of the output that we want and open AI will do a better job in placing the Json somewhere that you can reliably parse out so instead of going from string to string to string you get string to dict to string and then you still have to call Json loads and again you're kind of praying that everything is in there and a lot of this is kind of praying through the LM Gods um on top of that like if this code was committed to any repo I was managing like I would be pissed right complex data structures are already difficult to Define and now you're working with the dictionary of Json loads and that also feels very unsafe cuz you get missing Keys missing values and you got hallucinations and maybe the keys are spelled wrong and you're missing an underscore and you get all these issues and then you end up writing code like this and this works for like name and age and email then you're checking if something is a bull by parsing a string it gets really messy and and what python has done to solve this is use pantic pantic is a library that do data model validation very similar to data classes it is powered by typ pints it is has really great model and field validation it has 70 million downloads a month which means it's a library that everyone can trust and use and know that it's going to be maintained for a long period of time and more importantly it outputs Json schema which is how you communicate with open AI function calling and so the general idea is that we can define an object like delivery say that a time stamp is a date time and the dimensions is a toule of in and even if you pass in a string as a Tim stamp and a list of strings as tles everything is parsed out correctly this is all the code we don't want it write this is why there's 70 million downloads more interestingly time stamp and dimensions are now things that your IDE is aware of they know the type of that you get autocomplete and spell checking again just more bug free code and so this really want brings me to the idea of structured prompting cuz now your prompt isn't a you know triple quoted string your prompt is actual code that you can look at you can review and everyone has written a function that returns a data structure right everyone knows how to manage code like this instead of doing the migration of Json schemas and the one shot examples you know I've done database migrations I know how some of these things work and more we can program this way and so that's why I built a library called instructor a while ago and the idea here is just just to make open a function calling super useful so the idea is you import instructor you patch the completion API uh debatable if this is the best idea but ultimately you define your pantic object you set that as the response model of that create call and now you're guaranteed that that response model is the type of the entity that you extract so again you get nice auto complete you get type safety really great I would also want to mention that this only works for open AI function calling if you want to use a more a comprehensive framework to do some of this pantic work I think Marvin is a really great uh library to try out uh they give you access to more uh language models and more capabilities above this uh response but the general idea here isn't that this is going to make your Json come out better right the idea is that when you define objects you can Define nested references you can Define methods of the behavior of that object you can return instances of that object instead of dictionaries and you're going to write cleaner code and code that's going to be easier to maintain as they're passed through different systems and so here you have for example a base model but you can add a method if you want to you can Define the same class but with an address key you can then Define new classes like best friend and friends which is a list of user details like if I was to write this in Json schema to make a post request it would be very unmanageable but this makes it a lot easier easier on top of that when you have doc strings the doc strings are now a part of that Json schema that is sent to open Ai and this is because the model now represents both the prompt the data and the behavior all in one right you want good doc strings you good want you good you want good field descriptors and it's all part of the Json schema that you send and now your code quality your prompt quality your data quality are all in sync there's this one thing you want to manage and one thing you want to review and what that really means is that you need to have good variable names good descriptions and good documentation and this is something we should have anyways you can also do some really cool things with pantic without language models for example you can define a validator here I Define a function that takes in a value I check that there is a string in that value and if it's not I return a lowercase version of that because that just might be how I want to par parse my data and when you construct this object you get an err back out right we're not going to fix it but we get a validation error something where we can catch reliably and understand but then if you introduce language models you can just import the llm validator and now you can have something that says like don't say mean things and then when you construct an object that has something that says that the meaning of life is the evil and steal things you're going to get an validation error and an error message and this error message the statement is objectable is actually coming out of a language model API call is using instructor under the hood to Define that but you know it's not enough to actually just point out these errors you also want to fix that and so the easy way of doing that in instructor is to just add Max retries right now what we do is we'll append the the message that you had before but then we can also capture all the validations in one shot send it back to the language model and try again right but the idea here that this isn't like prompt chain this is this isn't constitutional AI here we just have validation error handling and then reasing and these are just separate systems in code that we can manage if you want something to be less than 10 characters there's a character count validator if you want to make sure that a name is in a database you can just add a post request if you want to but this is just classical code again this is the backwards compatibility of language models but we can also do a lot more right uh structured prompts get you structured outputs but ideally the structure actually helps you structure your thoughts so here's another example uh it's really important for us to give language models the ability to have an escape PCH and say that it doesn't know something or can't find something and right now most people will say something like return learn I don't know in all caps check if I don't know all caps in string right uh sometimes it doesn't say that it's very difficult to manage but here you see that I've defined user details with an optional role that could be none but the entity I want to extract is just maybe a user it has a result that's maybe a user and then an error and an error message and so I can write code that looks like this I get this object back out it's a little bit more complicated but now I can kind of program with language models in a way that feels more like programming and less like chaining for example right um we can also Define reusable components here I've Define a work time and a Leisure Time as both a Time range and the time range has a start time and an end time if I find that this is not being parsed correctly what I could do is actually add Chain of Thought directly in the the time range component and now I have modularity in some in some of how in some of these features and you can imagine having a system where in production you uh disable that Chain of Thought Field and then in in in testing you add that to figure out what's the latency or performance trade-offs you could also extract arbitrary values right here I Define a property called key and value and then I want to extract list of properties right you might want to add a prompt that says make sure the keys are consistent over those properties but we can also add validators to make sure that's the case and then Reas when that's not the case if I want you know only five properties I could add an index of the property key and just say well now count them out and when you count to five stop and you're going to get much more reliable outputs uh some of the things that I find really interesting with this kind of method is prompting data structures here I have user details age name as before but now I Define an ID and a friends array which is a list of IDs and if you prompt that well enough you can basically extract like a network out of this data out of your data so you know we've seen that structured prompting kind of gives you really useful components you can reuse and make modular um and the idea again here is that we want to model both the prompt the data and the behavior here I haven't mention too many methods that you could act on this object but the idea is almost like you know when we go from C to C++ the thing we get is object oriented programming and that makes a lot of things easier and we've learned our lessons with object Orient programming and so if we do the right track uh I think we're going to get a lot more productive development out of these language models and the second thing is that these language models now can output data structures right that you can like your old like lead code textbooks or whatever and and actually figure out how to Traverse these graphs for example process this data in a useful way and so now they can represent you know knowledge workflows and even plans that you can just dispatch to a classical computer uh computer system right you can create the dag that you want to send to airflow rather than doing this for Loop hoping it terminates and so now I think about 6 minutes so I'll go over some Advanced applications um these are actually fairly simple I have some more documentation if you want to see that later on but um let's go over some of these examples so the first one is rag I think when we first started out a lot of these systems end up being systems where we Ed the user query make a vetor database search return the results and then hope that those are good enough but in in practice you might have multiple back ends to search from maybe you want to revite the user query maybe you want to decompose that user query right if you want to ask something like what what was something that was recent you need to have time filters and so you could Define that as a data structure right the search type is email or video search has a title a query a before data and a type and then you can just implement the execute method that says you know if type is video do this if email do that really simple and then what you want to extract back out is multiple searches that give me a list of search queries and then you can write some like ASN iyota map across these things and now because all the prompting is embedded in the data structure your prompt that you send to open AI is very simple your helpful assistant segment the search queries and then what you get back out is this ability to just have an object that you can program with in a way that you've managed sort of like all your life right something very straightforward but you can also do something more interesting you can then plan right before we talked about like extracting a social network but you can actually just produce the entire dag here I had the same graph structure right it's an ID a question and a list of dependencies where I have a lot of information in the description here and that's basically the prompt and what I want back out is a query plan so now if you send it to a query planner that says like you're a helpful query planner like build out this query you can ask something like what is the difference in populations of Canada and Jason's home country and then what you can see is you know what like if I'm good at Elite code I could query the first two in parallel because there are no dependencies and then wait for dependency three to merge and then wait for four to merge those two but this requires one language model call and now it's just traditional Rag and if you have an IR system you get to skip this four Loop of agent queries you know an example that was really popular on Twitter recently was extracting knowledge graphs you know same thing here here what I've done is I've made sure that the data structure I model is as close as possible to the graph Vis uh visualization API what that gets me is really really simple code that does basically the creation and visualization of a graph I've just defined things one to one to the API and now what I can do is if I ask for something that's very simple like you know give me the description of quantum mechanics you can get a graph out right that's basically in like 40 lines of code because what you've done is you've modeled the data structure graph is needs to make the visualization and we we're kind of try to couple that a lot more this is a more advanced example so don't feel bad if you can't follow this one but here what I've done is I've done a question answer is a question and an answer and the answer is a list of facts and what a fact is is it's a fact as a statement and a substring quote from the original text I want multiple quotes as a substring of the original text and then what my validators do is it says you know what for every quote you give me validate that it exists in the text Chunk if it's not there throw out the fact and then the validator for question answer says only show me facts that have at least one substring quote from the original document so now I'm trying to encapsulate some of the business logic of not hallucinating not by asking it to not hallucinate but actually trying to figure out like what is the like par like the paraphrasing detection algorithms to to identify what the quotes were and what this means is instead of being able to say that the answer was in page seven you can say the answer was this sentence that sentence and something else and I know they exist in the text junks and so I think what we end up finding is that uh as language models get more interesting and more capable we're only going to be limited in the creativity that we can have to actually these things right like you can have instructions uh per object you can have like recursive structures right it it it goes into domain modeling more than it goes to prompt engineering and again now we can use the code that we've always used if you want more examples I have a bunch of examples here on different kinds of applications that I've had with some of my Consulting clients um yeah I think these are some really useful ones and I'll go to the next slide which is this doesn't have the uh the QR code that's fine the updated slide has a QR code but instead you can just visit jxl github.io instructor I also want to call out that uh we're also experimenting with a lot of different uis to do this structured evaluation right where uh you might want to figure out whether or not one response was mean but you also want to figure out what the distribution of floats was for a different attribute and be able to write evals against that and I think there's a lot of really interesting open work to be done right like right now doing very simple things around extracting graphs out of documents you can imagine a world where we have multimodal in which case you could be extracting bounding boxes right like one application I'm really excited about is being able to say give an image draw the bounding box for every image and the search query I would need to go on Amazon to buy this product and then you can really instantly build a UI that just says you know for every bounding box render a modal right you can have like generative UI over images over audio I think in general it's going to be a very exciting space to play more with uh structured outputs thank [Applause] [Music] you ladies and Gentlemen please join me in welcoming our next guest senior applied scientist at Amazon Eugene Yen thank you [Music] thank you thank you everyone um I'm Eugene Yen and today I want to share with you about some building blocks for llm systems and products like many of you here I'm trying to figure out how to effectively use these llms in production so a few months ago to clarify my thinking I wrote some patterns about building LM systems and products and the Community seem to like it there's Jason asking for this to be seminar so here you go Jason today I'm going to focus on four of those patterns evaluations retrieval AED generation guard rails and collecting feedback all the slides will be made available after this talk so I ask you to just focus buckle up hang on tight because you'll be going really fast all right let's start with evals or what I really consider the foundation of it all why do we need evals well evals help us understand if our prompt engineering our retrieval commentation or our fine tuning is it doing anything at all right consider eval driven develop uh development where evals guide how you build your system and product we can also think of evals as test cases right where we run these evals before deploying any new changes it makes us feel safe and finally if managers had open AI take the time to write evals or give feedback on them you know it's pretty important but building evals is hard here are some things I've seen folk Tri trip up on um firstly we don't have a consistent approach to evals if you think about more conventional machine learning regression we have root means square error classification Precision recall even ranking ndcg all these metrics are pretty straightforward and there's usually only one way to compute them but what about for llms well we have this Benchmark whereby we write a prompt there's a multiple choice question we evaluate the model's ability to get it right MML is a example that's widely used where it assesses LMS on knowledge and reasonability you know computer science questions math US History Etc but there's no consistent way to run MML less than a week ago iin and sash from Princeton evaluating llms is a mindfield they ask are we assessing prompt sensitivity are we assessing the llm or are we assessing our prompt to get the LM to give us what we want on the same day entropic noted that the simple McQ may not be as simple as it seems simple formatting changes such as different parenthesis lead to different changes in accuracy and no one is there's no consistent way to do it to do this as a result it makes it really difficult to compare models based on these academic benchmarks now speaking of academic benchmarks we may have outgrown some of them for example this task of summarization on the top you see the human evaluation scores on the reference summaries and on the bottom you see the evaluation scores for the automated summaries you don't have to go through all the numbers there but the point is that all the numbers on the bottom are already higher than the numbers on top here's another one that's more recent on the X sum data said extreme summarization where you see that all the human evaluation scores are lower than instru GPT and that's not even GPT 4 now finally with all this bench box being so easily available we sometimes forget to ask ourselves hey is it a fit for our task if you think about it does MML really apply to your task maybe if you're building a college level chatboard right but here's lus reminding us that we should be measuring our apps on our tasks and not just rely on academic evals so how do we do evals well I think as an industry we still figure it out bar point out is the number one challenge out there and we we hear so many people talking about evals I think there are some tant emerging firstly I think we should build evals for our specific task and it's okay to start small it may seem daunting but it's okay to start small how small well here's technum you know he releases a lot open source uh models he starts with an evil set of 40 questions for his domain Expert T 40 evals that's all it takes and it can go very far second we should try to simplify the task as much as we can you know while LMS are very flexible I think we have better chance if we try to make it more specific for example if you're using an llm for Content martion task you can fall back to simple precision and recall how often is it catching toxicity how often is it catching bias how often is it catching hallucination next if it's something broader like writing SQL or extracting Json you know you can try and run the SQL and see if it Returns the expected result that's very deterministic or you can check the extracted Json keys and check if the Json keys and the values match what you expect these are still fairly easy to evaluate because we have expected answers but if your task is more open-ended such as dialogue you may have to rely on a strong LM to evaluate the output however this can be really expensive here's Jerry saying you know 60 evals gp4 it cost him a lot finally even if you have automated evals I think we shouldn't discount the value of eyeballing the output here's Jonathan from Mosaic I don't believe that any of these evals capture what we care about they have a prompt to generate games for a three and 7 Y and it was more effective for them to actually just eyeball the output as it trains throughout the EPO okay that's it for evals now retrieval AED generation I don't think I have to convince you all here why we need retrieval AED generation but you know it lets us add knowledge to our model as input context where we don't have to rely solely on the model's knowledge and second is far practical right it's cheaper and precise and continuously fine-tuning to add new knowledge but retrieving the right documents is really hard nonetheless we have great speakers Jerry and Anon sharing about this topic tomorrow so I won't go into the challenges of retrieval here instead I like to focus on the L&M side of things right and discuss some of the challenges that remain even if we have retriev augmented generation the first of all is that LMS can't really see all the documents you retrieve here's an interesting experiment right the task is retriever or meod question and answering you you know historical queries on Google and hand annotated answers from Wikipedia as part of the context they provide 20 documents each of these documents are at most 100 tokens long so that means 2,000 tokens maximum and one of these documents contain the answer and the rest are simply distractors so the question they had was this how would the position of the document containing the answer affect question answering now some of you may have seen this before don't spoil it for the rest if the answer is in the first retrieve document accuracy accuracy is the highest if it's in the last accuracy is decent but if it's somewhere in the middle it's actually worse accuracy than having no retriever the generation so what does this mean it means that even if context window sizes are growing we shouldn't allow our retrieval to get back get worse right getting the most relevant documents to rank highly still matters regardless of how big the context size is and also even if the answer is in the context and in the top position accuracy is only 75% so that means even with perfect retrieval you can still expect some mistakes now another gotcha is that LMS can't really tell if the retrieve context is irrelevant here's a simple example so here's here are 20 top Sci-Fi movies and you can think of this as movies that I like and I ask the llm if I would like Twilight so for folks not familiar with Twilight you know it's romantic fantasy girl vampire werewolf something like that but I I think I've never watched it before so but I have a really important instruction if it doesn't think I would like Twilight because I've watched all these Sci-Fi movies it should reply with not applicable and this is pretty important in recommendations we don't want to make bad recommendations so here's what happened first it notes that Twilight is a different genre and not not quite sci-fi which is fantastic right but then it suggests ET because of inter species relationships I mean I'm I'm not sure how I feel about that uh yeah I mean how would you feel if you got this for a movie recommendation the the point is these LMS are so fine-tuned to be helpful and it's really smart and they try their best to give an answer but sometimes it's really hard to get them to say something that's not relevant especially something that's fuzzy like this right so how do we best address this limitations in rag well I think that there are lot of great ideas in the field of information retrieval search and recommendations have been trying to figure out how to show the most relevant documents on top and I think work really well and there's a lot that we can learn from them second LMS may not know that the retrieve document is irrelevant right I think it helps to include a threshold to exclude irrelevant documents so in the Twilight and sci-fi movie example I bet we could do something like just measuring item distance between those two and if it's too far we don't go to the next step next guard rails so guard rails are really important in production we want to make sure we what we deploy is safe what safe we can look at open AI moderation API hate harassment self harm all that good stuff but another thing that I also think about a lot is gu on factual consistency or we call that huc ations I think it's really important so that you don't have trustb trust buting experiences you can also think of this as evals for hallucination fortunately or unfortunately the field of summarization has been trying to tackle this for a very long time and we can take a leap from that Playbook so one approach to this is via the natural language inference task in a nutshell given a premise and a hypothesis we classify if the hypothesis is true or false so given a premise John like LI or fruits the hypothesis that John likes apples is true therefore it's entailment because there's not enough information to confirm if John eats Apple's daily is neutral and finally John dislikes apple is clearly false therefore contradiction do you see how we can apply this to document summarization the premise is the document and the hypothesis is the summary and it just works now when doing this though it helps to apply it at the sentence instead of the entire document level so in this example here the last sentence in the summary is incorrect so if we run the nli task on the entire document and summary it's going to say that the entire summary is correct but if you run it at a sentence level it's able to tell you that the last sentence in the summary is incorrect and they included a really nice ablation study right where they in they check the granularity of the document as we got finer and finer from document to paragraph to sentence the accuracy of detecting factual inconsistency goes up that's pretty amazing now another approach is sampling right and here's an example from sh GPD given an input document we generate a summary multiple times now we check if those summaries are similar to each other andram overlap Bird score Etc the assumption is that if the summaries are very different it's probably means that they're not grounded on the context document and therefore likely Hall losing anything but if they're quite similar you can you can assume that they're grounded effectively and therefore factual and Final Approach is asking a strong LM you know conceptually simple given an input document and summary they get the LM to return a summary score and this LM has to be pretty strong and we have seen that strong LMS are actually quite expensive um but in the case of factual consistency I've seen similar simple simpler methods outperform LM based approaches at a far lower cost so try to keep things simple if you can okay now to close the loop let's touch briefly about collecting feedback and I'm going to need audience audience help here so why is collecting feedback important because we want to understand what our customers like and don't like and then the magic thing here is that collecting feedback helps you build your evals and find tuning data set new models come and go every day but your evils and fine-tuning data set that's your transferable asset that you can always use so but collecting feedback from users is not as easy as it seems so explicit feedback can be Spar Spar means very low in number and explicit feedback is feedback we ask users for so here's a quick thought experiment how many of you here use chat GPT okay I see a lot of you how many of you here actually click the thumbs up and thumbs down button exent okay but these are the beta beta testers right but you can see it's very small in number so even if you include this thumbs up thumbs down button you may not be getting that the feedback you expect now if the ex if the issue with explicit feedback is sparity then the issue with implicit feedback is noise so implicit feedback is the feedback you get as users organically use your product right you don't have to ask them for feedback but you get this feedback so here's the same example how often do you click the copy code button the rest of you just type it out like a man man like okay so but that's clicking the copy code button mean that the code is correct in this case no n rows is not a valid argument for Pand rep P but if we were to consider all Cote Snippets that were copied as positive feedback we would have a lot of bad data in our training so think about that so how do we collect feedback I don't have any good answers but here are two apps I've seen do it really well first one GitHub co-pilot or any kind of coding assistant right for people not filming with it you type some functional signature some comments and it suggest as code you can either accept the code reject the code move on next suggestion we do this dozens of times a day imagine how much feedback they get from this right here has a golden data set another example is mid Journey for folks not familiar mid Journey you write a pro you write a prompt it suggests four images and then based on those images you can either rerun the prompt you can either vary The Prompt that's what the v stands for or you can either upscale the image uh that's what the U stands for but do you know what an AI engineer sees rerunning the prompt is negative reward where the user doesn't like any of the images varying the image is a small positive reward where the user is saying this one has potential but TW it slightly and choosing the upscale image is large positive reward where the user likes it and just wants to use it so think about this think about how you can building this implicit feedback data flywheel into your products then you quickly understand what users like and don't like oh sorry you can take your phone all slides available after the talk so that's all I wanted to share if you remember anything from this talk I hope it's these three things you need automated evals you need automated evals just annotate 30 or 100 examples and start from there right and then figure out how to automate it it will help you iterate faster right on your promp engineering on your retrieval augmentation on your fine tuning help you deploy safer I mean this is a huge Conference of Engineers I don't I don't think I have to explain to you the need for test testing eyeballing doesn't scale it's good as a final Vibe check but it just doesn't scale every time you update the prompt you just want to run your evals immediately right I run hundreds of I run tens of experiments every day and the only way I can do this is with automated evals second reuse your existing systems as much as you can there's no need to reinvent the wheel bm25 metadata fetching uh metadata matching can get you pretty far and so do the techniques from recommendation systems right right two stage retrieval and ranking filtering Etc all these information retrieval techniques are optimized to rank the most the most relevant items on top so don't forget about that and finally ux plays a large role in LM products I think that a big chunk of GitHub co-pilot and Cat GPT is ux it allowed you to use the llms in your context without calling an API you can useing an ID using a chat window similarly ux makes it far more effective for you to collect user feedback okay that's all I had thank you and keep on [Applause] [Music] [Applause] [Music] building our next speaker is AI lead at notion please welcome lonus Lee we got everything we need and then a little too much I know that you're starving for hey everyone I'm lus I'm here to talk about embedding I'm grateful to be here at the inaugural AI engineer conference who learned something new today yeah um before I talk about that a little about myself if you don't know me already I am lus I work on AI at notion uh for the last year or so before that I did a lot of independent work prototyping experimenting with trying out different things with language models with traditional LLP things like tfidf bm25 to build interesting interfaces for reading and writing in particular I worked a lot with embeding models and Laten spaces of model which is what I'll be I'll be talking about today but before I do that I want to take a moment to say it's been almost year since notion launched notion AI our public beta was first announced in around November 2022 so as we get close to a year we've uh been steadily launching new and interesting features inside notion AI from November we have ai Auto F inside databases translation and things coming soon though not today so keep an eye on the space and obviously we're hiring just like everybody else here we're looking for AI Engineers product Engineers machine learning Engineers to tackle the full gamut of problems that people have been talking about today Asians tool use evaluations data uh training and all the interface stuff that that we'll see today and tomorrow so if you're interested please grab me and uh and we'll have a little chat now it wouldn't be Al talk without talking about lat and spaces so let's talk about it um the the one of the problems that I find always motivated by is the problem of steering language models and I always say that prompting language models feels a lot like you're steering a car from the back seat with a pool noodle like yes technically you have some control over the motion of the vehicle so like there's some connection but like you're not really in the driver's seat the control isn't really there it's not really direct there's like three layers of indirection between you and what the vehicle's doing and that to me trying to prompt a model especially smaller more efficient models that we can use for production uh with just tokens just prompts feels a lot like there's too many lers of IND Direction and as even though models are getting better at understanding prompts I think there's always going to be this fundamental barrier between indirect kind of control of models with just prompts and and getting with the model getting the model to do what what we want them to do and so perhaps we can get a closer layer of control a more direct layer of control by looking inside the model which is where we look at lat and spaces lat spaces arise I think uh most famously inside embedding models if you embed some piece of text that Vector of 1536 numbers or4 numbers is inside a high dimensional Vector space that's a latent space but also you can look at V the Laten SPAC is inside activation spaces of models inside token embeddings inside image models and then obviously other model architectures like Auto encoders today we're going to be looking at embedding embedding models but I think a lot of the general takeaways apply to other models and I think there's a lot of fascinating research work happening inside other models as well when you look at an embedding you kind of see this right you see like rows and rows of numers if you ever debugged some kind of an embedding Pipeline and you print out the embedding you can kind of tell it has like a thousand numbers but it's just looking at like a matrix uh screen of numbers rating down but in theory there's a lot of information actually packed inside those embeddings if you get an embedding of a piece of text or image these latent spaces these embeddings represent in theory the most Salient features of the text or the image that the model is using to lower its loss or do its task and so maybe if we can disentangle some meaningful attributes or features out of these embeddings if we can look at them a little more closely and and and interpret them a little better maybe we can build more expressive interfaces let them control the model by interfering or intervening inside the model another way to say that is that embed show embeddings show us what the model sees in a sample of input so maybe we can read out what it sees to try to understand better what what the model's doing and maybe we can even control the embedding intermediate activations to see uh what the model can generate so let's uh let's see some of that so some of this some of you might have seen before but I promise there's some new stuff at the end so hang tight so here's some sentence that I have it's a sentence about this novel one of my favorite novels named asora it's a science fiction novel by Greg Egan that explores the evolution and existence of postum artificial intelligences uh something to do with alien civilizations and the questioning the nature of reality and Consciousness which you might be doing a lot given all the things that are happening and so so I have I have I've trained this model that can uh generate some embeddings out of this text so if I hit this enter it's going to give us an embedding but it's a it's an embedding of length 248 and so it's it's quite large but it's just a row of numbers right but then I have a A decod or half of this model that can take this embedding and try to reconstruct the original input that may have produced this embedding so in this case it took the original sentence there's some variation you can tell it's not exactly the same length maybe but it's mostly reconstructed the original sentence including the specific detail tells like the title of the book and so on so we have an encoder that's going from text to embedding and a decoder that's going from embedding back to text and now we can start to do things with the embedding to vary it a little bit and and see what the decoder might see if we if we make some mod modifications to the embedding so here I've uh tried to kind of blur the embedding and sample some points around the embedding with this this blur radius and you can see the text it's generated from those blurry embeddings are they're a little off like this is not the correct title the title kind of gone here um the the it still kept the name Greg but it's it's a different person and so there's kind of a blur a semantic blur that's happened here but this is kind of boring this is not really useful what's a little more useful is trying to actually manipulate things in more meaningful directions so now we have the same taste of text and now here I have a bunch of controls so maybe I want to find a direction in this embedding space here I've computed a direction where if you push an embedding in that direction that's going to represent a shorter piece of text of roughly the same topic and so I pick this direction and I hit go and uh it'll try to move push the embedding of this text in that direction and decode them out and you can tell they're a little bit shorter um if I push it a little bit further even so now I'm taking that shorter Direction and moving a little farther along it and and sampling generating text out of those embeddings again and they're they're even a little bit shorter but they've they've still kept the general kind of idea General topic and uh with that kind of building block build really interesting interfaces like for example I can I can plop this piece of text down here and maybe I want to generate a couple of a couple of sort of shorter versions um so this is like a little bit shorter this is even even more short but uh maybe maybe I like this version so I'm going to clone this over here and I'm going to make this the sentiment of the sentence a little more negative and you can start to explore the latent space of of lat and space of this embedding model this language model by actually moving around in a kind of spatial canvas interface which is is kind of interesting another thing you can do with this kind of end model is now that we have a vague sense that there are specific directions in this space that mean specific things we can start to more directly look at a text and ask the model hey where does this piece of text lie along your length direction or along your negative sentiment Direction so uh this is the original text that we've been playing with it's pretty objective like a Wikipedia style piece of text here I've asked CH PT to take the original text and uh make it a lot more pessimistic so uh things like the futile for meaning and plunging deeper into the ab of nism and uh and if I embed both of these what I'm asking the model to do here is embed both of these things in the embedding space of the model and then project those embeddings down onto each of these directions so one way to read this table is that um this default piece of text is at this point in this negative Direction which by itself doesn't mean anything but it's clearly less than this so this piece of text is much further along the negative sentiment axis inside this model when you look at other properties like how how muc