Personal AI Meetup - Bee, BasedHardware, LangChain LangFriend, Deepgram EmilyAI

Key Takeaways

so I wanted to kick things off a little bit with some my personal explanations and then I I'll hand it over to the actual expert uh Damian who uh who who actually will be where is DAV ah okay yeah who who actually be showing um how it works under the hood um so if you if some of you have tried this did anyone try calling this I put this up briefly uh how's the experience what what did anyone have an interesting experience it told me a Prett fun joke I told you a fun joke okay nice nice I mean it basically gives you back whatever you want from it give us uh so you said it a lot quicker than you thought it would be yeah wondering if I number I is it the number you can call from Joe sure I don't know if you have like one phone call you want to call in AI but whatever FL your boat I don't know um so we're gonna make this live yeah uh it's kind of it's kind of fun um so I was just messing around with with vapy uh it's it's one of these like YC YC startups there's like five of these so uh don't don't particularly create this as like an endorsement but they are very very easy to work with so I I was pretty I will definitely endorse that um sorry yeah yeah VPI I think it's like voice API um and so we're just going to create a personal AI um I think I think I can just kind of create like a a blank template um and we can just call this like I don't know Laten space pod I don't know uh latent space it doesn't actually matter um and we can do like a like a like a system promp right like you answer in pirate speak something uh and then we can publish um and then we can start calling it um I don't know if the The Voice actually works oh by the way that we have no volume control on this thing so it's going to come over the speakers really loudly and we cannot control that but it'll be short um for some reason it's not connecting oh it wants to use my microphone hello all right let's try to calling it again is this is this working is this on hello hey all is working M your voice be echoing through loud and clear what can this old SE dog do for you today oh my God uh how can you tell us how to turn down the volume in AWS LS or if You' be wanting to adjust the volume in Olex you'll be in a bit of a pickle for Olex itself doesn't have a volume control okay yeah anyway that was so um so you can uh you can you know I I thought a super nice experience like you set up a voice thing uh you can connect a phone number to it if you buy a phone number uh that's the that's the experience that uh that you saw if you you call this number um and you can customize it however you like you know whatever system message you have um on my live stream that I did last week um I added memory to this thing so if you call it back you know ideally you you should just remember the previous conversation you have and then you have a personal AI it doesn't take that long I just did it onehanded um yeah it's pretty great uh it's actually built today I discovered it's built on dgr which is uh Damian's next talk so I I'll hand it over to to Damien while he comes over and sets up yeah welcome to Damian oh um and then eigor if you want to come up and share your side goe okay so I want to go into share my personal setup am I the only person in the room recording right now like I see this perfect cool yeah this is a recording okay Meetup right yes uh something I want to do for my Conference in June is it's like everything should be default recorded and then you out yeah absolutely absolutely so what I do is I record all my life constantly 27 I have two setups two recorders and I believe in the Netherlands which allows me to record people without the explicit content because it's legal in the nland if you have any private conversation you are just allowed to record if you are not allowed to share so uh that's cool and uh yeah what what I like um my personal setup is pretty simple this this is the recorder uh it's running all the time I have quite a number of use cases for it already and I also but the main goal for me is to create a huge data set on my life one year of my life is less than terab of data or like one year of audio is less than terab data and we absolutely can afford just saving all this stuff and then mine this data for uh important insights for example I have really interesting and lovely conversations with my friends and I use it I just D them on my AUD player and just can listen to the best conversation with my friends or on T it's incredible but I also I also talk a lot to myself when I'm alone and I'm just explaining this future self uh the context of my life and how is my life started and what is going on I I record all my therapy sessions while I record everything and I can I'm pretty sure it's something that help that will help me to align AI with myself I will have this hug said I call this Ultra personal alignment because there was this road alignment problem but I also want AI to know me really really well to understand the me in the proper context and how to help me to succeed and I think I think this data great is really valuable and all of you who are not recording right now I think I think you should even even if you're not going to use it right now you you'll just have this data like there is no good reason not to have this data it's a it's a recording or meab I think you should like if you have your Apple watch just like open audio noise and start recording start recording conversations you have like it's super easy to do you just take your phone out of the pocket and just record stuff thank you um that's to me that's actually what a Meetup should be like that people bring their stuff and like talk about what their passions what they're working on it's not a series of prepared talk after talk uh but you actually reminded me I actually hacked on this um iOS shortcut where I can just always press my button and it starts recording um anything I anything I have in person and I've actually done this in meetings and when when I when I'm done recording I can click it it transcribes it saves the file and then offers me to do a summary right after so it's highly highly recommended when automation is simple that doesn't require walking around with this stuff I want to share my shortcut as well so my shortcut is I wrote some code to do it but when I press my action button on Apple watch it also starts recording and uh it listen to what I say and say this locally uh and then I I and when this internet connection it sends it to the cloud and transcribes it and I see it in my notion as like list of voice recording and then transcription next to it and it it works really well I can do it on on the plane it works wow okay um so I'm definitely going to if you if you want to share that shortcut with the rest of us i' love to steal it it's really messy but I do want to do this yeah I want someone to improve this code because I'm working on my St after code for you hey all right awesome feel free to talk to me after after this yeah this is what I meet about awesome okay so uh something a little bit more um polished I've already seen you do this song but this is great so uh you want hold I think the audio should come out as well when we get to the demo but uh yeah so hey everybody Damien Murphy I work as an applied engineer at Deep gramp um so yeah what what an applied engineer is is it's basically a customer focus engineer right so we work directly with startups like yourselves and we help you build um you know voice enabled uh apps um what I'm going to show you today is really around you know how to build essentially what you saw with vapy um but using you know open source software right so being able to make something like vapy yourself um some of the main considerations when you're building a real-time voice spot is performance accuracy and cost and being able to scale um you know you want everything um with a real-time voice spot like when you called it right you had subsec response time uh so being able to get that subsec response time is super important right so essentially if you go beyond say 1.5 2 seconds aot lot of people will actually say something again right they think that the person is no longer there on the other end um and you need to do that for speech to text the language model and the text of speech um and then on the accuracy right so you want to be able to understand what the person says you know regardless of their accent um and you want to be able to do that in multiple languages as well um and then on the TTS side um really being being able to be human life right that that's the big uh challenge um and then cost and scale so you know you can build a lot of this stuff with you know off the-shelf open source software and you know probably the text speech stuff won't be fast enough or the transcription won't be fast enough um but you can actually do a lot of this with with manage Solutions as well I'll go into the into the unit economics towards the end um so yeah this is the basic setup right so you have a browser and that's going to send audio obviously you need to uh interact with the browser to capture audio that's one of the requirements for security reasons and so you'll see a lot of these demos you have to click a button to actually uh initiate the speech uh The Voice spot server uh it's actually repeated here multiple times just to kind of simplify things um but you can imagine this is a back and forth right you know you're doing browser to the voicebot server to get the audio and then the voice Pro server sending that off um and the goal here is to get subc latency so um we have you know around 200 MC latency we can get that down a lot lower if you host yourself uh so that's actually what VY does they host their own uh gpus running our software um and you can crank up you know the unit economics and you can say hey you know what instead of doing it at this rate I want to do it 5x so you can get that 200 milliseconds down to about 50 milliseconds uh at the cost of extra GP compute uh and then GPD 3.5 turbo or four you probably get you know 400 maybe 600 milliseconds of latency under hosted API um if you go into a Zur and you use their services you can get that down a lot lower and um and then on the text of speech side I'm going to show you using uh deep grounds text of speech um there's a lot of other text of speech providers out there um what we try to do is have low latency humanik at a really good price point and you can get extremely humanlike at about 40x price um but that's just something to to keep in mind all right demo time my two tabs open here okay so hopefully this audio will play good evening how can deep or meet the specific needs of your projects modify the sound outp you're using my yeah yeah the sound output right now okay sorry it's GNA be loud we we can't control it I think I may have blow blown the speakers did you blow the speakers try try to tell that something speak yeah I can use just the laptop yeah okay let me do that yeah so so turn on the yeah all right okay let's see if it works good evening planning to integrate deep grams or a text to speech into your workflow let's discuss how to get started yeah I want to talk about something else today I'm actually at an AWS hacker space for a Meetup with personal AI sounds like a cool Meetup what's on your mind today feel free to share your thoughts or questions yeah uh could you give me some something funny to show in My Demo absolutely how about a joke or a light-hearted anecdote to break the ice during your demo laughter always helps yeah that' be great how about this one why did the computer go to the doctor it a virus smile feel free to share that how about this one why did the computer go to the doctor because it had a virus smile feel free to share that with your audience for a good chuckle take your time yeah so you can see there right it's pretty low latency oh sorry it's pretty low latency um and thanks and uh if you want to get the source code uh it's all open source right so you can just grab that whole repo you can build your own voice bot um and if you want to hook it up to like a telepon system and we we have that all available in Sample code as well so you can hook it up to a twio buy a number uh do those sorts of things uh so let's just jump back into the presentation oh yeah so so the the different uh components right so the speech to text um that that's going to be you know super low latency right if if you don't get that accurate you're going to get the wrong answer from the lln um and this is the code that you can use so we we have S python go ruby. net um and you can essentially use all of those this is actually a node.js SDK and it's very simple to set up right you literally just import it drop in your API key and uh you can listen to those events so we'll we'll give you back uh all the text that was actually spoken while it's being spoken and you just need to send us uh the raw audio packets uh and then on the GPD side so you can swap this out with Cloud we actually have a fork of that repo that uses Cloud as well uh Cloud hu is surprisingly good so you know if cost is is something we want to get down that's definitely an option um but some some some of our customers will actually run their own uh like Lama 2 model uh super close to the the gpus that are running the speech to text and text of speech H and that just removes all the network latency out of the equation um yeah so here's a simple example of how you would consume that I'm sure you're all pretty familiar with the open AI API um and that will basically give you streaming and that's one of the really important things here is you you want time to First token as low as possible uh the reason for that is if you wait till the last token you know you're going to increase that latency H and then on the last bit this is the text to speech part um it's a little more tricky right you've got to deal with audio streams um and you're going to want to stream the audio as soon as you get it so that you can actually start playing at the moment you know the first bite is ready H and that like if again with the llm if you wait till the last token if you wait till the last bite of your audio stream you know you're going to incur that uh bandwidth delay um so yeah if anybody want wants the open source repo uh go ahead and scan that I have a question yeah that's really cool demo the latency is great like what do you think the next Frontiers are like I mean you had interruptions right what about like can it proactively interrupt you um like based on maybe it knows what you're trying to say like a human would just cut you off or like um back channeling or overlapping speech and all the more getting more towards human level kind of dial yeah yeah so the question was could the AI interrupt the person um and that's definitely possible uh I don't think that would happen necessarily at the the AI model level I think that would just be business logic um so you'll you'll get you know everything that's spoken as it's spoken um so like if you were like you know what I think I know what you're going to say um you could preemptively do that I have seen some demos where that's a trick that they use to actually lower the latency is to predict what you're about to say right so then can fire off an early call to D llm um this that Dem openo everything interrupting very good yeah and you know the cost of a lot of these llm things uh is a big challenge as well so like if you're doing you know constantly sending it to an llm to achieve these use cases you know your cost per minute might go to like 30 30 Cent and so in this demo here and and these are all like kind of list prices uh you can get these prices down with volume and so like if you just signed up today and you know these are the sorts of uh prices that you would pay and and just to give you an idea right you know qbd 3.5 turbo has dropped in price dramatically over time um so you know Claud high C is even a fraction of this as well um and then on the Texas speech side and doing something like this with an 11 Labs would be about maybe a120 um just to give you an idea of comparison so you can do a 5 minute call here for about 6 and a half cent um if you're doing millions of hours of calls you know that that price can definitely come down um yeah so changing that then to be a a like a real-time callable voice spot like you saw with the vapy demo um you're essentially just swapping at the browser for this telephony uh service right so twio has about 100 millisecond latency to give you the audio when you get called and and then you're just sending it through that same system and then just back to the back to the telephony provider and yeah so if you sign up today you get $200 in free credits um for post call uh transcription that's about 750 hours uh for real time that's probably about uh 500 hours of real time transcription so it's a pretty pretty big um uh freebie there so if anybody wants it and and that's it any any questions yeah go ahead just in terms of achieving Real Time Performance G 3.5 versus four like how do you yeah four is going to be a lot slower uh especially if you're using their hosted API endpoint um you're going to see massive like second fluctuations in their hosted endpoint uh if you go on to like a Zur and you use their service um you know you're paying more but you're getting you know much better latency so you know you could deploy all of this on a Zur and next to you know gbd4 and and that's going to give you uh you know the sort of latency that you saw in the VY demo and the the demo I get actually is using all hosted apis so there's no like onr kind of setup there Ethan Nick and then I think Harrison just walked in so just warm you guys up thanks to D only sign up to speak today which is very classic e strategy um yeah I think so we don't have a screen for this we have you can share your screen I can share my all right so the uh I'm afraid the audio might be like shattering your eardrums so I might have to cut it off but we can try you said it was okay yeah I mean okay do the from the laptop no no no there's no laptop yeah yeah sorry so maybe unar what did you work on why yeah sure so I think actually like it feels good to be here here because I feel like I'm with my people like uh you pretty much summed up the philosophy like what what what can AI do for you if it has your whole life this context and you kind of experience it experiences your life as you do like wouldn't it be way better understands you has all your cont um so that's kind of you know I think a lot of us here having the same same idea and so that's what have been working on for since about November um when lava came out so it first started working on like the visual component it's not just audio you wanted to see what you see um but it's it's a lot more challenging to get a form factor with Contin of video capture so um built a really really small but simple device that's actually easy to use it up it up um it I don't know you hold it up for them I mean you guys can check it out after the talk um but you know I think you know there's a lot of advantages to not having to have an extra piece of Hardware to car around but at least we try and make it as small and as light as possible you only have to charge it every couple days um but you know there's also other subtle reasons why it's good to have an external piece of Hardware which I can show you in a minute but um so yeah this captures all the time and in fact you know I can show you here in in the app so you can see this we call the the you know it's got the battery level here um but here you can see all the conversations I've had um and this is actually an ongoing conversation um so you can see you know you know where I am um and you know transcript in real time um we do a bunch of different pipelines so like after the conversation is over you know we'll run it through a larger model um and then do the speaker verification so it knows it's me which is important so we can understand what I'm saying versus other people in the room um actually conversation endp pointing and the Deep gr person probably knows like even just utterance level endp pointing is complicated at hard like when am I stopping talking or am I going to keep talking and I just paused for a second that's hard but then conversation imp pointing like when is this a distinct conversation versus another is even harder but that's important because you don't want just one block of like dialogue for your whole day it's much more useful if you can segment it into distinct kind of semantic conversations so that involves not only voice activ detection but um things like signals around your location and even the topics that you were talking about like at the llm level um so it's very difficult um you know there's still a lot of work to do but but you know it does it does work so like in the end result I will get a summary generated um some takeaways you know kind of summary the atmosphere then like uh oops didn't mean to click on the link um then you know from the major topics you know I'll I'll find some links and then you still up the ra um so that's like kind of the foundation layer is to like have something that does the continuous capture does the basic level of processing but you know that's just kind of the the base layer um you know I can I can query against like the whole context of everything it knows about me I'm trying to T with one and so um help so I think this is I was talking to my developer a few days ago so here we have you know it's it's through retrieval on all my conversations this is a conversation I was last talking about and it will even site the source so I can just jump to the actual conversation which is a few days ago or a week ago or something and so here we were debugging some web view issues um so that's just kind of like the basic memory recall use case um I just have maybe one or two more then I'll I'll turn it back over um so again that's all kind of the base layer but like the real you know I think everybody here believes like the real future will be using all of this context so that AI can be more proactive can be U more autonomous because it it doesn't need to come to ask you everything like if you if you had the meager co-workers every day and they like it's a blank slate you know it's way less productive than if they have the whole history um so there's a voice component to it and this is I'm a little nervous because it's blow ear that but we can try um sure so we're using hot words here which I I don't believe is the best Paradigm but but for now we have some other ideas but but for now similar to Alexa or Siri I can um basically inform my AI that I'm giving it a command that it should respond to in voice um so I will just um Scarlet can you hear me yes I can receive and understand your messages how can I assist you today okay that's Fring um I will I will just now do um one one example so like you know you have the ability to interact with the internet your AI should so um I can have it go do actions forming using any app so um I'll just do a very simple example uh Scarlet send a message to Maria on WhatsApp saying hello one sec I'm on it starting a new agent to send WhatsApp message to Maria so now this is on my personal WhatsApp account Maria's right here um she can verify that she received it yeah I received that message hello sent to Maria on WhatsApp so maybe maybe one more and and then I'll I'll let you guys go so like that's just opening one app and doing something but can it do multiple apps and have a working memory to remember between app context so um Scarlet find a good taco restaurant in Pacific Heights and send it to Maria one sec I'm on it starting a new agent to send taco restaurant details to Maria so it's open Google Maps it's going to try and find a Taco restaurant hopefully remember once it does um and then send it to Maria which it it learned that I I implicitly meant WhatsApp right hopefully because it picked up that I talked to Maria on WhatsApp um so going to Whatsapp pasting the link the details of taco bar a well-rated taco restaurant in Pacific height San Francisco have been successfully sent to Maria on WhatsApp yeah so let basically it um yeah that was a great Dem there's got to be some questions so you're not AG GRE with okay there's it's open source uh components are open source so there's a history you know Adam's here great Nick some really great people in the open source space here we um we open sourced a lot I I really learned a lot about about open you know I've done some minor open source projects myself you know mostly I'm just a contributor but trying to run and launch one was kind of a new experience for me and I learned that like like we did not make the developer experience very good it was very complicated um like because we were using like local whisper local models and like getting it to work on Cuda Mac windows we didn't do a good job so it was very difficult for people to get started and so it was a little disappointed with the uptake you know and they're much better projects that that that are way easier to you um so really been focused now on just trying to focus on figuring out what the right use cases are and what the right experiences are going to be and and it was like really difficult to try and and fit everything into an open source project that would be actually used so um yeah you can you can see the the repo and I'd recommend Adams and next two um but uh we'll definitely be contributing a lot back more to open source um is it owl or B owl is the op the repo yeah yeah yeah okay um and and you should I'll plug Adams adus and next repo we can we can send out to I don't know what the actual gith hubs are called but they're easy to find yeah he's gonna talk so think show um yeah you talk about the hardware sure tell us more about it um this is like V1 um so we have another like v1.1 uh which is actually even um about 25% smaller um and um way better charging situation in terms of wireless charging so like get the size down but the real thing we're most excited about is like the next version with vision like I was say vision is really really hard I don't know of any device that can do all day capture of like sending video there's like ton of challenges around power and also bandwidth um but we have some really kind of novel ideas about it um the um it's it's Bluetooth low energy which I mean I'm sure you've seen other ones that operate that way and that has a lot of advantages we also have one that's LTE and so there's like LTE m is like a low power subset of Lte it's only like two megabits but it's way more power efficient I think Humane and rabbit are both um LTE and Wi-Fi but um to get like a wearable you really need Bluetooth just quick question saw the part of the interface where it had a was that streaming from iPhone it's an emulated Cloud that picture picture how are you doing the where you the it's that's on the cloud so we were streaming that is feedback I think the ultimate goal is that that disappears entirely right that's actually mainly for the demo to like show that it's real and that it's just working but like I think ideally like it should be totally transparent that you're just you have like a personal AI it can do whatever it needs to do and it will just give you updates on if it if it you know needs more info or or status but um like it's kind of just the interim solution until we have I guess so sorry no no no no yes it is uh okay last question and then we have to move on okay I also want I've been experimenting with this app and you have like a pocket here yeah I just put it what what I wanted to say about about vision is that like I try to experiment with capturing vision and like the best solution so far I found is like to buy cheaper Android smartphone your here like outs it's incredible it has it has internet connection good battery incredible extremely cheap bable like you have SP I I had I've had I've had whole um whole uh whole demos of doing that because also you also it doesn't put people off as much yes yes and no one no one even thinks you you just you just have a but I I I do I like I do think in privacy sense of maybe slightly different than you I I do want people to understand but like it was interesting that uh yeah her style um basically her style but nobody nobody nobody thinks twice thing you can just write on it is like I'm reporting it I'm just saying the convenience of not working on the hardware you just take the off the sh but you do need a front you need a front pocket so maybe there'll be new AI fashion where it's like these are my my phone pockets okay give it up here that was awesome um uh yeah I mean everyone will I think will be sticking around so you can obviously go up to him and uh get more insight awesome um all right hey everyone um one sec me oh yeah it's not connected now it is nice so where do I even start um yeah um unfortunately I was kind of not supposed to be here because I'm organizing a brain computer interface Katon tomorrow and uh I had to like somehow get a 50 headsets um which is why there will be no presentation but I will still try to be useful to you as possible this is the heon I was telling about uh right now we have like lots of people there will be people from neural link um we'll have like 50 different BCI headsets and so on so if you're interested in like BCI Stuff Etc uh and if you want to attend the hackaton scan this QR code and mention that you have been here and oh yeah my bad sorry and uh you might get much more chances to be accepted uh because we have like 5050 50% rate we try not to accept people who don't have experience anyways um so yeah what I will try to help with I honestly really really love open source and I believe like all this stuff should be open source which is why now on this short demo I will just show you all current open source projects um and I will try to highlight most important things you need to know about them and uh I'll probably first start with owl which uh you have just seen uh by Ethan so he started that um he I think he was like one of the first people who started like open sourcing any kind of variables probably Adam actually was before but you announced first so I remember but yeah yeah yeah anyways so uh yeah this is his repo uh you can like check it out I think I have a QR code here opened as well if not just give me a sec I will just generate it quickly for you uh all right should be yeah just scan this QR code and you can just access his repository uh yeah so this is eans then there is another one um which in my opinion well is is definitely the biggest one uh it's by the guy who's sitting right there Adam I truly believe that Adam is like the guy who started all Open Source Hardware movement so at least I uh started doing everything because of Adam so thank you for that uh and they have a lot of traction here 2.6 th000 stars and if you want to kind of like ramp up your way into open source um um like variables of any kind I suggested to start with this repositor probably the biggest one you'll be able to find right now and the QR code you can scan I think this one yeah just like feel free to scan I'll send s yeah yeah cool Co cool cool awesome so they use uh I think Raspberry Pi also use p32 um which is kind of technical you probably don't need this information but anyways um yeah and now who I am like a little bit about myself as well uh some marketing so my story starts uh very recently maybe two months ago after I saw Ethan's and Adam's launch of like their Open Source Hardware stuff and uh I launched my own um it all started with um basically season disas from Humane um and uh we launched like just for fun honestly P the idea here was like you take a picture and you scan the person's face and uh we search the entire internet and we find the person's profile um like and sent it to you VI like I don't know like by notification or on the site and so on so this is how it looks uh started with that had a lot of contributors was a fun project but not really I mean like yeah we don't really want to bring any harm to human and so on and so forth just to findun just fun stuff so after that uh we did oh you are for this I think we also same right like cool cool cool cool cool okay um another one I will promote here a little bit is this one this we launched literally like last week this is pretty much what Adam and um uh and Ethan have have done with the only difference that we use right now the lowest power chip I think Ethan also uses that I just try to like um you know like as soon as possible to let everyone know that like I think it's probably the best opportunity you can currently find on the market uh it's called friend you can check it out this is how it looks or actually have I have it with me um and uh uh I was supposed to actually show you like the live demo as well but uh we just did a very cool update uh where we made it work with Android by the way so it's IOS and Android right now and also we updated the quality of speech literally three hours ago and made it like five times better so when we launched I'll be honest it was like completely horrible and now it's like five times better it's amazing um and I'm really excited about that I would want to show you the video but I know that it's like you will not want to have your ears you know like oh yeah really oh nice okay let's try anyways yeah this is the chip that is being used inside of that variable um and it works pretty much the same as what Ethan has wait the permission oh it doesn't it doesn't work right next scan for the devices wait wait let me figure out uh this which one CH yeah yeah you can go back okay let's try oh nice okay cool giving the permissions here and H next scan for the devices here and then select the device and I should just connect and yeah I think I think we can just start speaking um yeah so pretty much testing this for the first time and uh let's see if it's able to transcribe what I'm trying and speak and uh yeah pretty much waiting for the first 20 30 semester finish and for it to return return us uh the output of uh the speech from the open AI uh whisper and point and it says if it's checking yeah there we go and yeah it says yeah so as you have seen it recorded the speech on the lowest power chip probably ever right now accessible in the market and I'm very excited that we actually made it work because um it's it was very very hard it took like a lot of a lot of time anyways um that's pretty much it I guess I don't really have anything else for you to show uh this is the final QR code I know you will have all the links but this one I really really advise you to scan because it has basically the collection of like all links in one place um so yeah that's pretty much it use open source uh I think it's cool and let's try to build cool stuff together that's that's pretty much it thank you any questions did your record that uh no because uh yeah you're you're not recording right now uh no no no I'm not recording because we launched the update like 4 hours ago and I wanted to bring it here so I broke my thing and now it's like how broken unfortunately but anyways any questions cool yeah what do you think the biggest next challenge biggest next challenges yeah I definitely agree with you that the biggest challenge is like eding video and images and so on it's like very hard as hell and I think to make software useful as well is very very hard um like yeah we can all do maybe like recording from the device and like attach maybe some battery and so on but like how to make it actually like sexy that's that's hard like how to make it you know do actions how to make it remember everything and so on and so forth that's the biggest challenge so yeah but I agree with everything you saidar it yeah Adam the guy who created the biggest open source uh thing he said that uh the biggest challenge is to make people want it basically right I mean say Yes um so that's that's that's one of his suggestions um yeah go what's been the challenge in reducing lat what's been the challenge of reducing latency um honestly it was just like software issue um because like this chip is like not that widely used there's not so much documentation not so many projects and so on so just like a matter of like trying a lot of things and also it doesn't have huge onboard memory so like how do you store very quality memory on a very small very quality audio a small memory chip and then send it to phones like streaming basically nonstop that was pretty hard but we sold it like 5 hours ago and uh that's pretty much it so how did you improve the quality of voice um we just made it work like we used uh we had 4,000 uh it's pretty technical don't think I don't know if you want this but anyways we used 4,000 HZ like quality it was like super bad because the memory was too small and now we just found a way to like compress it and improve it to like 16,000 which is like pretty great which you have CED on the um on the video so it can like recognize pretty much anything even like multiple people speaking even if you will be there and the device will be here so yeah anything else we can leave the last one go how far can detect how far the device can detect a right cool so good one will be like you know 2 ft from from person to person like good uh if you have maybe like 4T from each other the quality will be 50% ACC your so yeah cool thank you all you open cool cool so what I want to talk about is a lot less cool than all this Hardware stuff I feel a little bit out of place um but my name is Harrison uh a co-founder of Lang chain uh we build kind of like developer tools to make it easy to build LM applications one of the big parts of llm applications that we're really excited about is the concept of kind of like memory um and personalization I think it's really important for personal AI because you know hopefully these uh assistants that we're building remember things about us and remember what we're doing and things like that um we do absolutely nothing with Hardware so uh when we're exploring this we are not building kind of like Hardware devices so we took a more kind of like software approach um I think one of the use cases where this type of like personalization and memory is really important is in kind of like a journaling appp um I think uh for obvious reasons when you Journal you expect it to be kind of like a personal experience where you're kind of like sharing your goals and learnings and I think in a scenario like that it's really really important for uh if there is an interactive experience for the llm that you're interacting with to really remember a lot of of what you're talking about so um this is something we launched I think last week um and it's basically a journaling app with some interactive experience and we're using it as a way to kind of like test out some of the memory functionality that we're working on so I I I want to give a quick walkthrough of this and then maybe just share some high level thoughts on on memory um for llm systems in general um so the ux for this that we decided on was you would open up kind of like a new Journal um and then you basically write a journal entry um and I think uh this is kind of like a little cheat mode as well because I think um this will encourage people to to say more interesting things so I think if you're just taking like a regular chat bot there's a lot of like hey hi what's up things like that and I don't think that's actually interesting to try to like remember things about um I think it's more interesting if you talk about uh personal things and so let me let me try this out um I'm giving the talk right now and then I can submit this and then the ux that we have is that a little chat with the companion will open up um okay so yeah so right before this I told you that I was about to give a chat about a journaling app and so it uh kind of like remembered that I was going to um uh do all that um is there a particular part I'm most excited to share the memory bit I don't know so this is this is this actually worked on the first try so I was a bit surprised by that so um that's good um and oh okay so how do you plan to tie in your love for sports with the theme of memory during your talk so uh before when I was talking to it I mentioned that one of the things that I wanted to talk about uh was you know how a journal app should remember that I like sports so I guess it remembered that fact as well um amazing so I can end the session so so the basic idea there and again this is you know we're we're not building this as a real application we would love to enable other people who are building applications like this um I think the thing that we're interested in is really like what is what what does memory look like for applications like this um and I think you can see a little bit of that if you click on this memory tab here um we we have like a user profile um where we basically kind of like show what we learned about uh a person over time and then we also have a more like semantic thing as well so I should could search in things like Europe um and I'm going to Europe uh kind of like after a wedding I love Italy um and so basically there's a few different forms of memory um and uh if if you'll allow me two minutes of kind of just theorizing about memory um we're doing a hackathon tomorrow and maybe some of you are going to that um swix signed up I don't know if he's actually going to show up um so uh very quickly like how how I think memory is really really interesting um it's also kind of like really vague at a high level I think like there's some state that you're tracking and then but how do you update that state and how do you use that state these are like really kind of like uh vague things and there's a bunch of different forms that it could take um some examples of uh kind of like yeah thanks uh some examples of of memory like that that a bunch of uh real apps do right now like conversational memory is a very simple but obvious form of memory like if you it REM is the previous message that you sent like that is like incredibly basic but I would argue that it can can fall into this idea of like how is it updated what's the state how's it combined semantic memory is another kind of similar one um where it's it's a pretty simple idea you take all the memory bits you throw them into a vector store and then you fetch the most relevant ones and then uh I think one of the early types of memory that we had in linkchain was this knowledge graph memory where you kind of like construct a Knowledge Graph over time which is maybe like overly complex for some kind of like use cases but really interesting to think about um so Lang me like name TBD um is some of the memory things that we're working on and we kind of wanted to constrain how we're thinking about memory uh to make it more tractable um so we're focusing on like chat experiences um in chat data um we primarily focused on like one human to one AI conversations um and and we thought that flexibility and defining like memory schemas and instructions was really important like one of the things we noticed when talking to a bunch of people was like the exact memory that their bot cared about was different based on their application if they were building like a SQL bot that type of memory would be very different from the journaling app for example um so there's a few different like memory types that we're thinking about all of these are very like early on um uh I think one interesting one is like a thread level memory an example of this would just be like a summary of a conversation you could then use this uh you could extract like follow-up items and then in the journaling app you could kind of like follow up with that in the next conversation we actually might have added that that might be why it's so good at remembering what I talked about in the previous talk I forget um another one is this concept of a user profile it's basically some Json schema that you can kind of like update over time um this is a this is one of the newer ones we've added which is basically like you might want to extract like a list of things um similarly like to find a schema extract it but it's it's kind of like a pend only um so an example could be like restaurants that I've mentioned like maybe you want to extract the name of the restaurant on what city it's in um if you're kind of like overwriting if you put that as part of the user profile and you overwrite that every time that's a bit tedious so this is like a pend only um and then we do some stuff with like knowledge triplets as well and that's kind of like the semantic bit um I think I think probably the most interesting thing for both of these is maybe like how how it's like fetched um so I don't know if people are familiar with the generative agents paper um that came out of Stanford last summerish but I I think one of the interesting things they had was this idea of fetching memories not only based on semantic but also based on recency and also based on like importance and they'd use an llm to like assign an importance score um and I thought that would really really novel and and enjoyed that a lot um and yeah that's basically it so yeah questions well yeah yeah I think you know about all the same things I think a lot of your approach makes a lot of sense same kind of compromises you have to make complicity but like to give true knowledge you talk about like the triplets like um like how do you think that we can get to the point where we can have more of a dense graph rather than just simple proposition about you because like our memory works it's all relational to like you know to people to places and that that's actually important information and so like do you think we'll be able to figure out a simple way to do that or is it just going to be too hard yeah the honest answer is I don't know um I think even today like if you had that and I think like what there's there's two things like one's like constructing that graph but then the other one's like using that in in generation and like even today like for most like rag situations com if you combine a Knowledge Graph it's often not taking advantage of a lot it's it's really like it's it's there's different ways put a Knowledge Graph into Rag and it's not uh yeah it it's very exploratory there I'd say so I'd say like one is she is just even creating that and then the other one is like um using that in rag so I think that's like a huge yeah I I don't know yeah we'll see it's interesting um yeah cool Stu on the memory things um I just had like a couple points I think um no matter which memory option you're using like um let's say you're using like a pen only memory model versus the knowledge graph one like eventually that has to be inserted in context right like you're picking the relevant sections and then there might be like a semantic layer to figure out okay which ones which facts should I embed into the context during generation or during the during the query but like I was just curious like is there work on defining semantic models that don't leverage like the reasoning of the model that um sort of like let's say you want to build up like not just memory but like an understanding of the semantics of um of the operations for example outside of depending on the language model to provide um that semantic interpretation on top of whatever memory context you inject um if I'm understanding correctly or you asking like is there a concept of building up a memory about how the language mod do like the generation aside from just inserting it into the prompt yeah like like you know the memory is just externalizing it's basically get cash for uh to like get rid of the token limit and also bring things into more attention by by transforming The Prompt a little bit and injecting instructions about how to treat bits of memory um yeah I was just wondering like have you has has there has there been work done to basically decouple the language processing versus like the actual like operational semantics of using the M yeah I'm um I'm assuming there's yeah I think like an alternative approach to this which I think is what you're getting at is like rather than doing kind of rather than kind of like creating these strings and putting them into the prompt you could have some kind of like attention mechanism which would like attend to different parts of previous conversations or something like that I'm sure there's I I think like I mean you could take this to the extreme and basically say like for there's there's a lot of stuff that could fit into like even like a million token context window or a 10 million token context window um and so an alternative strategy could just be like hey put everything all these conversations and all these journals I've ever had into one llm chat and kind of like Let it go from there um and yeah I I I'm sure people are working on things like that and doing things like some sort of like cing to make it easy to kind of like continue that I don't don't I don't know of any details um but yeah I think that's a completely valid completely alternative kind of like approach that's also really interesting I don't think anyone knows how to deal with memory and so I think all these different approaches are I think in a general way yeah yeah hard going on questions yeah so when I see systems I guess that's a question pretty much for everyone who presented right like when I think about these systems I always think what does 10 years of memory look like and and a lot of the facts that we remember are are not relevant anymore probably F so how do you think about like memory yeah I I think there absolutely needs to be some sort of memory Decay or some sort of like invalidating previous memories I think it come in a few forms so like with the generative agents kind of like paper um I think they tackled this by having some sort of recency uh waiting and then also some sort of importance waiting so like it doesn't matter like you know how long ago it was there's some memories that I should always remember right um but then otherwise like you know I I I remember what I had for breakfast this morning I don't remember what I had for breakfast like 10 like and maybe I should but I don't think that's like important um so yeah I think like recency waiting and importance waiting are two really interesting things in the generative AI or the generative agent paper um another really interesting paper with a very different approach is M GPT um so M GPT uses the language model to like actively kind of like construct memory so like in the flow of a conversation like the the agent basically will decide whether it should write to like short-term memory or long-term memory um I I think that's a yeah that's a I think it's actually quite a different approach because I think in one you're having the application like actively write to and read from memory and then in the other one the one that we're building is more in the background and I think there's pros and cons to both um but I think with that approach you could potentially have some like overwriting of memory or or yeah last question last question also that generative agent Smallville paper amazing there's like one page they have like an exponential time Decay every day stuff is less relevant definitely recommend hey har thank you for presentation I just want to ask about there was a new paper called Rapture I think and it feels like it's a really cool approach to memory because sometimes when you want to say something like who am I oh roast me it's really hard to do with Rag and these type of approaches but the Raptor could be a nice way tole that your opinion can you summarize uh I think it's about like uh doing like partial summarization and in like a free form and we are trying to experiment with that and it seems like but I think you know more about it I just like found out about couple days ago so yeah no so I think the idea is basically for rag you you chunk everything into really small chunks um and then cluster them and then basically hierarchically summarize them and then uh when navigating it you go down to different nodes I hadn't actually thought of applying it to memory but that actually makes a ton of sense so one of the issues that we haven't really tackled is in this journaling app if you notice in here there's um a bunch of that are really similar right um and so like there's a clear kind of like consolidation or update or something procedure that that kind of needs to happen that we haven't built in yet and so I actually love the idea of doing this kind of like hierarchical summarization and um maybe like collapsing a bunch of these into one and maybe that runs as some sort of background process that you run every um I don't know you yeah you run every day week whatever it collapses them accounts for recency to account for the issue that was brought up earlier around wanting to like maybe overwrite things yeah I think that's I had not B it at all but I think that's really kind of like sleeping sorry like when the brain consolidates cool I think I got it wrap it thank you so much haris Round of Applause for I think there's the real reason why this is not just the hardware only meet up it started as meet up but then we added voice and then we added memory uh because this all components of incling your personal AI uh so we have 15 minutes until we have to clear out of here uh everyone is look like they sticking around to just chat if you want to just see the devices and talk to Harrison go ahead you I guess um there are like three hackathons spawning from this thing so uh I'll send all the links Ona thank you thank you very much thank you