Delete - Reimagining Product Development with AI - AI PM Community Session #30

Key Takeaways

so um I think give me a little bit time so I go through the the discussion so and uh at the end I can answer questions and obviously these are kind of my personal thoughts uh based on the last I said 12 months or almost like maybe 15 months that this new trends of llms is out there so it's just um be aware of that um I choose a topic as a embarrassing the unpredictable and I think that's the most important things that we have to to learn here um as I said the goal is uh basically highlight some of the challenges that we have with uh developing llm applications it's kind of a little bit different and kind of little bit maybe a lot different than uh classic uh product development and also thinking about ways to actually um embrace it kind of this Dynamic and undetermined of AI um a lot of things that I'm going to talk about is just basically pointing that hey watch out for this watch out for that I won't have time enough enough time to get kind of get into the details of how to solve these problems but my point is just think about it when you're building this this product and um so yeah and and also kind of hopefully product managers and Engineers can work a lot closer L to kind of address this problems because I feel like the new nature of this these applications create a whole bunch of different responsibilities that in a lot of different organizations from a small startups to a very large organizations kind of who is doing what is something that everyone is trying to figure out but for your project to be successful I think as a stakeholder that you want this to be successful you have to watch out and making sure that we are think about those stuff and and you have answer for it no matter who's doing it right so that's kind of the part of go so thinking about like how we were doing the product M building the product you guys know it a lot better than me I'm just kind of going very hand wavy on on it is that you basically start some kind of conceptualizations planning all those kind of stuff um do Market fit all those things and then you go through the design development as effects and then you know testing and finally deployment right so it's it's how we always have been doing now we always had the ad testing we always had the concept of like generating like doing MVP and ret trting and reiterating all those kind of stuff are are same and we are very familiar with it but one interesting things that I think it's a little bit different is that when you're building classic application let's say you have a user interface there is a form there's a database at the end there is some way that you can Fitch data or analyze data so it's very classic applications uh you do AB testing to see how the users go through your scenario so that part is unknown right it has always been unknown you would you would need to test it but the success of a scenario saying that if the user start from typing something in a form it will get recorded into the database reliably and you know good performance has always been on a engineering side and they were kind of able to with a very high confidence with a very high SLA solve those problems and even if in cases that you know you had a design issues or miscommunications or boggy things you could always spend like a week and and kind of push it and kind of do a bog bash and resolve all the kind of problems right but the burden of getting a scenario deterministically successful has been always on a engineering and we knew that our engineering site can solve this problem easily relatively right uh I think with a lot of this llm and generative AI one thing is that and and you've seen it even in a talks from Sam or people who are really really good at this is that the nature of it is UND deterministic so you might actually Define a scenario you say hey my user is going to start here go through this path and and then end up here which here is a place that he or she is successful um in the middle if you are depending on these llms or different kind of this generative AIS there's always a chance that randomly things goes the way that you do not like it to go right you there's a whole bunch of different random out outcome that can happen and if any Engineers come to you and say I'm 100% sure that I can get it right 100% of time that person does not know what he's talking about or she's talking about right so the it's all about the amount of UND deterministics not if it's going to exist or not I mean we are getting a lot better like in the past years I can say that we are learning a lot and the industry is like building and building and and the applications are BEC more reliable but you always have to think about instead of one n now you have to think about the statistics it's not about one number that you hit it's about the range that you are going to hit and as a product developer you need to think about that and you need to plan for it because you cannot just say Hey My My scenario is going to be exactly 100% of time like that um yeah so basically the message is that think about unexpected and plan for it uh the other topic that I think it's very interesting is um generally AI alignment and um I don't know if you guys know I'm just basically uh pointing out what AI alignment is I mean um you might also hear about like responsible ai ai safety all those kind of stuff but it basically mean that how you can get your AI Behavior to be exactly what you want and this exactly what you want it means like it actually finish your scenarios or um it doesn't you don't have your users or maybe a bad actor getting it to do something that is embarrassing or something is damaging all those kind of stuff or even like you know in a more flal way I mean if if you think if you talk to a whole bunch of people like Ilia that you know if you listen to their talk or or people that are working in this this area there's also another General General war is that if AI is going to kill us or not I mean that's probably a very more high level or maybe futuristic part of it um which might or might not you know I don't think it's it applies anytime soon to the products that we are building um but just generally speaking for things that are um VR is starting to build and using the gbt and stuff like that it's very important to see that if it's actually aligned with what we want and it it changes sucus rate of it so if your product um and and yeah so so that's kind of like part of your success now the interesting thing is that it is not something that you get the right at the first right so it's it's a very iterative process and even what what is right is very interesting like how you would Define this alignment so you have to watch out and and be ready for an evolving process a process that you go through until you get to a acceptable or perfect maybe if it exists a perfect point right um now one interesting thing is that if you want to think about all this risk of misaligned uh AI uh you can think about like damages that it can cause to your customers right so it can corupt your customers data it can um create uh wrong result like we had there was an example that when Chad GPT just came into the market there was a lawyer that just used it to generate some text and use it uh in uh for his case and then he realized that it was just all Hol donation that that the case didn't exist that all of it was made up by the AI so it's interesting to kind of make sure that um we have at least understanding and guide guides or guards against all these confli so you can think about all these different tools that you have to making sure that that things first is making sure that user have a understanding um we guard our product to make sure that it doesn't really get out of hands and also um kind of educate our users like those kind of stuff are very interesting now how you can manage those things it can also be damaging to your company right so so your your brand can be damaged your you might have a data loss you have you can you know Revenue loss or even security right security taxes is a very new things and we are just learning um so that's our interesting risk and the legal risk as well right so we don't know what's going to happen and and there is a lot of different legal jurisdictions and everything is new so it's it's very interesting especially in a areas that are uh you know there's a lot of regulator like healthare finance and those kind of stuff it's kind of like a very interesting now how would you like the the ways that you want to think about how you manage those risk or even assess it is uh in my opinion it's kind of goes to to few higher level areas one is uh how do you set expect first of all set expectation for your users um when we are releasing all these AIS there is a huge hype right so people read about it they they feel or and and there's a lot of people that actually create a false hype as well on Twitter or on other different places just because they want to have attentions or or uh followers but generally it's the job to makeing sure that the expectation of the users is aligned with the capability of the product and it's all most of the time what we are seeing is it's extremely higher than for example the scenario that we are building or the capability of the the product sometimes it's it's it's lower and they get a really nice interesting um surprise but it's very important to make sure that that what we are promising is what we are actually delivering here um because otherwise sometimes moving your product toward that expect might be one of those a little more harder problems that scientists are working on it or so that's that's one thing um as say your risk profile I mean the a lot of these things that we are talking about might not even apply to your company or might apply it's really depend on uh the the company the size of it where you are or even the product that you are building for example if you're building a game and and you're using AI in that game and maybe if the game is for adults you have a completely different set of expectation versus if you're building it for the um for the I don't know finance company or or a legal firm or stuff like so there is a very different risk and um so I think that's that's part of building it and communicating it with the with the engineering and data scientist to make sure that it actually we hitting those um you and it's it's also because because different efforts to get things right for each of these areas it can be it it can have a like a completely different costs right so it's important to know those um of course early consideration everything with these things is that kind of think about it because once you're starting shipping it and then you realize that you cannot really get it to the to those benchmarks or those minimum bar that you want um you might have a product in your hand and you spend like weeks on it and you have to spend another six months to kind of get it into the frame that you can ship it right um of course legal consultations and privacy consideration is also to True especially a lot of these models when they do inference uh they are not as much exposing private information you know the inference is stat less at least in most cases um but if you're trying to kind of start to learn and and and um kind of improve or or kind of uh have a personalized experience then you have a lot of this this kind of Now privacy consideration especially if you're working in across Europe and and other places um so that's that's one so again this some of this might apply to you some of them might absolutely not apply to you so this just kind of really depends on that that where you are right um it's this is one examples of many examples of been seeing on a on a on online and or or different places that uh people shift something with the GPT and then someone comes in or sometimes very easily sometimes very it it involves more but uh they get these these machines to do things or say things that are not um right in this example this guy actually started to talk to this um dealership uh chatbot and kind of convince it to sell a car to it for $1 and it was kind of like going around the Twitter um so this all kind of stuff that you and I've seen it for xedia had actually another issue like this and um it's it's all around the place you want to watch out for these things um yeah so the interesting thing as I said is that if you assume that you are basically at this point it's your perfect point that you are basically 100% hitting your customers ask and requirement right so you don't really say no to any legitimate customers's request but at the same time you don't do anything wrong right so this is your golden point you most likely don't hit here actually not most likely you won't hit here so you always are either on this side or on this side and your job is to see and and by you I mean the whole product te from the data scientist Engineers product managers designers all of the team um need to work toward how they can get closer to this there are few ways that that there is a mechanism to get to there and how you would start to build your product you getting there I'm going to talk about it but remember that again it's a kind of a it work that you have to do so [Music] um the first thing is defining some kind of Matrix so we all know how to define matrixes we have been always doing it but in this case you need to have a specific matrixes not just about usage is not about like a um Revenue that this brings all those kind of of stuff I mean those are very important I'm not saying they're not important I'm saying that you need to have a new set of measurements and these measurements are measuring how successfully you are hitting or getting closer to that that bow point that I was I was talking about and you can be creative this really depend on on your business but you basically want to have two kind of things and measure them in the same way one is that um something that measures how successfully you are uh completing the legitimate um users questions uh or or requests and also having some kind of measurement that that kind of measures how often um you get out of those de rate like you either generated something that was fation something that was useless or maybe uh you know offensive or things like that so you need to have both of that so you can basically start to build those and fix it quickly and get closer and closer and closer to the to the point that that you want um and I I think you would have to kind of Define those and Define where is the place that you want to you are ready to ship that product so um usually to me it's kind of like you build it you you test it you test it and and you start with a smaller Community or or group and and try to build it and you define and say hey if we get to this point we are going to be able to ship it uh or we are going to be able to ship it with for these people or that I think this is very important to to have those defined and it's not a straightforward sometimes you need to talk to a lot of different stakeholder and uh as I said it's all about managing there where you are so it's not like a mathematic let say to be able to kind of uh optimize it as you go um and again makeing sure that you are giving a balance to all of this if you have a um AI that is just very restrictive but it just doesn't do anything that your customers want you know it's just uh it's not good product but and on the other side it's true let's say it's just always answer your your customers but at the same time it also create a whole bunch of Poli naations or or U does these things that we saw like those chat box so it's it's very important the other way is that now what once you have your this matrixes how would you try to kind of improve that right so again each of these things are really for itself it's a big talk but um what kind of infrastructures you want to build to be able to kind of uh improve your system right uh one of it is that you need to have a feedback mechanism so without a feedback mechanism you're basically blind you don't know what's going on and you might kind of just learn something from like a Twitter right so but you want want to have a feedback mechanism and this feedback mechanism we've saw it a lot you know it can be um um inside a that UI with the thumbs down thumbs down like asking users to give you feedback immediately like for example um GP does it really greatly like every time that they responds you get it options you get it thumbs on Thumb Thumbs Up down and you have ability to kind of FedEd with the um information um so those kind of stuff is interesting getting the feedback at the time that user is not happy with your with your outcome is I think it's very crucial like um and a lot of companies do that like Tesla you you are doing your even like you're doing your self-driving like if you're using self-driving options as soon as you start to get out of it and uh you disengage it it asks you to give a feedback and and even when you are driving you know you you can talk it it ask you to talk to it and get a feedback so it's very important to be able to get that um AB testing as usual like being able to test whole bunch of different options um with different people is is is very important um you can also like sometimes I mean even the GPT does that is that you can also generate two kind of results right so you can say okay I'm I'm going through this request or this scenario that you have and now I have two results which one do you like more right or uh you can you can show it to the same person you can show it to the different group of people at the same time uh but these are very important for you to be able to build this especially now that everyone is early the more you know the more and faster you can actually get closer to that point it's it's more valuable so I think that's a t also one important thing is that when you're thinking about your scenario it's good to kind of have a measurement from end to end of your scenario when the user is actually start to engage in this this user story and when she's successfully or maybe non successfully is out of it so if you have a way to measure that and you have a number of outcomes that are which how many times it was successful or not successful and Telemetry there it can help you to understand where you are and where in the unpredictability is and see how you can address it uh and iterative learning U it's more involved a lot of people cannot do iterative and reinforcement learning but if you have the ability to it uh that can be really helpful and that's kind of a you a word of like GPT being able to do a lot of things and getting more stronger if you are trying to build your small team that it's just you and you want to make money out of it it's very important to be able to learn and and get better in that a specific task compared to the general models that are getting better every day uh the final topic that I want to talk about so I think the first topic was like just generally how the reliability and alignment and how you want to make sure that your product is actually do what you expecting and it's kind of you define a range of areas that you feel your product might hit and have a kind of plan for it but the other part is a performance cost it's easy to say that hey I'm just going to go and focus on this scenario and make it happen and completely forget about the performance cost of L and traditionally it was really okay because first of all we wouldn't really get to the performance problem anytime soon right like when you're are creating a classic applications from the point of you kind of relasing to the point that you actually have a performance problem is probably a few years in your start of life right or even your your your initial product right so it wasn't really as hitting us as fast as now with the L right so with L um let me go to that next slide uh some of these models are first they're very expensive um you might try it and say oh this is just few cents or a few dollars but it adds up very fast especially if you're are trying to give it them more context right GPT for is going to have 120,000 token context size which is great because now you can solve a lot more problems right so you might be very excited so now I can just put everything I have and put it in a prompt and get get the result out of it but you have to pay for it so it's good and powerful but it also means a lot of money and also the the the performance of it and some of this performance botle name it might not be able to easy to solve it because now to solve it you might need to completely go to the different model or even the technology might not be here or you're in this word of GPU shortage you might not even be able to get the enough gpus for your scenario to be able to actually make it better so it's very important to to design your scenarios and think about the performance think about it is my scenario going to per be performance sensitive and what's the range that users are happy to accept it and if I'm taking like seconds to kind of go and do my inference and all those kind of other stuff that I have to do um and remember it's just in a production you're going to have a lot of traditional layer you have your inference and then you have a maybe looks up and research and and then you have all those um AI safety layer that they have to make Mak sure that the output and inputs are kind of on a safe layer so it's this is going to take some time so think about your that scenario and uh Define your performance range um experiment that if you can hit it and if you cannot it might be still fine as long as you think about the way to kind of still keep your user engaged and and so she won't get bored or you won't get bored and and and you still have them um also uh yeah so and the models are important like don't start to say oh I have a gp4 I got access to it and now let me solve all this problem with gp4 uh yes you're probably going to get better answers out of the box faster but sometimes this G these models are extremely different on a performance like I think I think sometimes like g54 can be 10 times slower than 3.5 right or even more or even less really depend on your your configuration so it might takes you longer to get a 3.5 to work and create what you want compared to the four but the cost and performance might actually worth it and also there's a lot of other models that out there and I think there is a there is a good there's an area that I'm really excited I love to study about it is all the smaller models that people are building that you can also run on your MacBook like on my MacBook um uh with my M2 machines it's it's a MacBook Air I could run a model with 5 billion parameters and 30 billion parameters even it was slower but I could do that and so it's interesting to kind of encourage to explore all other options rather than just saying oh yes of course if I ask gp4 I might get the result immediately but is it going to perform from cost or um speed um and explore other options that are out there so anyway uh that's kind of the end of it um the takeaway as I said I want to kind of point out to things that uh how some of the stuff that we used to be able to do trivially is now is challenging and we have to kind of build it into your scenario cost and performance is very very important and they might not be trivial to solve at the end of when you're kind of having your product ready and then you realize that oh my God uh it takes me 30 second to finish something it's it's not going to be good so think about those and um yeah prioritize user experience as well um cool