Feature Engineering for AI Product Managers - AI PM Community Session #1

Key Takeaways

uh all right hey folks uh great to meet you all my name is uh nikl I'm currently running a company called fennel which is a realtime feature space you know machine learning infrastructure for recommendations and fraud and insurance and other problems like that uh I worked with Mahesh at pytorch uh and before that I was running a group of 100 machine learning engineers and my group was responsible for recommendations and more generally AI mL of multiple product Lines within the company part of heed part of search most of groups dating events several more product lines in my previous life I worked at uh Kora um and you know I started a lot of their machine learning efforts as well based out of bar area uh struggling to balance my life with a one-ear old and uh a company that I'm running as well and a huge fan of rap so if any of you are rap fans you know hit me up and we can you know chat about that um all right so as you know mesh mentioned today we'll be talking about you know modern recommendation systems I'll introduce the idea of you know machine learning going real time and I'll take a few minutes you know talk a bit about you know what Fel does and and through that you'll hopefully understand more about the process not just about the company uh my promise is I will not use any jargon I will not use any math I will not use any technical terms uh there have been multiple generations of uh recommendation system Technologies it's become a Hotpot out there I'll try to give a systematic you know thought framework to thinking about all of that and then I'll show you that all the different ideas you hear about the special cases for single in underlying part framework so let's start with you know the blueprint of modern recommendation systems so at an extremely simple level you have let's say a million or billion whatever some crazy gazillion number of things that you could choose from uh and you have to show only top five top 10 to the end user um at an extremely high level you know if you were to do this yourself manually or you know you know someone were to intu about it one simple idea is you know let's first bring down let's say 100,000 10,000 some number of candidate videos to show candidate products to show uh because out of a billion almost most of them are not relevant to me as a user let's say some of them are let's In tha language some of them are in Spanish I don't speak either of them they're clearly not relevant to me right so if you just filter out to my universe you can probably bring it down to a small number quickly and within that Universe of things that are relevant for me you then can do some sorting some ordering on them and give me the final results the first phase is called as retrieval where you're retrieving items basically sometimes it is also called candidate generation because you're generating candidates um and then you're ranking them sometimes you might hear the word r ranking so if you hear your engineering teams talk about retrieval and reranking this is all they're talking about there's no rocket science behind that um now this is a very simple view there are couple more stages in Real World Systems uh I'll talk about stages 1 to five I've added couple more details and I'll you know help you visualize that as you go along uh but what is happening is you know you retrieve some candidates you filter some of them out you then somehow rank them and you give the final recommendations to the end user and user then reacts on something they like some videos or buy some products which then comes back and becomes you know underline data set using which you can you know improve your recommendations for them that's all basically this slide is really saying so let's go through some of these stages one by one and understand a bit more about that so the very first stage is what is called as retrieval um generally you filter down the inventory to few hundred maybe thousand items and then do some more heavyweight processing of those the goal of this phase how however you do it techniques don't matter what the what really matters in the end is you are able to bring all or enough of the good candidates if you're bringing noise along with it don't worry about that that will get filtered out later somehow so let's say for me if you are bringing let's say I'm a huge fan of &m if you're bringing new &m videos what whatever else you bring in the retrieval phase almost doesn't matter as long as you bring the golden inventory the golden content for the specific user so call is the name of the game when it comes to retrieval and if you further dig deeper into retrieval it is actually very very simple you know people Engineers we all like to use buzz word but it's actually very simple it's a bunch of heuristics and each heuristic gives us you know some number of you know items let's say 10 20 50 items you line up you know 10 of them you take a union of all of that and these are your 500 items at that point of time so if let's say I were to ask you uh you you know relatively little about the user at this stage you don't have machine learning you know capabilities how will you find those thousand things which may contain golden content for me as a user well you might say that you know what let's look at the most popular content in all of all of the platform and bring you know 10 of those they're probably good we don't know if they are personalized but maybe some chance that you know there is good gold golden content here you might say that let's look at the you know artist that this person follow let's say we YouTube I literally follow some pages you know bring out the five new pieces of content that arrived from those pages these are all both valid heuristics you'll write more heuristics literally at this level of detail only nothing you know more complex than that and take a union of those um and then you can you know begin to add some more complexity here so you can then say that um you know look at the content I liked in the last one week then for each content look up the Creator I have not explicitly followed the Creator all I did was light some content from those and then now bring you know five pieces of content from each of those creators so these are my point is these are all simple heuristics at the end of the day which and you're using your intuition and understanding of the product to intu where could gold be for this content and if this is not accurate it brings noise like I said doesn't matter what is important is what truly matters should come through from one of these you know generators as they are called of course you know things are truly not that simple we end up doing a lot more machine learning within this phase as well you might have heard of you know Vector search embedding nearest neighbor lookups collaborative filtering these are all different techniques of filtering out the inventory a little bit but they have their own problems as well so if you look at any real world recommendation system there are a couple of generators that are based on embeddings and math and machine learning but there are also several generators that are literally based on the ideas as simple as you know I've been you know giving examples about so in the end you take some in advanced machine learning some simple product idea some trending content some recent content you know things like that get 15 items from all of them put them together and that becomes your retrieval effectively now once you've done your retrieval sometimes it brings invalid inventory like forget even personalization literally invalid inventory which you need to filter out maybe let's say in case of you know uh you know e-commerce maybe you know there's a item that is now out of order there's just no point recommending it to any user whatsoever or maybe let's say you are you know let's say you are Netflix and through this mechanism you ended up retrieving some inventory which there's no licensing in the country of the viewer for that content there's a bunch of reasons like that um so you basically want to filter that out and that's all the second stage of filtering is about you define what invalid inventory means to you and try to filter it out um some products in fact this is the norm these days there is a notion of personalized filtering as well which is you know kind of oxonic but you know it is what it is and a lot of you know content feeds in particular try to filter out content you've already seen before and so so they then extend the definition of invalid inventory to say that if I seen this specific you know uh you know real on Instagram before not eligible for further circulation to this particular user either way the concept is very simple you got some inventory out through a bunch of youris and some Advanced machine learning and then you filter out things that are not relevant to you the next three stages feature extraction scoring and ranking are basically expansions of what I was calling as ranking or ranking earlier retrieval and filtering or expansions or what I was calling is retrieval so now you have let's say 500 you know contents or you know 500 products you have to somehow order them and choose top 10 out of them how would you do that well the intuitive idea is you want to assign a score to each of them and do some you know ordering based on that maybe so the idea of assigning a score is exactly what storing cage is and the idea of ordering them is exactly what the ranking stage is you have to come up with score somehow which is where feature extraction comes in a feature is nothing but a signal about the item uh which is useful in the task of recommendation system so for instance you know what is the creator ID of this video or what is the zip code where this product sells the most or how often has this user interacted with this particular vendor all of these are signals you can come up with any signals there is no right or wrong here there is no such thing on this is a good signal and this is a bad signal some signals are more useful than the others for the task you have at hand but these are all valid signal these are all valid features as well one thing that you probably want to take away from this is almost always it's been seen you know decade over decade by now and it's somewhat counterintuitive behavioral features end up dominating uh instead of you know static features so you know the natural human way of modeling this is you might say that okay let's look at the topics this user follows and let's also look at the topics of this content and clearly in a machine learning model should be able to match on them somehow that this does not work if you look at you know specified preferences of user that almost never works instead what you need to be looking at is how are people actually behaving what are they clicking on what are they liking what are they watching what are the listening what is the actual behavior of the people on the product instead of any other stated uh information or preferences and you want to use that for future engineering but either way we have thousand you know pieces of content from retrieval we filter out 100 we are now down to 900 for each of those 900 we start extracting a bunch of signals let's say 20 predefined signals imagine that so for each of these 900 we extract those you know 20 signals for each of them once you have extracted those 20 signals you then need to collapse them down into a single score uh but before that in terms of you know features like I said behavioral features are you know what matter a lot so if you look at successful features this is typically how they look look like you know you choose some user attribute some you know item attribute some behavioral action you know some context you know some aggregation mechanism some rolling window and a typical feature might be let's say for a given uh you know um let's say content You' say let's look at the click rate of users of a certain age demographic on the publisher of the content that's a that's a you know typical feature you'll see in a real world recommendation system and what this feature is supposed to capture is the Affinity of a certain age group with this publisher so you know a more direct way for doing this could have been that you ask the publisher to tell you what are the age groups for which the content is valid but like I said stated preferences almost never are you know reliable source and they just almost never work instead the behavioral substitute for that you simply look at are people from a given demographic interacting more with that content if they are then you have behavioral signal that this page is relevant for you know people of this demographic by taking different cuts like this instead of you know taking a cut on you know age you could have then said instead that you know how are people of this gender interacting with this content these are still very you know broad demographics you can go more interesting things as well you could say of the people who follow this particular group so that limits down people of a certain taste and certain you know interest within those people what is the click rate with this other content as an example and so you are basically finding some user cohort some you know content property you know taking some Cliff Etc on that but either way once you have you know 900 pieces of content left 20 features each about all of them you need to somehow collapse them into a single score because how how can you even compare you know 20 different numbers across you know 900 different things that process is basically what scoring is called you could build a simple heuristic you that I'll take the average of those 20 numbers that's a reasonable scoring mechanism it'll probably not be very good in practice that's a different thing but it is a valid mechanism you could say that you know I will have only one feature I would not even have 20 and that feature itself becomes a score it's also a valid mechanism completely all of these are valid mechanisms really you don't even need to have machine learning model of course machine learning models end up being one of the menu ways you can score your content and collapse those 20 numbers into a single number you could do know L linear regression or you know statistical techniques like that that many of you may have experienced with or you can bring in the big guns and neural networks and you know things like that these are all simply mechanisms to collapse those 20 numbers into a single number now the benefit of machine learning is it can learn many more patterns than you can program yourself manually and it can learn directly from the underlying data itself and so there's a you know you know bit of you know it imp improves itself as data arrives and it can capture very very complex Advanced patterns but at the end of the day it is simply a pattern matching thing where you say oh if this feature is high and this feature is low then give a score of3 but if this feature is high and this is low and this is low as well then instead give a score of 0.1 only that's basically all that is happening under the but you do some scoring either way and once you've done some scoring you basically order things based on on that and that becomes a final ranking this is what it is in the beginning slowly as you work with an industrial product it begins to get more complex know sometimes you'll uh you'll come in and say you know what if I have you know four pieces of content next to each other all you know all from you know KT per it's probably not a good idea we should inject some diversity somewhere um or you might say that uh you know you know we have this content that's showing up it's a real story at Kora you know we have this uh you know good thing where our content at long shelf life but if you open you know Kora and the first seven pieces of content are all let's say three years old it creates an illusion that this is a ghost to now that may still be the most personalized most relevant content that's probably going to get the most engagement from the user but it has now a different product dimension on which you know what let's add some more you know perturbation to it of some degree and you know give a penalty if the content is very old so to speak either way this whole process of going from scores and some other meta information to final ordered list is basically what I'm calling as ranking now I'm not describing stages 678 here you know they are a little bit more technical I think from you from a product point of you need to understand these five stages obviously I'm somewhat lying when I say that it's not complex it's very very complex these are just know basic building ideas but there's a lot of math that goes in behind all of those and they quickly become very complex but if you understand this blueprint once every other idea you might have come across for recommendation just a special case of that if you want to say that let's just order things by you know creation time literally the first recommendation system at Kora was you know reverse chronological sort of content you can almost visualize that there is no retrieval in that system you have a single feature which is a creation time of the content and your scoring is basically just that and you're ordering on that there is no diversity nothing of that sort if you have some heuristics you want to say that now let's look at the number of likes divide that by number of Impressions and that ratio is how I will run content valid you have no retrieval you maybe two or three features maybe like count impression count and so on and your scoring is now a manual weighted average or ratios or you know things like that if you have let's say let's just sort it on popularity you now have a single retrieval generator which is you know most popular content and you don't have any features any sorting any scoring nothing on top of that you literally just have a generator and whatever output that creates those you know 100 things you take the most 20 popular content and that becomes your final recommendations collaborative filtering content based filtering neural LS two Tower networks combination of all of these they're all basically special cases of this blueprint so as a product manager I think to me I think what is the most valuable place for you all to create value is understanding this blueprint and asking yourself does my product need work on the retrieval do I need to be filtering out more invalid inventory that I'm not filtering out what signal should my team be thinking about how do I go about it collapsing them to score and you can share your product constraints with the engineering team and they can usually convert them into math quite well but if you start telling them about two two Tower neur like you don't know what you're talking about and it's probably not going to work anyway so um anyway so all of that is good but I think another thing that's happening and probably happening in all of your companies is machine learning is going real time everywhere which uh which is also somewhat counterintuitive because you know the general impression is isn't that only for you know Tik toks and Facebooks of the world and Google of the world you know we just need to get something going you know at20 and all of that jazz I should just get something out probably be good I don't need to do fancy stuff that's actually not the case so let's look at why that's actually not the case here it is really really bad for product experience for your recommendation systems more generally machine Learning System should not be real Time Imagine a couple of scenarios let's say I just signed up you have a batch pipeline that computes recommendations every midnight I signed up let's say you know 7: a.m. for the next 17 hours there are no recommendations for me or you're at least not you know utilizing the information I'm giving you to give me something personalized maybe you have the most popular content and you show me 10 of those or something they're not a good experience and I think we all know that the retention drops most rapidly in the first couple of sessions itself similarly imagine you are on the other side of the marketplace there's content a producer you know vendor whatever they come in they upload some content and for the first 24 hours they don't get any distribution for their content you know they're probably not coming back their attention also drops um and there are many many more reasons I've written a whole post on it there are seven different reasons I've identified at least s there are actually many more why you know you need these systems to be real time but if you care about cold start and if you care about long tail which you almost certainly do in modern economies like modern digital products you probably need to be somewhat real time and you know one thing is theorizing about these things the other thing is you know what has been seen time and time again so you know science Works machine learning works and similarly realtime machine learning ends up producing significantly better products as measured by the topl line engagement and top line revenue and you know those kinds of things so when you are leading your team to set up machine learning don't brush it off as a you know like a like you know first world problem let's first build a 820 solution this actually pretty fundamental part of how these things are built these days the next question that comes up is aren't these things too costly well um imagine a counter example where you are going to pre-compute things every midnight you don't know what users are going to log into your platform the next morning and so what you'll do defensively is you'll say let's pre computed for literally damn everyone and probably you know unless you are Facebook only 5% of your users are going to come out a given day probably less than that and so all of that 95% of the computation and storage you did is now wasted when you go to realtime world the per unit cost of computation is higher but you end up doing much less amount of computation because it's done incrementally and it's done on demand instead of you know pre-computing gazillion things every every hour every day and then throwing almost all of it away so it is important for product experience and from an infrastructure cost point of view it actually ends up saving you money as well and then you know the last question sometimes people have isn't it too hard to build isn't this you know too uh you know too much of a heavy lift and we don't have you know head count for that well these systems historically have been harder to build but good tools are coming in which make it easier and I think what people also don't understand and recognize is in many many many ways these are actually simpler to build and operate compared to Bath Systems so imagine there's a bath system that is running you know it last and you know you know 21 hours ago and there's very complex Jungle of pipeline that feed into each other and from that comes recommendations and your CEO goes to the product they're like this is a horrible recommendation why am I seeing this you know good luck backtracking from there and debugging and then trying to improve that if you're trying to you know do AB testing which you should be doing all the time uh how many versions are you going to pre-compute for the whole population it's going to be really costly really really quickly um the pace at which you can run AB test ends up getting limited by the frequency with which you can compute these batch pip there are many many many many reasons why in practice these actually turn out to be a lot harder to you know build and operate compared to a well-built realtime system with the right thing so I think the second takeaway in addition to the five stages and where you should be putting your Leverage is if you see your teams telling you that or maybe your Executives or maybe yourself telling you more likely than not let's you know think about all of these stuff later try to resist that tempation at least investigate a bit further and you'll usually find that it's in a high enough Roi okay since I have the platform I also want to you know use a few minutes to talk about you know what fennel does and um obviously you know trying to build up you know awareness of what my company does but through this you'll also understand some other important aspects of you know real world concerns that actually matter um so you know we are what is called as a realtime ml feature store uh when you are doing feature engineering you need some mechanism some infrastructure to compute all of those signals and keep them fresh as new information arrives and do low latency serving them but also you know uh training data presentation a lot of complexity drift detection monitoring governance all of those things um I I worked at Facebook for a while my founding team worked at Facebook as well um atct of Facebook current CFO of Facebook head of AI platform at Facebook they're all investors advisers in our company along with you know creator of Kafka and you know cofounder a bunch of companies and you know VPS and more companies than I count um it's based on ideas built at uh Facebook through I'm going to say maybe 8 years of hit and trial not by me or my teams only but through you know thousand different machine learning Engineers so standing on the shoulder of giant sh um Facebook is obviously several years ahead of the rest of the world so we have the luxury of you know doing kit and trial and figuring out you know how to build and operate these systems but at the same time it is adapted to the needs of the rest of the world most of you do not have 100 gazillion you know size inventory and like a several bazillion people using you every day but you know the same idea has been adapted to your scale and your needs um now these are the problems that we set out to solve and these are the problems your teams are experiencing and if you already work on machine learning you probably hear about some of them on a daily weekly basis and some of them come to you from higher Executives as well as I said If you're trying to make your systems more real time which I think you should there's no reason not to in 2023 you want your uh you know features to stay stay fresh and update with new arriving data the signal that I talked about let's what is the click through rate of this user on this you know demographic whatever as new clicks are happening New Impressions are happening you want that to change and you also have to do low latency serving when the user does come to your website you quickly have to find out those 20 signals for those 900 candidates that's 18,000 data lookups in practice it is actually a lot more than that and so you know latency starts mattering a lot on the authoring side your machine learning teams and machine learning teams worldwide write Python and they are productive in Python the whole ecosystem is in Python in fact at py our whole strategy was let's build a very strong pyto EOS system that is exactly my role at P was as well and almost certainly the rest of your text tack is not in Python and not in data science friendly Technologies and if you force your machine learning teams to write Java you know good luck you're not doing any Innovation there I can assure you that and so you know you need some mechanism that they can continue doing python but then you know play nice with the rest of the tech uh stack you have you obviously want good unit testing cicd all of those things to maintain software health and that is really important because software Health translates into quality of features as well far too often what happens is your features are not what you think they are doing they get corrupted pipelines break they become you know null values is all over the place and the year worth of forward progress of machine learning has been undone because of that single bug and you don't find out about that bug for like six months until your C is literally shouting or you're telling you that hey you need to figure out why rml models are not working in production anymore every ml system drifts people's behavior shift and the models you trained three months ago they are not effective today and this is not a theoretical concept like every single time in every single context I have been if you take a model and you plot its Effectiveness over time it's high in the beginning but then Falls very rapidly and a few months in it's basically not you know more than maybe you know 60 70% of Effectiveness it had when you started with it and so drift is a real thing and you want to understand that more likely that not if machine learning starts working you will have many many use cases of machine learning very soon you may have a couple of headliner use cases like a recommendation but you'll be doing machine learning everywhere and you want to reuse the Machinery you built up you know reuse features you built up in some compliance Industries you need to know lineage of what the data feature depends on as well and finally there's a lot of operational burden of you know just running these things and you building them and managing the reliability and managing their Cloud cost and bringing it all down these are all the problems that you know we are trying to solve at Fel your machine learning team still write the features they still train the model they and you as PMS are still in charge of the product direction and you know product personality so to speak we are just giving you a piece of infrastructure to make all of that easier we are not giving you intelligence you know a bad you know metaphor I make sometimes is we are not giving you brain we are giving you the skull that will hold the brain so to speak so you still do all the intelligence it runs inside your cloud in a way the data and code never leave so if you are in a privacy security sensitive you get all of that as well and the effect of this is you get you know more powerful machine learning capabilities more realtime capabilities your models are better your products are better your business value is higher your metrics are higher your iteration Cycles have become shorter and so time to Market is lot higher both launching a new machine Learning System and pro to the first time and then ongoing iterations on it forever and all the dimensions of cost from Cloud to you know people cost you know are coming down we are easy to use we have a lot of focus on quality we operate on the whole spectrum of batch and real time in a very powerful way I will not bore you with the details here and that's basically all about Fel we as mahes said we write a lot about uh recommendation systems and more generally machine learning on our blog uh with the level of you know detail and Clarity that I showed here and so if you want go go to panel. A/B good educational content if you want to read about what fennel does you instead go to fennel dos and if at some point of time you or your compan is interested in evaluating Fel commercially for your use case just learning more in a casual way not even you know any commitment you can reach out to me I have my email here nf. and then you can take it from there I think that's all I have I want to stop here I think we still have some time for questions so I to take some of those down thanks Nik uh people from aroh feel free to ask questions uh nikel has been a great source of uh knowledge and learning for me when I was at PCH so please use this time go ahead if you have any questions others please write your questions I might be able to take one or two just raise your hands and [Music] go think there's a question in the chat uh yeah Moine you want to ask the question go ahead uh so I have some noises so if you can hear me I can ask yeah we can hear you can hear okay okay so uh my question is where is where the most transition is happening now from fash to like real time um machine learning maybe spal industrial uh or I can phrase it as like what is your ICP idea for your compan yes I think uh the answer to the first question is lit is happening everywhere um and and you know that's probably not the answer you're looking for so let me now give you a slightly different answer if you you know zoom out there are two kinds of machine learning problems in the world one set of problems that operate on you know language words and video frames and image pixels and so on and your CH gpts are a new addition new incarnation of that those are often by definition static like your video content is not going to change very much there's a different subset of problems which are based on domain specific you know uh structured behavioral data like fraud detection like Insurance underwriting like recommendations like Dynamic pricing like you know inventory forecasting and optimization all of those areas are going real time very rapidly there are some where it is you know like it's like it's just a business imperative for instance if you're trying to do fraud detection of any kind you absolutely have to do it real time there's no point blocking a transaction 24 hours after the credit card swi went through you have to do it in the moment and you have to use the information that happened recently if you're Insurance underwriting company or credit card lending company someone applies for an application you know you can take 3 seconds to get back to them but you cannot take you know 6 hours and so there are many many verticals where this is just a part of business imperative but even when it is not like recommendations system it's very rapidly going towards that direction the next question I think let me just read them out in spirit of time from RJ is how would explainability work with real time explaining why an add product was recommended I think I'll flip that question around and I'll ask you how will it work in the bat side and whatever you can do in batch you can do that in real time but in real time you can do a lot more so for instance uh usually what the way it is done is as recommendations are created like I told you that there's more complexity than giving you the hint for as recommendations are getting created people typically tag each recommendation with some metadata about it and pass it along the funnel and so that metadata typically has information about you know what retrieval generator created that so if I can show you that we are showing you this because it is trending or we can showing you this because you are following this page or something of that sort that's basically good enough for explainability point of you don't need to go the nuts and bols of how it happened you can do that in bch and you can do that in real time but in real time you can do a lot more what you can also do for instance is Trace a full path of a given piece of item along the full funnel on demand when you want to whereas in batch pipeline you know that is a lot harder to do does that answer your question a yes that does thank you so much I can go with the next question n uh when people are starting on real time like we learned like you know if you want to go with rule based or simple mathematical systems you need less data then you go to deep learning then you go to further large models the amount of data that you need to bring or build recommendation system goes to 10x every time at least what's the bare minimum funnel needs for people to get started I what kind of data you expect them to have yes I think in my experience you know the number of companies that need need large models for recommendations is probably in single digits as of right now and the number of companies that need deep learning for recommendation is in a single digit percent and so for almost all of us what we really need to be using is more conventional simpler machine learning there's a particular technique called as gradient boosted decision trees or gbds as they are called 98% of us will just be using that for all of our careers they work so well and they are so simple relative to neural Nets and the kind of data that most people have you know that that is needed for neural net most people don't have that kind of data for fennel in particular there is no scale limit even if you're trying to do simple rule based characteristics you still need features for that and a lot of the problems that I mentioned here some of them are skill related these problems but many of them are not and so final can still create value obviously Beyond a certain point it begins to you know make lot more sense but for fennel in particular there is no limit and outside of fennel in general I think you know you should probably assume that you do not need deep learning for your structure data problem you need them for inovision and language and those kinds of things but not for structure data problems n you want to ask your question go [Music] ahead yeah you spoke about this two two Trav model and we have user features and product features but I will I'm trying to understand how do we deal with negative samples like if we just learn uh what the user clicked then it could lead to overfitting right yeah so I think um so this is a pretty Advanced question uh but I think the answer to your question is negative so let me first help all of you understand I think with the question is trying to ask uh if let's say I am training my models only on the things that us clicked model could learn to parot that user is interested in everything and they still be correct on the training data on which I'm training my system which is clearly not the behavior we want with them to have a very fine sense of discriminatory B so to speak and so for that you need to do negative sampling there are many techniques of doing negative sampling you know you could do just literally random negative sampling you could literally Choose You Know random thousand pieces of content and just pretend that user saw them and did not like them and just say hey these are you know bad pieces of content for this user and that Works reasonably well and from there there are many more you know complex techniques of doing negative sampling but once you have chosen some you know negative examples you still need to come up with the same features for them you still need to you know do the scoring for them and then teach your ml models to be behaving in a way that the score for them turns out to be long answer your question yeah yes thanks thanks last question for today [Music] uh I'm seeing two questions here I don't know if video froze one is from Filan how do you deal with governance in your product uh we deal with governance beautifully I think that's a short answer uh there are a lot of nuances uh you know when it comes to dealing with governance we have automatic lineage uh understanding we do lots of different kinds of monitoring we first of all try to um have things built in a way that your system does not even get into bad states of data quality to begin with but if it does we have lots of different kinds of monitoring and alerting on it we also have role based access at you know those kinds of fancy things that you need to ensure compliance and governance so Lance if you are interested hit me up and we can have a conversation on that offline um and I guess in spirit of time the last question is from anushil how difficult is it to change a legacy recommendation system the goal changes from acquisition to retention or Revenue so I think I I probably want to mention one thing which I think is quite important the highest leverage point in a recommendation system is not the algorithms or the features or even the data I think the highest leverage point is the measurement and so if you as you know product manager are able to come up with what is the success criteria of the recapitation system in a way that it can you know be written almost like a math like it's very precise that this is the metric that we care about optimizing reworking the Machinery to optimize for that is some effort but relatively speaking a small amount of effort compared to building the whole thing from the first place what will most likely happen is you'll still do retrieval you and more or less almost the same generators will work you'll still have invalid inventory that you'll filter out probably the same way you'll still extract exactly the same features more or less because they have predictive power of user behavior only the scoring phase is one that will probably change it's instead of you know taking let's say weighted average of ABC maybe now you know you take different weights if you were to do heuristic as an example if you're training an model obviously you tell L model this is a metric I care about and LL model internally comes up with the right you know pattern or right formula internally to do that for you so the answer to this is you focus on the measurement side that then gets converted into the scoring stage and the whole recommendation system has now adapted in personality from there for