Stanford CS330: Deep Multi-task & Meta Learning I 2021 I Lecture 9

Key Takeaways

hello everyone uh thanks for coming today we'll be talking about the multitask and goal condition reinforcement learning but before that a few reminders homework 2 is due today homework 3 is out it will be a more lightweight homework so that you can spend a little bit more time on the start spending a little bit more time on the projects but you should start with it ASAP especially with setting up all the dependencies and so on to make sure that everything works and if it doesn't please post something on that then we'll try to help you um and secondly there is a mid-quarter survey today and we really appreciate your feedback so please take the time to to fill it out we've actually Incorporated a lot of the changes in the course because of that survey things such as switching to pytorch from tensorflow before adjusting the lecture topics the homeworks introducing homework zero and so on so we really take it seriously it's a please please uh fill it out and also thank you very much for the high resolution feedback that you've been that some of you have been submitting so far it has been very useful to us and we we take it really seriously so thank you uh one other point is that office hours can be actually either in person or virtual and there's been a few misunderstandings where somebody thought that it was in person and wasn't so please check the calendar the calendar should be able to tell you whether it's in person or virtual and follow that So the plan for today's lecture is first we'll do a little bit of a recap from last week and we'll finish the Q learning aspect of reinforcement learning which we didn't get to talk about and then we'll dive into the multi-task aspect of things so we'll talk about multitasking mutation learning and multitask policy gradients multitask queue learning and data sharing and then finally about the goal condition reinforcement learning which is also the topic of your of your homework all right so let's start with the short recap so last week we talked about the anatomy of a of a reinforcement learning algorithm and we talked about these three boxes so there is the box where we generate the samples when we run the policy there's a box where we try to estimate the return of the policy so what's the sum of the rewards that the policy is going to get and we have few different options here we talked about the Monte Carlo approach where we just sum the rewards that we achieved uh we talked a little bit about fitting a q function or the advantage function that allows us to do this a little bit better and we haven't talked about the model based part which maybe we'll talk about it next week or on Wednesday actually um and then there's the the last box where we improve the policy and the only part that we've talked so far about is the policy gradient approach where we take a gradient with respect to our objective and we do gradient Ascent procedure to improve our policy and today we'll briefly talk about Q learning part as well and then next weeks you will talk about the model baseboard and then we introduce this equation which expresses the gradient with respect to our objective and this is the gradient that we use in the policy gradient approach and if we were to parse this equation we have the part that corresponds to the green box which tells us what's the return of the of the given policy and this is just the sum of the rewards starting at the current action and the current state Up Until the End these are the the actions and state that we have actually experienced then we have this part that corresponds to The Orange Box where we do multiple rollouts and we take the average and then this whole part all together is how we estimate the gradient with respect to our objective and then we use that gradient um to uh in our policy gradient procedure uh in gradient in a sense to actually improve on our policy all right cool then we talk a little bit about multi-step production about how in standard policy gradient you would just sum the rewards from now up until the end of the trajectory we had this example from from a movie where what you could do instead is if you had an option to teleport to the state over and over again and live multiple Futures you would be able to estimate the sum of the rewards much better um so this is a different way that you can do this which is the true expectation of the rewards given that you start in a the sum of the rewards even that you start in a given State and perform a certain action and then we talked about the the multi-step prediction where you for instance play chess and uh you know when you play chess you don't really have to wait until the end of the chess chess match to update your predictions so you don't have to wait until you finish the game you can actually start updating it as soon as you make a move so if you made a move and you feel like you you did something wrong you can straight away update your prediction and then we talked about how we can actually use this in the so-called bootstrap bootstrap estimate and uh the way we do this is the following we try to fit the value function and we do this using supervised regression so we have our value function which is the neural network parameters with some parameters Phi that takes State as an input and outputs a scalar a value and then we need to fit it based on some kind of targets so this is our value function the text States us the input and output is just a single scalar that tells us how good is our policy what's the sum of the rewards from now till the end so one target we could have is the ideal Target which is the expected sum of the true expectation the expected sum of rewards under our current policy given that we start in a certain State and we can approximate it by taking the current reward plus the value function the next day and we can approximate it even even more whereby rather than taking the True Value function which we don't have access to we'll just take our current estimate of the value function at the next day and this is what we refer to as the bootstrap estimate so this is what allows us to make these predictions and update those predictions as soon as we make a step we don't have to wait until the end of the trajectory all right so we talked a little bit about that and then we introduced two algorithms we introduced the reinforce algorithm where we just sample a bunch of trajectories then apply our policy gradient formula to estimate the gradient with respect or objective and then the gradient ascent and then at the very end we talked about the online acrocretic algorithm where we also do a bunch of different rollouts we save this kind of Tuple State action next state and the reward then we update our value function according to this bootstrap estimate that we discussed in addition we evalu we evaluate the advantage function and then we still do the we still compute the gradient but now it has a slightly different form and has this Advantage term here as opposed to the sum of the rewards but once we do that we apply the gradient Ascent procedure and we do this over and over again all right so this is just the prediction Parton that's when we finished last time are there any questions to this part all right there are no more questions than let's talk about how we can actually use these banks these Q functions value functions and Advantage functions to improve the policy so not only to estimate better what's the sum of the returns but how we can make the policy itself better so far the only way we're making the policy better is by doing gradient ascent all right so so far we've been doing this policy grading approach and this was our way of computing that gradient uh or in the actor critic way we will just replace that with the advantage so we could make the green box a little bit better by fitting our value function here but now the question is how can we improve the policy using some other procedure not necessarily using gradient ascent and to think about this just remember that the advantage function is the difference between the Q function and the value function so the advantage function tells us how good is an action compared to our standard policy Pi that we would run otherwise so to give you a little bit more of an intuition of what's going on let's go back to our exercise that we had last time so we have this little stick figure that is dreaming of being a drummer right and the way that this stick stick figure will be evaluated in their dream is that um they will have to play in a month this little drumming test and if they can play they'll get the reward one and zero otherwise right and they have these free actions to choose from they can either chill they can look they can watch other drummers perform and try to learn from that or they can actually practice drums and they start at some State as the let's say St equals zero so at T equals zero so they start at the very beginning and the recurrent policy is this policy where they want to chill all the time so the the probability of taking action one is one so they always want to be doing this and then we talked about the value function Q function and the advantage function in this case so to test your understanding let's change this policy a little bit let's say that uh it's not a stick figure that wants to chill all the time but it's a stick figure that just wants to watch people play drums all the time so I will say that the pi of A2 given s is equal one all right so it just wants to watch people playing drums so what do you think is the value function in that case yep um all right yeah let's say something like 0.25 so you would have like a fourth probability one fourth of a probability that you will actually play this thing in a month by just watching people playing drums all the time okay I think that's reasonable what about the Q function of our current policy Pi so Q Pi of St comma i t radius here so Q function tells us what's the expected sum of rewards given that we start in a given state so this is the same state we start at the beginning given a specific action and then from we perform this one action and then after that we continue over our policy pi so what do you think this would be yep foreign so this will be a scalar for a particular action right so you're saying that the Q of s t where T equals zero of a comma a one A2 okay so if a t is equal to A2 it would be equal to what sorry foreign yeah so it will be higher than what A1 would be but lower than what A3 would be but it it would it should be exactly the same as the value function because the value function was thinking that you're always your policy is always acting as if the A2 is the action that you're performing right so the Q function for St and A2 should be equal exactly to the value function so we established that there is one-fourth of a chance that you will play this thing at the end of the month so let's say 0.25 does that make sense okay okay and what would it be for A1 then yes that's right it will be slightly less than 0.5 that's right because you perform this one action where you just chill for this particular moment but then after this you'll continue watching um people play drums and for A3 it will be a little bit higher all right great so now what about the advantage function and let's just consider the advantage function only for A3 all right so the advantage function would be of SD comma A3 yes one positive number yep that's right that's right so the Q the advantage function is the difference between the Q function and the value function so it kind of tells us how much better is this action compared to the action that I would usually take all right so given that we have all of these things we have the value function let's assume that we know it we know the value function the key function and the advantage function and we have our current policy and let's say that our current policy we go back to chilling all the time so it's this so given all of these quantities how can we improve the policy do you have do you have any ideas of a simple algorithm that would allow us to improve the policy yes that's right yeah so just to repeat the the answer we'll have the positive Advantage for A2 and A3 so we could choose among these two and that would if we do that then we'll have a policy that is better than the policy we had before yeah that's great cool so in particular what we're trying to do is we have this Advantage function that tells us how much better is a t than the average action according to our policy and one particular way we can do this we can take the ARG Max of that right so we don't necessarily have to choose between A2 and 3 we can just evaluate all of the actions and take the arc Max and that action will definitely be better or the policy that performs that action will be no worse than the policy that we currently perform and so it will be at least as good as any action from the policy regardless of what the policy actually is right so we don't need to have access to the policy we just need to have access to the advantage function cool so let's say our policy would look exactly like this so with probability one we'll be choosing the arc Max of the advantage function and this will be our action one second and the probability zero we'll be doing anything else yes it seems like the hardest part is function because is it common in practice that actually fits very accurately yeah that's a great question so the question is uh it seems that just fitting the value function is the problem and uh whether it happens in practice that we can fit it very well yeah so I think in practice we usually don't fit it extremely well we also use algorithms that we'll learn about in I think two minutes or so where we are constantly changing the underlying policy and the value function is kind of constantly changing because it's evaluating value of different different policies all the time and we don't fit it perfectly but we fitted well enough to actually get better and better policies but you're right fitting the value function is actually difficult cool all right so we have this hour we have this new policy that we know that should be better than our current one and uh this is what leads us to an algorithm called policy iteration so it's a very simple algorithm where in the first step we'll be evaluating our current Advantage function right so the advantage function of the current policy and once we have that Advantage function we'll set the policy to this new policy and we'll do this over and over again and the new policy will be equal just the arc Max of the current Advantage function right so very simple algorithm and we are iterating over the policy um so that's why it's called policy iteration okay cool and as before the advantage function is equal to the Q function which is the reward plus the value function the next time at the next step minus the value function of the current step all right so now there's two steps that are sort of magical that get us to the final algorithm that is actually extremely useful yes there's a question right so the question is why even bother defining Advantage function if it's defined in terms of Q and V why can't you just use Q instead um yeah that's uh so the reason why people usually use Advantage function is because it's better normalized right so you subtract kind of the the think that that is your Baseline which is the value function uh however we'll just learn about it in a second people also use Q functions instead yeah so because it's normalized it gives us another variance and because of that we can deposit ingredients all better all right so now let's discuss two steps that are kind of interesting that would allow us to get finally to to Q learning algorithm which is applied all over the place all right so we had this policy which was taking the arc Max of the advantage function and when we're doing this Arc Max what we're actually doing if you remember the the drummer example we do this argument just to give us the index of the action we should be picking right so we take the arc Max over the advantage and that just tells us pick A3 right it just gives us the index then we know that the advantage function is equal to Q which is expressed like this minus the value function and we know that the arc Max of the advantage function would be exactly the same as the arguments of the Q function right so we can remove the value function because it's not dependent on the action and the arguments will be exactly the same so let's do that it will be a little bit simpler there's we get rid of one term so now we can just say that the Q function is equal to reward plus the value function at the next step cool so now we can change the The Arc Max here to be the ARG Max over the Q function and in the policy iteration algorithm rather than having the advantage we'll have the Q function over here so we kept everything the same we just changed advantages to Q so now here's one track that we that we'll start with so because we are using this Arc Max just to give us the index and then as soon as we have that index we'll have to evaluate it in the next step again right so we kind of have this new policy but this policy is only useful as far as evaluating the the Q function of that new policy we don't actually use the policy itself we just use the value of that policy in the next step so the idea is let's skip the policy entirely let's kind of get rid of the middleman and just compute the values directly so the ARG Max of our Q was the policy and the value of that new policy would be actually the max of the queue right so the argument was kind of giving us the index of the action but the max of the queue is giving us the value of that new policy which was the The Arc Max right this is a little bit of a a little bit of a trick but it's actually extremely useful all right so this approximates the new value right and that leads us to a value iteration algorithm which is kind of the a similar version to policy iteration but now we'll be iterating on the value itself so this is how it works we set the queue to be equal to the reward plus the value of the next step so this is what the Q Q function is but then they will update the value function to be the max of the queue all right so this is kind of like in the previous step here we were updating the policy here we are updating the value of the new policy so because the new policy will be the arc Max of the queue we can just say that the value of that new policy that is the r Max of the queue is just the max of the queue and we do this over and over again and we are iterating on the value and this is the value iteration cool so that means that in the green box we have this queue now and to improve the policy we are just setting the values equal to the max of our actions of the queue so now the question is we have this algorithm right and now we can we have to implement it right like we have to have some neural networks that actually um learn something and try to fit either the value function or the Q function so this is called value iteration so let's maybe start with value function all right we'll try to have a neural net that represents a value function so it takes a status input and outputs a scalar which is the expected sum of Rewards and uh let's see if we can do that all right so let's write it on the board here so let's see our value function V of s parameters with some parameters pi would be equal to the max over actions of the queue but we don't have a neural network that represents the queue so let's Express Q in terms of the V and we have that in the in the line in the in Step number one so this would be the max of r s comma a Plus this counted expectation of the value at the next state all right and this expectation is with respect to P of S Prime given s comma I all right so this would be our Target for the for the value function right and uh we'll be trying to minimize the difference between the two and we'll optimize the neural network to do that there's a question yep that's a great point so the question was uh don't you need to know the transition Dynamics because you're taking the expectation and you need to know what the next state is yes that's true so there's this problem of this equation which is if we were actually trying to implement this we have the value function that takes a state as input so we have state that is available to us and then we have to take the max over all the actions that are possible right so we can try different actions so we would need to know the reward function let's say that we know that for a specific State we know given a specific action what the reward function is but then we would need to know what would be the next state that I would end up in given that action so that we can compute the value function of that next state right but to know this we will be testing different actions right because we'll need to take the max of our actions so to know this we would need to know what would be the different states that we would end up in given those different actions right we would have to you know let's say we have 10 different actions to choose from for each one of these actions you would need to know what the next state is so that you can input this to your value function to your neural net so you need to know the transition Dynamics which was which was the question yep so the question is can you just randomly sample because you haven't expected value [Music] right but to sample from this distribution you don't actually have access to all the different actions that you could perform you only have access to the actions that you did perform yeah all right okay so in that case you would need to kind of know had I performed that other action that I've never actually tried what state would I end up in and you don't have access to that you don't know what what the future would be all right so the value function doesn't quite work and that's what leads us to the final algorithm Q learning so again we have the value iteration algorithm but now instead of fitting the value function let's just fit the queue function instead right so we keep everything exactly the same but now instead of learning the value function we'll be learning the Q function so let's do that second okay cool cool so now our targets will will be trying to fit a q function parameters with some parameters as comma a and this would be equal to the reward of s comma a Plus the expected next value and the next value is the max of the queue right so the Max over actions this is a different action this is not this action that is the input this is let's call it a prime um of the Q at S Prime comma a prime all right and this is this is the Q function that is parameterized with the same parameters right so it's exact same thing now we're fitting the Q function instead of the value function yes how are we getting the S Prime yeah that's a that's a great question so let's let's think about this equation first and then we'll get how we how we get the S Prime so now we the input Tower Q function is State and action all right so now our Q function looks a little bit different right our Q function is now uh has two inputs states in the action and then outputs the value right so this is expressed here and given state of an action we have access to a reward function so we can compute the reward and now we have this Max term again but now it's a little bit different so the max is taking the max over all the actions of the Q function of the next state right but now let's ignore the next state for a second but in terms of the actions we can actually do this right our Q function is dependent on both state and action so we can try we can plug in all kinds of different actions that we haven't taken and compute the max right we don't have to simulate now what's going to happen next we can just plug in the actions because this is part of the input of our function and just try it out and see what happens now how do we get the next state so the way we would do this is actually similar to how we did what we did in the active credit case we would just be saving so this this next state here this is just the function of the current state and the action right it's independent of the policy or of the Q function or anything like this so what we'll be saving as we as we run the policy in the world will be saving so-called SARS tuples so we'll be saving State action reward that we got and the next state that we observe and then from that we'll be approximating this expectations from the states that we have actually seen so we can just record as we as we run the policy we'll be recording these four values so that we have access to S Prime that we've actually experienced and here we will be plugging that as Prime that we've seen foreign [Music] right cool so this is what gets us to Q learning doesn't require simulation of actions and we can do this so a quick recap a few definitions that we introduced here we have the value function that tells us the total rewards starting from State and then following our policy so in other words in that answers the question how good is the is the state then we have the Q function which is the teller reward starting from s taking a single action and then following the policy which answers the question how good is the is the given State action pair and then for this optimal policy we just introduced this this equation right here we have this this dependency that says the Q function the optimal Q function which will be depict of star so this is now the optimal policy is pi star is equal to this expectation and this is the reward that we got for the given state in action so this expectation is actually unnecessary for the reward plus the max over actions of the Q of the next action and this is what we refer to as the Belmont equation or the Bellman optimality equation and this is one of the most important equations in reinforcement learning all right so let's go back to our drumming example so we talked about the value function the Q function what is the Q function of the optimal policy so our Q star so this is not Q Pi which is the current policy and the current policy is you know the stick figure wants to chill all the time what's the Q function of the optimal policy Pi star so what's Q star any ideas yes because because that's right yeah so just to repeat this for everyone um a for A3 it would be one that means that you're practicing all the time including the current time step for A2 it would be a little bit less than one you just watch for a second and then you practice all the time and for A1 it will be a little bit less so you take a an hour off let's say you do chill and then you practice for the rest of the month cool and what would be the value function the V Star of s0 please just don't be shy raise your hand ideas or just shout out the answer yes that's right it could be one because the optimal action would be A3 yeah so this takes uh this assumes that you're acting optimally straight away cool all right so we introduced this fitted queue iteration algorithm which works like this we have a few hyper parameters so first we collect the the data set using some policy and we collect the source to put so State action next state and the reward and let's say we have a data data size and data set size n and there's some collection policy that we collect the data with then we set our targets for the queue function according to the Belmont equation that we just introduced and then we minimize the distance between the Q function and the targets and our Q function is this is this neural net that takes States in action as an input and outputs a single scalar value so we do this over as gradient steps right we can do this a few times and then we do this we we can iterate in terms of updating the targets K times right so we can apply a few gradient steps and then we would update the targets because now our Q function has changed so this Max term will be changing and it's an iterative procedure so we're getting kind of we're approximating Q star better and better over time and then we go back to collecting the data set and as a result we can get the policy by just taking the arc Max of the queue there's a question [Music] yeah that's a great question so the question is do we have to do in step two a bunch of forward passes uh one pass per a prime um uh technically yes but you can batch them together yep uh in terms of stuff I think you should do that fine but instead of my course but it's not foreign so we're saying that this queue over here should be a should have a different parameters than the Q over here yeah that's like uh right yeah yeah so this is one of the implementation um things that people introduced to make this procedure much more stable and they introduce a Target Network here because this network is constantly changing we're constantly applying gradient steps to it and it causes some instability in learning so instead what people do is they copy the values of this network once and then use that Target network with copied values here and then apply the the gradient steps to this network that's constantly being updated and every now and then they would update the target Network to the network that they've been using yeah that's a that's the implementation trick that people use that's right all right so a few important notes we can reuse data from previous policies right so here we are just using some policy it's not an on policy algorithm we don't have to be rolling out from the current policy this is any policy there's a question uh it's not a good question yeah that's a great question and we'll answer this in one minute oh yeah correct question though the question was uh how would you do it in the continuing how would you take that Max in a continuous case and can you batch the actions then um all right so um we can reuse data from previous policies it's an alph policy algorithm we don't have to be running the policy that we're currently learning it could be we could be running any policy um in practice what people do is they push these Source supposed to replay buffers and then they have one big replay buffer with all the data and they just sample from that buffer to do the updates and importantly this is not a gradient that's an algorithm right so even though we are trying to fit the queue function to the targets and this is where we apply gradient descent the the algorithm how we learn better and better policies um is a recursive iterative algorithm uh which uses dynamic programming all right so here we are constantly updating our Q function according to this formula and this doesn't require gradient descent we use gradient descent to just fit the Q function to the current Target all right so now a quick example and this is the example of continuous actions so we'll try to apply Q learning to robotics so this is our algorithm so now the question is how do we take that Max of our actions where robots move in continuous space right like they their actions are continuous they need to get to a certain pose for instance so we can just enumerate all the actions this is a continuous action so one way we can take that that Max here is by running a simple optimization algorithm and there is this gradient free optimization algorithm called cross-entropy method or cem and the way it works is actually really simple you would start with a normal distribution all right and then you would sample a bunch of actions from the normal distributions normal distribution and then score each one of them right so I would start with let's say 20 actions I sample all of them and then I query my Q function what's the value of each one of those actions right and some of them will have higher values some of them will have lower values so the higher values here are in Gray and then I can do another iteration of CM where I would now fit a gaussian to my Elites to the best actions that I the actions that achieve the highest score and then perform this procedure again yeah so this is a type of an evolutionary algorithm or a gradient free optimization method um you can use any other optimization method um to just find the max of a of a function this is just a very simple thing that was actually applied to the robots cool all right so let's do all of this so this is the work called QT opt done by Dimitri Kalashnikov at all at Google brain and the way it works is the following we store data from all the past experiments then we have a bunch of buffers the off policy buffer where we put all the data in as well as the on policy buffer where we currently collect the experience we have our queue Network which is parameterized by some parameters faded and then we have this separate process this is running at Large Scale these are parallel so we have the separate process that's called the Bellman updater which is just Computing the target values for the current Q function all right so all it does is just computes the reward plus the max of the next queue and this uses the CM optimization to get that Max and then once it does that it would push these targets to a buffer as well and then from this buffer we'll have a set of training jobs that are just doing gradient descent that are just trying to fit the queue function to that Q Target and update the parameters of the neural net this is heavily parameterized and running on thousands of machines cool so this was applied to grasping and the the mdp definition for grasping was the following so the state was the the image captured by the camera that was here over the shoulder camera and this is what the image looked like the action was four degrees of freedom post change in the Cartesian space so it was controlled it was the top top down grasp and it was controlling the position as well as the yaw the orientation here plus the gripper control so it could also tell whether the gripper should close or open and the reward was binary so it would just say at the very end of the trajectory if you grasp something successfully you get reward one zero otherwise that's it and we actually implemented an automatic success detection mechanism for this which was very simple so at the end of the trajectory the robot is either holding something in its hand or it's not so what we will do is we will take an image when it's holding or at the end of the episode and then we'll open the gripper and take another image now if you're holding something then the difference between these two images would be non-zero right because you just dropped something into the bin if you didn't hold anything the difference would be easier so very simple automatic success detection mechanism right so in terms of the results uh we used seven robots to collect over 580 000 grasps and it was evaluated on previously unseen objects so we'll just like drop a whole bunch of different objects into the bins let the robots run them run with them and practice grasping um and then we would evaluate it on objects that the robots never seen before and at success rate was really high it was 96 96 success rate and that was actually really surprising to us that we could get to a number this High because grasping is not a new problem people have been working on grasping for 50 years or so and this is I think the first time where we were able to see that this problem can be actually solved and we can grasp objects that the robots never seen before and here are some examples of strategies that the robots learned so here you can see that this grasping is kind of full of contact and it's interacting with the environment which is very difficult to prescribe or to engineer upfront so here you know it's trying to simulate the object that's moving the other objects apart and so on to grasp it here there is an example of a reactive Behavior where it's trying to grasp this tennis ball and I think in a second you will see an example where there's a mean experimenter and it's moving the tennis ball away and the robot understands that you know the ball is not there anymore and it follows it and gets to a state that has high Q value or here's one more case where it's a very flat object that is really difficult to grasp and the robot learn how to retry and kind of fully understand what it means to grasp something and it tries it a few times until until it gets it right so it kind of understands that some actions are some Q values for some of those states are not as high as they should be so it runs this Max to find the better action until the the moment where it actually grasps it and then succeeds all right so to summarize Q learning uh in terms of Pros we have uh this is a More Sample efficient than on policy methods we can incorporate off policy data including a fully offline setting that we'll discuss in in I think two weeks and we can update the policy even without seeing the reward kind of like in the chess match that we discussed before as soon as you make the next production you can update the policy and it's relatively easy to parallelize how we just saw in in QT up case and in terms of the cons there's lots of tricks to make it work including Target networks and other things and potentially it could be harder to learn than just the policy itself because you need to model kind of the landscape of all of the actions all right any questions yes um yeah great question so the the question was how do you encode the fact that you wanted to grasp a tennis ball um this was just an accident so we didn't encode the thing anyway so this is not an instance grasping system it just grasps anything that it sees yep right so if you let it run for a little longer it would eventually pick all of them one by one to download disease right yeah so the question was why didn't we use some other method like uh ddpg that is fairly popular that is another way of getting the max of the Q function in a continuous case um the main reason is because it was much simpler to do it this way and we didn't want to have a separate actor and there's actually a lot of instabilities that come with using an actor critic method like this so we just wanted to make it simple easy to paralyze and it was good enough all right cool all right so that's the end of the recap and Q learning let's dive into the the multitask stuff the the exciting part about imitation and policy gradients so we talked a little bit about imitation learning before um we had this example where we had human drivers collect some images and the actions that they were performing and uh we would put them in in the big training data set and then supervised learning on it doing imitation learning so now the question is how do we do this with multiple tasks right so first question as well how do we optimize multi multitasking mutation learning setting and the answer is really simple and that's this is the reminder from lecture two the Chelsea that Chelsea gave we just use the vanilla multitask objective where we minimize the loss across different tasks right so we can have some kind of way of specifying which task it does and then we can compute the loss for each one of the tasks and take the sum of those and minimize that so it's exactly the same as supervised learning so we can use same architecture stratified sampling everything that you've learned about soft soft or hard weight sharing and so on very simple right so the other question is how do you specify a task right so let's say you're trying to teach a robot something you need to specify multiple tasks using imitation learning you collect demonstrations for each one of them how would you specify the task any ideas how would you communicate to a robot what task you wanted to do levels that's right but you have to specify the task to robot somehow so what would you use for that how would you describe the task yes um as a as a by classifying a reward function right so different reward function the different tests would have different reward functions but still you would have a neural network that would need to be conditioned on a task right and this conditioning needs to be specified somehow right like you need to tell the robot or you need to explain what kind of task you want it to do it would have different rewards for all of these for all of these tasks but how would the robot know which tasks to perform yes yeah you can use a one hot Vector right so it would be just you know one heart telling you which task you're in yeah right yeah you can use demos so you would perform a demonstration and that would be some kind of embedding of this demonstration will be the description of a task you can specify it by natural language you can specify it by video by a goal image all kinds of different things and these different things have different trade-offs so there's this one work that was published at Coral by Eric Jenga doll I called BCC and the way it works is the following so they collected a big data set of of demonstrations which resulted in this diverse multitask data set so they're doing many different tasks and then they were actually conditioning the policy on two different things on the video it's a human video of performing a task and a natural language description as well and then they would based on this they would test how well does it generalize to a task that it's never seen before right so they'll generalize some tasks that have that haven't been seen which is kind of a novel thing to do their architecture looks like this this is the state so they take the the image from the camera then they random crop it down sample it so they have the smaller image which goes through a pre-trained resonant Network and then does a few operations and eventually ends up in outputs the action that the robot should do and then they have this way of encoding the task and they have two different ways here so they can either take the human video so they record human videos that are paired to the to the tasks that the robot is trying to do they pass it for a video encoder and this ends up being and this ends up in some kind of embedding vector in some latent space and they also have a language natural language description of the task which goes through a language encoder in this case I believe this was a pre-trained language model which ends up in some space in this in this embedding space and then they use film conditioning to condition this resnet on the task that I want to do in terms of the last two years so they take the they do the minimum negative log likelihood so they're minimizing this overall the tasks that we just discussed and they have the behavioral cloning loss where z i here depicts the embedding of the task you're in and then in addition to this for this to work well they added one more loss which is the language regression loss so they're trying to minimize the distance between the video encoder and the language encoder for a specific for a specific task and this makes the video encoder a little bit better all right so in terms of the results here are some of the tasks that head out held out tasks that they that they've been evaluating it on and you can see that you know some of them kind of worked some of them work fairly well like 82 or so some of them don't work at all like some of the language descriptions that you see here like Place banana in a ceramic cup and uh overall the held out task success is just 32 percent but keep in mind that this is the first time where we're actually trying to evaluate something and tests that you've never seen before so they trained on a completely different set of tasks so this here we are testing generalization to new tasks so there is non-zero success for 20 of the 28 tasks and the average was the average success of these held up tax was 32 and here are some examples of what this actually does so here is a test that it's never seen before for instance push purple ball across the table and you can see that it that it does that we hear another test that it's never been seen before place a bottle in a tray right so in this case they use natural language as the task specification and this allows them to do some zero shot um generalization where they can specify a different text command and the robot does that all right cool so now we also talk about reinforcement learning and we talk a little bit about what is a task in reinforcement learning and we talked about this definition the the mdp where we have the state space the action space the initial State distribution the Dynamics and the reward so an alternative view that uh was already said in the class was to add the task identifier as part of the state so we can just say that our original state was this S Bar and then we'll add our task description zi you know either natural language or something else and the the stuple would be our new state so now our new state also encompasses what task you're trying to do and if we take that view then we can just say that this multitasker reinforcement learning problem is just another mdp right so in that case the mdp would be the the state space is equal to the union of all the state spaces of all the tasks the action space is the union of all the action spaces the initial State distribution is the mixture and then it has some Dynamics and the reward function so it can be cast as a standard MDB all right we also talked about the goal of reinforcement learning in the previous lecture this was the single task case the multitask case is very similar so it's the same as before except now we have this task identifier that tells us which task we're in an example of this would be one hot task ID or a language description or a desired goal image now if we use the or the gold state if we use the the desired gold state then our zi our description of a task would be equal to some State as G right so it will be in the same the same dimensionality in the same space as the states themselves and this is actually a really important case that we'll spend a little bit more time on and this is what we'll be referring to as single condition reinforcement learning all right so we're conditioning on the goal that we actually want to see in the world if we want to achieve so what is the reward in this case it could be the same as before or for gold condition RL the reward could be just the distance or the negative distance between our current state and the state that we want to get to right so it will be just the negative distance between the two some examples of that would be the euclidean distance or a sparse reward so if the two states are exactly the same you get reward otherwise you don't or something else so this is the gold conditioner rail case the multitask oral case fairly simple all right um yeah so in terms of the conditioning we can you know we already discussed two different ways of of specifying the task to uh to an agent and it turns out that it actually matters how you specified and it's not that it's not just that it matters because you know it's maybe more convenient to talk to a robot than specifying a one hot Vector to a robot but it actually also helps with learning so to discuss this we'll look at this Benchmark called meta World which consists of 50 different tasks 50 different manipulation tasks that have the same robot the same state space and action space but they do slightly different things right simple manipulation tests that you see here and it turns out that when we evaluate whole bunch of different multi-task versions of popular um of popular multitasker enforcement learning algorithms they don't work that well so even though you can solve each one of those tasks separately quite well if you put them all together and you try to learn them all together it doesn't work that well so the success rate is at like 50 or so so there's this idea introduced by sudhani at all called care where instead of specifying tasks as one hot vectors which is what was done here before they would introduce so-called metadata metadata right here so the metadata would be a short language description telling you what the task actually is so for instance it would say you know insert the peg inside the hole or something like this now they would use that metadata with a with they would pass it through a pre-trained language model and then have another feed forward neural net to get some kind of context right and they'll take this context vector and then separately they would have the the state that has to pass through K different encoders that are being trained and then they would look at the output of all of these encoders and then do attention overdose according to the context that they got from the metadata from the pre-trained language model so kind of the language model tells you what part of the context or which which encoders you should be paying attention to then they pass it through an MLP so a feed for neural net and then they combine these two and then this is the representation of the task and then they run it through the policy algorithm so it's kind of it's a different way of conditioning right so they're still running the same multitask reinforcement learning but now the task is specified by the metadata and there is some um there's some kind of trickery as to how this is being processed but the important part is that there is some language description that tells you the connection between different tasks and it turns out that it works a little bit better and they actually showed a few a few different ablations in their paper showing how this works and why that is but one particular one that I found interesting is right here s