Stanford CS330: Deep Multi-task & Meta Learning I 2021 I Lecture 11

Key Takeaways

hi everyone i hope everyone had a great weekend and enjoyed the rain yesterday um a couple reminders homework 3 is due on wednesday the optional homework 4 will come out on wednesday also for people who are remote and on zoom so far i think everyone has been asking questions through chat but i wanted to kind of have a quick reminder that if you want to instead of asking a question via chat you could also raise your hand on zoom and when i call on you we should be able to hear you verbally if you want to ask verbal questions cool so today we're going to be talking about meta reinforcement learning and we're going to be talking about the problem statement for meta reinforcement learning and then we're going to be talking about two classes of methods black box meta rl methods and optimization based meta-rl methods black box meta rl methods are going to come up in homework four and then on wednesday we're also going to talk about meta rl but we're going to talk about a specific part of the meta rl problem statement which is learning how to explore um this will really be the focus of homework four but these methods will use kind of some of the basics in black box meta rl and then the goals of the lecture will be to really try to understand the problem statement of meta-rl how it differs from kind of standard metal learning as well as understand the basics of black box and optimization based meta rl methods cool so as a little bit of recap previously we talked about reinforcement learning algorithms as a as kind of a primer on a lot of what we'll talk about today we also talked about multitask girl and we introduced both model 3 methods as well as model-based methods and we kind of saw this kind of overall anatomy of a reinforcement learning algorithm where you collect some data you estimate return in some way and then you use that to improve your behavior and of course one of the things that's pretty different about i guess there's two things that are different about rl compared to supervised learning one is that you collect your own data and the second is that instead of giving getting direct labels you're given reward feedback and this changes how the learning algorithm operates and so we saw a few different ways for estimating the return directly through monte carlo or fitting a value function to estimate reward as well as just estimating the dynamics of the environment and then we also saw ways to use these estimates of the return to improve your behavior through policy gradient through kind of taking the arc max of your value function and also through planning with your model great and then before we talked about reinforcement learning we also talked about these different problem settings including multitask learning transfer learning and meta learning and all these different scenarios the tasks have to share some amount of structure and what we'll really be looking at today is how we can look at the meta learning problem in a reinforcement learning scenario where our goal is to quickly learn a new reinforcement learning task given experience from previous reinforcement learning tasks and one thing that's going to be different is that instead of just given data from these tasks we're actually going to have to collect the data from each of the training tasks ourselves okay and then we saw that a reinforcement learning task corresponds to a markov decision process with a state space an action space essentially the domain of the problem the initial state distribution the dynamics of how the environment evolves over time as well as the reward function and we saw that generally any of these things can vary across tasks although in general the dynamics and the reward are often the things that are changing uh across tasks and so meta reinforcement learning is essentially going to correspond to meta learning but with reinforcement learning tasks and where essentially we need to collect data from our training tasks in a way that allows us to solve a new task through exploration and adaptation in that new mdp cool so concretely what this looks like is before we talked about the metal learning problem we saw supervised learning and then we saw how metal learning is essentially trying to kind of take as input a small amount of data for a new supervised learning problem and a new example and leverage the data in order to effectively predict the label for the test example so this was meta-supervised learning we did this with a data set of data sets across all of our tasks where each data set has input output pairs and this view is useful because it essentially reduces the problem of metal learning to learning something that takes this input a data set and makes predictions about new examples now in meta reinforcement learning we're no longer going to be trying to solve supervised learning tasks uh instead we want to learn a policy so our input is going to be estate st our policy is going to output which action should be taken at that state and so generally we'll be learning this policy that maps from states to actions um and in reinforcement learning instead of having input output pairs you're instead given or you still have experience in that mdp that has states actions rewards and next dates and so we're not told directly what to do we're just given rewards for behaviors that are good and we try to maximize the rewards for those behaviors so this should mostly be review up until now and in meta reinforcement learning we want to be able to quickly solve a reinforcement learning task and so it's going to be quite analogous to the meta supervised learning scenario but um our training data is not going to be input output pairs it's now going to be experience collected in a reinforcement learning task and you can think of it as k episodes from running your some policy pi um or k rollouts from pi and we want to be able to use k rollouts from some policy in order to maximize reward in that mdp and k might be very small typically in reinforcement learning you may require a lot of experience we want to be able to solve a new task with a very small amount of experience and so given this experience we then want to be able to map from states to actions and be able to quickly learn how to solve a new mdp from just these k rollouts now the data for this um we'll again have a data set of data sets that's collected for each task but one thing that will be different is that we won't necessarily just be given these data sets this is the data from each of these tasks actually has to be collected ourselves through the meta reinforcement learning process okay so now the key parts of this problem is that we have to figure out how to design and optimize f uh this we saw this uh before and actually we'll talk a lot about this today and how this differs a little bit in the reinforcement learning setting and then we also need to learn how to collect appropriate data so we need to collect all of the dis we also need to learn what policy pi will help us collect informative data for solving the mdp and so essentially we need to learn how to explore and this is usually pretty important because if you just run a random policy for example that might not give you informative information about the task and if you have a good exploration policy then this should allow you to very quickly infer the task so you both need a good exploration strategy here as well as a good adaptation strategy in f data cool and the other thing that i'll note here is that your exploration policy pi doesn't necessarily need to be the same as the policy that actually adapts to the task which is represented by f so we can look at an example so say we want to learn how to quickly navigate in a new maze so we collected some experience in a maze so maybe we just had one episode so k equals one and we collected some experience that's shown in this directory that goes from red eventually to blue and the goal is to get to the blue box and you don't necessarily see the agent only sees where it kind of locally is in the maze it doesn't actually see the whole picture of the maze and so given this training data the single episode in the mdp we want to be able to ultimately get a policy that can then directly go to the goal and directly solve the mdp very quickly so this is what we'd like to be able to do so this would correspond to meta test time and then what we want to do during metatrading is to prepare ourselves for solving this meta test task and so if this is what we want to be able to do then what we can do is we can at mediterranean time practice solving lots of different kinds of mazes so we can collect data in different tasks where different mazes correspond to different tasks and learn how to solve each of these mazes so that when we're given a new maze at test time we can very quickly learn how to navigate in that maze with maybe just a single episode um and of course uh here the example of different tasks corresponds to different mazes but the different tasks could be something different as well yeah so the question is can you have multiple exploration policies when you're collecting data um when you collect your training data in in an mdp you um it could really be whatever you want i i guess i listed a single policy that could also be a mixture of multiple different policies such that you collect data according to that mixture of policies um the other thing is that the exploration policy can also be recurrent and so it can incorporate its experience in the maze so far when determining how to continue to explore in that maze okay and then one more note about the meta rl problem is that um this is the formulation that i discussed before i'll refer to this as like the episodic variant of meta rl where you're given k episodes from pi but the training data could also be corresponding to um k time steps from your policy you could essentially learn in a more kind of online fashion when you're given even less data from pi rather than k episodes you have k time steps and like i mentioned before the exploration policy and adaptation policy don't necessarily need to be the same cool any questions on the problem statement so essentially what we did is we just took the metal learning problem and replaced the tasks with mdps so how do we actually go about solving this problem statement we're going to look at two different kinds of approaches block box approach is an optimization based approaches we're actually not going to talk at all about non-parametric methods i'm curious if anyone has thoughts on why that's the case or why it would be difficult to apply non-parametric my learning methods to meta rl [Music] yeah exactly so in with non-parametric methods they're really designed for classification problems where you are trying to classify among a few different classes in general when you're trying to predict actions sometimes those actions might be continuous in which case it's more of a regression problem and you just can't solve it at all with with non-parametric methods um and even in cases where you do have a discrete action space it isn't really analogous to this concept of class it's more analogous to like which of these actions should you take it seems possible that you could apply non-parametric methods to meta-rl when you have discrete actions but that's why they haven't been used at all because you don't really have this notion of class and it's not really a nearest neighbor's problem anymore it's more a problem of using your experience to figure out how to solve new environments and you can't really apply this sort of idea of matching or nearest neighbors to solve these kinds of problems okay so we'll start with black box methods and um in black box meta learning before we saw that we can essentially take this form where we have a function that takes us and put our data as well as our new test input and predicts the corresponding label and in general the trend will almost be the same in rl where we take as input a data set as well as a new state that we're observing and we're going to predict the corresponding action for that state um so for example our black box model could be a recurrent neural network that takes as input some of our training data seen so far and as well as our query data point st and then predicts the corresponding action now one thing that you can note here is that the training data is actually going to be getting larger over time it's essentially going like at every single time step we're just adding to our training data and so it's going to be incrementally growing and essentially growing on a rolling basis where kind of your query set at the current time step will then actually become part of the training data set at the next time step and yeah so you can view it as essentially a recurrent policy it takes as input the states and the rewards so that it can infer what the task is from that experience in order to figure out what to do at the current state so now i have a couple questions for you one thing you might notice here is that we're passing in the states and the rewards but we're not passing in the actions um does anyone has any thoughts on why we don't need to pass in the actions in in with the support type um so you're saying that the um did you contribute any additional information um you're saying that given the state that you transitioned into um the past actions don't matter because of the markov property essentially you you know what you're you see the state that you're in um and that does like that tells you essentially about your previous action that's sort of right um anymore dad yeah so the other thing is that the policy is actually predicting the actions itself and so its hidden state is actually going to contain information about the previous action and so because the hidden state already contains all the information that's needed to predict that action then the action information is actually already stored and you don't need to pass it in as input again and one reason why that information is important is that even just despite the markov property it is helpful to have those previous actions because if different tasks vary based on different dynamics then you actually want to know information about sas prime about what the dynamics are and so it is actually important to have that to know that action information because it allows you to identify differences in the dynamics but the hidden state already captures it because the hidden state is what was used to generate that action okay and then the other question is how is this different from just doing like standard reinforcement learning with the recurrent policy so it is quite similar to just doing standard rl with the recurrent policy but there are a couple differences and i'm curious if anyone can spot any of them yeah so one thing is typically in standard rl we don't pass in the previous reward function at each time step we just pass in the state and so this is one thing that's different good any other things that are different so the i guess the first thing that i had here is that the reward is passed as input we're also training this rnn across multiple mdps rather than within a single mdp and therefore the rewards might actually this is one reason why you might need to pass in the rewards as input also i can note that if the rewards are the same across tasks and the only thing that's different is the dynamics then you don't actually need to pass in the reward any other differences so one other thing that's pretty important is that if you want to learn across episodes if you want the syst this thing to kind of take as input incorporate a training data set that isn't just like a single episode but it's actually multiple episodes then it's really important to maintain your hidden state across different episodes in your task and so one of the things that's really different about kind of doing rl with the standard recurrent policy is that typically you're just going to be maintaining your hidden state within an episode whereas here you're actually going to be not resetting your hidden state at the beginning of each episode you're only going to be resetting at the beginning of each task and it's going to try to essentially aggregate information about the task across these different episodes so that it can essentially gain as much information and knowledge about the current task as possible yeah so this particular plot doesn't necessarily imply it unless the dot dot dot kind of includes um includes additional episodes i have another diagram that that illustrates this more explicitly so um here say your episodes are 10 or t time steps long then you are going to have an arrow that continues that hidden state between the boundary of t to zero um and then once you get to the end of the last episode you will then see this will have two episodes if you only have two episodes then you'll kind of cut off at that point and then start the new episode for the new task um at time step zero yeah yeah so the question is what's the point of having different episodes now given that we're now kind of treating different tasks different episodes and one of the things that's important here is that s1 is going to be kind of drawn again from the initial state distribution and so uh when you're exploring in your mdp it can be helpful to kind of sample multiple times from the initial state distribution and so like sample multiple possible trajectories and so we're going to essentially have this notion of episodes and maybe even a notion of kind of meta episode or trial that captures all of the episodes within a single task and sometimes the literature refers to that as trial i've also seen it sometimes referred to as meta episode because it captures multiple episodes within a task yeah so the question is isn't this very difficult to optimize because like essentially we need to like remember a lot and we need to remember like all the way back from the first episode and if you have a lot of episodes um then you essentially like learning things from the earlier episodes might be very difficult there's a great question uh we'll talk about this in a slide or two but there are a couple architectures that aren't recurrent networks that will that can potentially aggregate information in a way that isn't like this sequential and those kinds of architectures can help the optimization a lot um you can think of for example things like transformers um or things like things that kind of take the episode information and then average it into a single embedding that's going to make it easier to back propagate into the first episode for example there are pros and cons to teach approaches but some of those architectures make it easier to optimize cool so as an algorithm what this looks like is we first sample a task um from our kind of the set of tasks that we have then we're going to roll out this recurrent policy for up to n episodes where n is basically the kind of the horizon of the learning problem um and as i mentioned before this d train is going to kind of be updated incrementally with each time step it's not actually it's no longer fixed it's actually growing online then this will be under the dynamics and the reward for task i and then we'll store the data that we collected in some replay buffer for that task and then we can update the policy by sampling data from that replay buffer to essentially just max like optimize this recurrent policy so that it maximizes reward over all of your tasks um and this this last step can be done with a variety of different learning algorithms it could be done with policy gradients it could be done with um with uh like q learning based methods for example as well um so i was trying to keep this general in the sense that it can be applied with a few different approaches and then of course you also repeat this process once you update your policy you'll then collect data for another task and so on and so forth yeah okay is it important for the reward functions between tasks to be sort of in the same magnitude for the values yeah so the question is um does is important for the word function to be in the same value for different tasks in general this is quite important because if they are on very different magnitudes then when you do this last step it's going to care a lot more about the task that has a higher magnitude reward than about tasks that have lower magnitude rewards and so if you want it to ultimately be able to adapt to all of the tasks in the task distribution you generally want them to have similar magnitudes cool okay so that's what happens at metatrading time and then once you learn this kind of recurrent policy that can learn how to adapt to a task then at meta test time you're given a new task and you simply just take the policy that you learned and roll out that policy for kind of the horizon of the learning problem generally you don't want to roll it out for more than n episodes because if you roll out an rnn for example for longer than the horizon that it was trained that it may not perform well um although one thing that you could do is something something where you try to freeze the behavior um that you get at episodes and no longer update that policy and just keep on writing that policy but if you do really care about like learning for end episodes and then continuing to kind of execute that policy you can also optimize for that behavior during metatrading okay um and then to touch a little bit on architectures and optimizers for this um the approach of using black box matter url is quite general but you can there's of course lots of different design choices you can choose so one of the first approaches for this is referred to as rl squared because they're trying to do fast reinforcement learning via slow reinforcement learning and they essentially did exactly what we talked about on the previous slide where they had this recurrent neural network that takes as input states and and rewards and is able to predict the the next action and in these two papers they used policy gradient methods trpo and a3c um instead of using your current neural network you can also use an attention-based architecture that has kind of connections from the current time step all the way back to that first time step and these methods are going to be better at kind of propagating across very long sequences this particular architecture was alternating between attention and 1d convolutions but you could also have like a purely attention-based architecture as well and then they were again optimizing the policy with trpo which is a policy gradient method and then the last architecture that i'll mention is something more analogous to an architecture that we saw before where we essentially encode all of our transitions into a into some embedding and then we average that in vetting and then pass that average embedding into our q function and into our policy um and so this was kind of more of like a feed forward architecture than average um and then the actual optimizer that was used with soft actor critic which is a value-based reinforcement learning method and you can note that there's a q function and a policy here and that's kind of indicative of the fact that it's a value-based method yeah so the question is how do you choose the the reinforcement learning optimization um like there's a reason why multiple of these use trpo um the i guess uh we'll talk about this in a second too but the i guess things like policy gradients are quite simple and um like quite simple to implement and are also pretty easy to get to work um if you give it enough data uh and so i think that these were like two of the first meta rl black box meta url methods to come out and so usually going with simple things that are kind of easier to get to work that don't require quite as much tuning is helpful for that um that said methods that are on policy tend to be very data inefficient and so these methods are also extremely data-hungry during the meta training process um and so that's uh that's a reason not to use like policy gradient-based methods and a reason to favor value-based methods yeah so in in practice you will sample a mini batch of tasks on task one so the question is is the reason why we're just sampling one task here um in practice you will actually sample a mini batch rather than just one task and then you'll do this update on that mini batch of tasks so you won't roll it out for so if you have um k tasks and n episodes in your batch then you're not going to roll it out for k times n because you need to break the rnn state at the boundary of each task and so you'll roll it you can essentially roll it out in parallel for each of the k tasks um and so you will eventually get uh the data that you'll get will be k times n um episodes but you do want to make sure that you cut the hidden state at the at the task boundary so that it's only aggregating information about that task and not about the previous tasks um yeah so yeah and in general um i wrote it as one task for kind of to make the notation simple but in practice you will want to use a mini batch of tasks um also if this sometimes this policy update it can be with that mini batch it can also just sample directly from the replay buffer and so it doesn't necessarily need to include the data that you just collected okay so i guess one other note about architectures is that um this is actually just doing feed forward and averaging over individual transitions you can also um do an rnn over transitions within an episode and then average across episodes so you could sort of potentially like mix and match different components here okay so let's look at a couple examples of what these algorithms do so the first example will actually be just the maze navigation example that i talked about before where we want to be able to learn how to navigate a maze and we're going to train on a thousand different small mazes and then we're going to evaluate the algorithm's ability to quickly learn how to solve new mazes both small mazes and large mazes cool and then after the kind of meta training process we'll look at some videos of what it can do at meta test time and so here's a video this is showing the first person point of view this is showing kind of a top-down view of what the agent is doing this is the kind of the first episode of the um the first episode at meta test time where it's essentially exploring the maze and looking at different different parts of it and then this is the second episode where it just goes straight to the goal um and so we can probably just try to play that again so here's the first episode where it's exploring this is essentially to try to understand the mdp or the maze that it's in and then it's incrementally kind of updating the hidden state of the rnn based on the training data that it's seen so far and then at the end of this episode it's going to keep its hidden state the same and then use a policy that can adapt to the exploration episode that it collected in order to go straight to the goal the system can also work on larger mazes as well so here's a here's an example of a large maze where it is currently exploring in the maze in the first episode i'm still exploring and then it sees the goal here and then after that episode it can very quickly go directly to the goal so both of these are held out mazes that it hadn't seen during training and in some ways you can think of the first episode as the support set and the second episode as the query set but in this case it is actually also just like trying to run things online as well um so yeah um and then in terms of quantitative results uh the this is comparing um using an lstm versus using the kind of the mixture of attention and conflares it's also comparing to just a random policy and what we see is um i think this is evaluating the um the number of time steps that are needed to reach the goal in each episode and so the first thing that we see is that um the random isn't doing very well um we also see that generally it takes longer to find the goal in the large maze than the small maze and then generally we see that it's able to more quickly find the goal in episode two than in episode one because it's leveraging kind of the exploration that it did in episode one in order to find the goal in episode two more quickly any questions on this and one thing i didn't show here is the metatrading process itself those videos were just from meta test time and during metatrading time it was essentially doing that same sort of kind of two episode process but across all of these different mazes in its mediterranean set and so it's kind of learning how to explore and solve these mazes so that when it's given a new maze it can quickly solve that maze [Music] if anything yeah so what is the learners sharing across um mazes so one thing that you can see is that because it's doing better in episode two than in episode one it is clearly storing information in its hidden state that tells it about that maze as it's going to basically be encoding something about the geometry of the maze um i don't think that they did any sort of visualization in the paper about what um whether it's storing the full geometry or just storing like the solution path for example to the goal um but that something like that could be interesting to visualize did you ever watch yeah distribution but in this case i feel like the large mazes have come out from different distributions with a small meeting so by the same assumption on your business yeah so a great question the question is um we assumed before that the kind of mediterranean tasks and the meta test tests are the same um it seems like now when we're testing on large mazes there's actually this mismatch um so in general we generally still need to make the same assumption that training and testing tasks are from the same distribution um qualitatively they found that the larger mazes were similar enough that it was still able to generalize i expect that if you gave it like a maze that was 10 times as large it probably wouldn't work one reason why it might be generalizing well is that it is still a first person observation and so the data that it's seeing is still going to be very similar between the training the test tasks the like for example the quarters are the same size the visuals are generally the same um and so it is nice that it's able to generalize it's also possible that in cases where there's a longer path towards the goal maybe those are the scenarios where it's not able to do as well whereas if it's in a larger maze but there's still a small path to the goal maybe it's still able to do well because it saw a smaller path to the goal during training yeah um yeah so the question is um what kind of what is the support set in the query set and like what policy are we using for to getting the support set and what policy are we using to get the query set so um you could essentially like you can think of it as support set being episode one and queries that being episode two um in reality it is more like a rolling online basis where the support set is all the time steps you've seen so far and the query set is the current time step um and so you're actually going to be trading on support sets that have lots of different sizes um and then in terms of what policy that you're using um and this work is actually using the same policy for collecting the first episode and the second episode which is just this recurrent neural network that takes his input some hidden state that encodes the support set scene so far um and that said in practice they could be different policies and and we'll especially see in the next lecture on wednesday there are scenarios where you want them to be different policies so as a kind of an example of that say in the mazes say they're on one of the walls there's actually a map that like shows the layout of the maze in that setting in the first episode you want to just go look at the map you don't want to like go figure it out um and so you want a policy that looks at the map for exploration and then uses that information from the map in order to solve the task and so in scenarios like that you often do want different like pretty different policies yeah i have been feeling like these black boxes especially but of course imagine it's way more complicated [Music] it seems like this is going to work decently well with on policy methods but it's going to be a lot more complicated for off policy methods like like dqn or like you learning based methods that what you're asking yeah so there have been successful instances of people training q learning algorithms with recurrent neural networks that have kind of an online update um that said generally recurrent policies are pretty difficult to optimize with q-learning and there are just a couple tricks that that people use there's this paper called r2d2 that shows that if you essentially store the hidden states in the buffer rather than recomputing the hidden states that actually works a lot better um that said it does get more complicated with value based methods and um i can actually show one example of value-based methods maybe before getting to the other question so in particular we want to be able to use value-based methods because they're a lot more efficient this paper is going to look not at navigation problems but at continuous control problems where you might want the agent to run in different directions or at different velocities or be able to adapt to different physical dynamics um and so in general reinforcement learning algorithms are very efficient at meta test time they can solve the task in like two episodes but what's challenging is the meta training process might be very inefficient with things like policy gradients and so what we'd love to do is to use value-based methods or off-policy methods during meta training in order to optimize for the black box approach um and so i guess maybe i gave this one away but you would expect off policy meta rl to be much more efficient than on policy meta rl because these meta-rl methods are going to inherit the sample complexity of the algorithm they're using to optimize the meta-learning process um and so essentially what this what this worked is it used the kind of feed forward and average architecture in order to compute a context just like i showed on the previous slide and it fed that context into both the policy and into the critic and it found that if you did something like that you are able to actually solve these metal rl tasks much more efficiently than all policy methods and so what these plots are showing is the x-axis is the amount of meta training time steps and the y-axis is how well you're able to adapt and the blue curve is is this algorithm that uses uh that builds upon a value based on all methods soft factor critic and all the other methods are using policy gradient um and so you see that there's kind of this pretty huge difference in sample efficiency the on policy methods their asymptotic performance is shown with the dashed line and so we see that asymptotically they do actually do pretty well but the time it takes for them to learn is much much slower than value-based oral methods does that answer your question cool did you have a question in the box no cool um and then one uh short digression is that we've talked about black box methods and these actually look somewhat similar to multi-task policies um so for example in a multi-task learning policy we'll be conditioning on a task identifier so for example maybe we have a robot we want to stack in different locations z might be the location where we should stack or maybe we want to walk in different directions and zi would encode the direction that we want to walk um when we have these these black box metal learning methods you can essentially view the data or the experience as the task identifier that you're passing into the model um and so it's sort of like the same thing as multitaskrl except where experience is identifying helping you identify the task rather than an explicit identifier and then of course you have to figure out how to explore to collect information like that and then the the policy the multi-task policy is kind of the last part of that network so essentially like a multitask policy with your experience as a task identifier we've also previously talked a lot about goal condition policies and value functions one thing that's nice about meta rl is that rewards are essentially like kind of a strict generalization of goals goals can specify reaching certain states whereas rewards can specify all sorts of behaviors including reaching certain states and so really the meta-r objective is allowing you to adapt to new tasks from rewards for that task whereas goal-conditioned rl's allowing you to generalize to retain new goals you can also think of this as k-shot adaptation versus zero shot generalization yeah um [Music] the question is in the maze example um what was the reward function um was it only like reaching a goal or does it account for the time it takes to reach the goal um i don't know exactly what they used there are a few different choices one could just be like one for reaching the goal in zero not for reaching the goal and if you have a discount factor that's less than one then it's going to encourage you to try to reach the goal faster another thing that might be common to do is to use a positive reward for reaching the goal and a small negative reward for every time step in which you haven't reached the goal and that will also encourage it to get there faster the last thing that you could do is you could give it a shaped reward that indicates the distance to the goal that is something that you could also use although that's going to give it a lot more information about where the goal is and whether it's getting closer or further to the goal it's more like a game of hot cold than um than actually trying to navigate and find where a certain object is um on that note the other thing that i could mention is that the rewards that you use in your support set that you feed into the network those could actually be a different reward function than the one you use to optimize and so one thing that you could do that's kind of cool is give it a very sparse reward that it has to learn from in the support set but then train your medi train to have your whole mediterranean process with more of a shaped reward and then you can essentially learn how to um explore and solve sparks forward tasks using more dense rewards during metatrading okay so to summarize black box meta rl these methods are quite general and quite expressive there's of course a variety of design choices in the architecture um and they're generally pretty easy to combine with different rl optimizers essentially you'll kind of optimize yeah optimize the behavior either with respect to some bellman error objective like eq learning or with respect to policy gradients um they do tend to be somewhat difficult to optimize like we've talked about especially with recurrent networks you have to back propagate through this long string of time steps um also in general you are like in black box metal learning you're learning how to learn from scratch you're not building any structure into the meta learner and so this also makes it difficult to optimize and then like we saw before it's in terms of the data efficiency it's going to inherit its data efficiency from the outer rl optimizer that's used and so if you use policy gradients it's going to be very data inefficient and if you use value-based rl it's going to be probably a lot more efficient okay any questions on blackbox meta are all before we move on to optimization based methods cool um so uh like we just talked about black box metal learning methods can be difficult to optimize and so one thing that might make it easier is to build in the structure of optimization into the metal learner um and so if we want to turn this kind of approach into a meta-rl algorithm we have a few different choices for doing this and in particular one of the choices we have is what sort of optimization should we use in the inner loop of this optimization um as a kind of a few possible choices we could use policy gradients um we could use q learning or we could also use something like model based rl i'm curious if anyone has thoughts on kind of what some of the trade-offs might be for using these different approaches in the inner loop irrespective of what you do in the outer loop yeah i guess from what we discussed earlier since policy is type of approach yeah so policy gradients can be very uh sample and efficient and so that might be a reason not to use them in the inner loop [Music] yeah so if you if your state space is very large and you're trying to like solve the maze navigation example um for example then your model may actually be very inaccurate if your support set doesn't cover the entire relevant part of the state space yeah so if the tasks all have the same dynamics but have different rewards then you can actually just use a single model and you don't actually need to adapt the model um and so using mammal with model based style probably makes the most sense when the dynamics are changing across tasks any other points cool so um those are all great points um i'll overview some of the kind of trade-offs and why we might use one of these versus the other in mammal so with policy gradients one thing that's nice is that they are very clearly gradient based they very clearly give you a gradient and so that makes it nice to apply a gradient in the inner loop um they are on policy and so this means that they are going to be inefficient um the other thing that's maybe a little bit more subtle about policy gradients is that they don't actually carry that much information especially when you have sparse rewards so say for example um you're maybe in the maze navigation example or you're just trying to kind of learn something about the dynamics about the structure of the dynamics in the environment then if you look at the policy gradient if you remember it looks something like the gradient of log pi summed over your time steps it looks something like this times the sum of rewards for those time steps and if you're trying to learn something about the dynamics and you end up maybe maybe you're in the maze navigation example and you collect a trajectory and you didn't get to the goal so you got a reward of zero then that means that this is going to be zero and then your entire gradient is going to be zero and that's really unfortunate because that means you're not going to get any information about the dynamics of the environment if you didn't get any reward and so this is one reason why policy gradients can be pretty difficult with with an algorithm like mammal um it's because that you want that integrated to be very informative about the task and whenever you're in a sparse reward setting and you don't get any rewards it's not going to tell you any information about the task and about the environment so it can work well with shaped words but if you have sparse rewards it can be rather problematic yeah yeah so if you um you can reward shaping will certainly help and that will give you a lot more information if you give it like a negative reward for everything um uh like maybe this is kind of a fixed concept of like negative t because maybe you give it a reward of negative one at every time step this still isn't going to be that informative because um it's uh for um four different episodes like it's not going to tell you essentially which episodes are better than the other um if you're just getting a reward of negative t for all of your episodes cool um so those are kind of the pros and cons of policy gradients um now for key learning um learning actually isn't gradient based it's key learning is more like a lot like dynamic programming and if you look at the update for q learning it is [Music] it's something like uh kind of q hat of s a you're trying to make this match i guess if you look at the loss function at something like um the reward plus gamma times max of a prime of q of s prime a prime something like that and one thing that you might notice is if you take a single gradient step on this objective then this is only going to propagate kind of one time step of information because this is um the state and action at the next time step this is the statement actually at the current time step and so a single gradient update isn't going to give you a lot of global information about the task it's only going to give you very local information at a particular time step and as a result this means that to get a good kind of update for your like inner loop you're going to need a lot of steps to propagate information from the future to the current step and as a result it could actually be very difficult to use q learning in the inner loop unless you take a very large number of steps but then you have to also back propagate through that very large number of steps in the inner loop it is going to be off policy so it's going to be more data efficient um that said there's been very few successful examples of optimization-based q-learning meta-learning methods okay and then one last example is for model based rl uh learning a model if you want to kind of adapt to different dynamics model based rl is just a supervised learning problem on the model and so updates to your model are going to be gradient based it's also you can do this in an off policy way and so it can also be data efficient this makes the most sense when you have a more local update and when you have varying dynamics across tasks but it does address some of the shortcomings that we see in policy gradients in q-learning okay so at a high level essentially optimization based meta rl can look like embedding and optimization where you choose different objectives for the inner loop and different objectives for the outer loop next i'll just go through two instantiations of optimization based metaurl so we can look at some kind of concrete examples of actually implementing these algorithms um so the first thing that we'll look at is mammal with policy gradients like we mentioned it can be a terrible choice if you have sparse rewards although it can still be reasonable if you have shaped rewards and plenty of data so here's kind of to help us uh kind of remember the objectives here's the mammal objective here is the policy gradient objective which is hopefully the same as what i wrote here i guess there's also kind of an expectation with respect to pi theta and so what the meta training process looks like is we will sample a task we'll then collect data by rolling out our policy pi theta this is the policy according to our set of initial parameters then for the inner loop adaptation we're going to compute a policy gradient using those rollouts by computing this objective or sorry this is this is just the gradient so by computing the gradient then we're going to collect data by rolling out our adapted policy and this will be used so that we can compute the um the policy gradient with respect to that of that adapted policy um and so each of these cases we're going to be using a policy grading both for the inner loop and a policy gradient for the outer loop objective is yeah the question is like for this step two is there any benefit from using your current policy versus um a random policy um or some other policy so the um i guess the reason why we might want to use pi theta here is that for this to be like the correct policy gradient your trajectory should be drawn from pi theta and if it's drawn from another policy then this gradient isn't going to be accurate or as accurate it's certainly not going to be consistent it's not going to be something that you will use to like ultimately improve your policy and so if you do use pi theta here then you are kind of more guaranteed that this gradient is going to be pointing in the right direction um there are like slightly more off policy policy gradient methods like ppo um and in those cases you can potentially use older policies to collect this um but in general it's going to be cleaner to use your current policy one downside that you might notice here is we are doing these two steps of collection and um and this is going to get pretty um like sample expensive in terms of sample efficiency and um it's one i guess one thing that i'll also notice is we we you talked about like policy gradient being sample and efficient in the inner loop because we have to also roll out pi theta and every iteration this means it's also sample and efficient at the outer loop as well like at the level of the metal learning process i should also mention that there is uh this is of course an iterative process so do you still pass previous reward into the policy um in this case no the policy is is just a policy that maps from states to actions and the way that we're going to infer the task is through the parameters of that policy rather than passing the rewards as input of course you could also imagine hybrid methods that are kind of both recurrent and do this sort of gradient-based adaptation um although it kind of in the kind of standard optimization based algorithm we're just updating the parameters rather than passing in the rewards cool and then so the result of this process is going to be a set of initial parameters such that if you explore with those parameters and fine-tune those parameters you should be able to solve new tasks and so what meta