Make Your Pandas Code Lightning Fast

Key Takeaways

pandas is an extremely popular python package for working and exploring data sets it's an essential tool for anyone interested in working with data in python and while many love and use it every day i'm surprised when i come across code that could be sped up in its efficiency just by making a few adjustments in this video i'm going to show you a trick that in our example makes the code run two thousand times faster and as you start working with larger and larger data sets this trick is going to become essential my name is rob and i make videos about coding in python and machine learning if you do enjoy these videos please consider subscribing liking the video and giving me a follow on twitch all right so let's look at speeding up some pandas code okay i hope you're excited to speed up some code here i am in a jupiter lab notebook and i am just going to start by making some imports so of course we want to import pandas as pd and import numpy as np and then we're going to create our data set we're only using our data set as an example so we're actually going to create some random data and we're going to create that as a data frame like this our data is going to be about fictitious people we'll give these people random ages using numpy's random let's give them a random integer between one and a hundred and let's make all of these the same size which will be ten thousand we'll also give the time in bed for these people ran random randint we're going to give the percentage of time sleeping now here we'll give some categorical features so we'll give the favorite food of the person let's give them pizza taco and ice cream and we'll give them some food that they hate so they hate food random choice of broccoli that's right we need to add the size candy corn and eggs and we can do this and see that we have our data set here with random data for the the people that we are simulating but we'll wrap this in a function so we can call it uh let's call it get data and add this size as a parameter and have it return this data frame so now if we call get data we get our random data set so we're going to make a fictitious problem up that we're trying to solve with pandas here the problem is we're taking the data that we have here and we're going to uh given some conditions we want to give each person a reward so reward calculation if they were in bed for more than five hours and they were sleeping for more than 50 percent of the time or 0.5 we give them their favorite food otherwise we give them their hate food and just to make it a little trickier we're also gonna say if they are over 90 years old give their favorite food regardless they've lived to be 90 years old so they deserve their favorite food so we can write this so we can write out this problem or this calculation in the form of a function that we will eventually apply to our data frame we can do it pretty easily let's call this reward calc and we'll give it a row or a person from our data frame writing our problem out as code we would say if the age of the person is greater than or equal to 90 will return their favorite food if the row time in bed is greater than 5 and the percent time sleeping is greater than point five then we return their favorite food and then otherwise we're going to return the hate food so this is the sort of thing you come across a lot when working with panda data sets you want to apply some sort of a function or logic across every row in the data set and we're going to do this three different ways from slowest to fastest now the first one we'll call level one which is looping this was always my first way of trying to solve this sort of problem is looping over each row of the data set and applying our known reward calculation now what we're going to find is it's not necessarily very fast but let's give it a try so we're going to first call our getdata function then we're going to iterate over each row using the it arose function so we're going to say for index row in data frame it arrows and then for each row we'll call our reward calc and we need to store this into our data frame at the index location that we are located in so we'll do loc dot index call this reward and store it you can see it's running now and there it's finished so we want to time this and see how long it actually takes and there's this nice magic tool that we can run on the top of our jupiter cell called time it and this will run it multiple times and give us the average amount of time it took to run it's nice to time things this way because then you get an idea over multiple times as opposed to just one try and it's finally done all right we see it ran for seven runs and each run was about 3.4 seconds that seems pretty slow let's make it faster so we're gonna go into level number two which is the apply function you may have heard of using the apply function it can be very useful in this case we are going to just take our data frame and we'll apply this count reward calculation function that we created and we're going to do it on the axis equals one so that we make sure it runs through each row and it's essentially going to do the same thing as our loop but more efficiently so let's store this as reward let's also get a new data frame every time and time it like we did above all right so it's done so it ran seven times and averaged 189 milliseconds we're already seeing a huge speed improvement by using apply instead of iterating over each row but there's more let's make it even faster let's go into level three and this is the key to speeding up most of your slow pandas code and that's using vectorized functions vectorized functions work very efficiently and they're when you apply functions like this across the whole data set instead of each row i'm going to show you how we do that here so what we would do is instead of using our pre-built function for each row if we actually apply each of these conditions to the whole data frame itself let's write each of them up so we're going to say the percent sleeping remember needs to be greater than 5 for it to be true and the time in bed needs to be greater than 5 or if the age is greater than 90. so this is a vectorized version of the same code that we wrote above but it's going to run a lot faster and what it will provide us back with is an array of true or false for if it meets this condition now what we're going to do is we're going to call our get data function again we're going to call reward is equal to the hate food except for when these conditions are met so we've already filled the reward in with all hate food but we're gonna locate when these conditions are met and when they are we'll give the reward of the favorite food and let's just split these lines up so it's a little bit easier to follow and let's go ahead and run time it on this wow that's a lot faster you can see it's 6.91 milliseconds and actually the a lot of the time that was taken to run this was just in the get data function so if i remove that get data function out of this 1.57 milliseconds let's just quickly plot what the difference is in time that it takes and i have some data saved off here from a previous run these are the results from a previous run where i had the milliseconds that looping applying and vectorize each took set our index to the type plot this as a bar plot and we can see here at the time it took to run each reward type the huge jump down was by changing from a loop to apply but then you can see that going from apply to using vectorized functions made it even more fast so the key is whenever you're writing functions on pandas try to use vectorize as much as you can it's not always the case that you're able to do it but when when possible use a vectorized function i hope you enjoyed this quick video showing you how you can speed up your pandas code always use vectorized functions as you can don't iterate or loop over it unless you need to thanks for watching and i'll see you in the next video