Profiling Performance in Python: Getting Started & Benchmarking Code Snippets

Key Takeaways

Welcome to the profiling in Python video course. I'm Neagar from Real Python and I'll be your guide on your journey into the world of profilers. What is profiling? When do you need it? And how can it actually help you improve your code? Let's start with a real life scenario. You're working on an e-commerce site with a shopping cart that has three main functions. Add item that adds a product to the user's cart. Then you have calculate total which adds up all item prices including tax and shipping. And you have save cart which stores the cart in the database. And your program is working but there's a problem. Your customers are trying to buy multiple items but each time they click add to cart they have to wait around eight whole seconds. You have to fix the slowdown. But where do you even start? Maybe the issue isn't add item. It could be making unnecessary API calls. Or maybe it's calculate total doing something inefficient like recalculating shipping and tax in a loop instead of caching results. It might even be safe cart taking too long to write to the database because of poor indexing. But then you'll just be guessing and that means you would spend hours trying to fix the wrong part of the code. How can you know for sure which function is to blame? Your savior is profiling. It shows you exactly what's making your program run slowly. So, how do you actually profile your code and measure how much time each function is taking? That's exactly what this course is all about. In the next lessons, you'll understand when and why to profile your code. You'll use one of Python's built-in timing modules called time it to measure the average runtime of a function. You'll also learn the difference between different profilers like deterministic and statistical. And finally, you'll investigate two main profilers called CProfile and PI instrument to identify performance bottlenecks in your program. Time to get started. Next up is all about when you need to profile your code. profile before you optimize. So, you're thinking about optimizing your Python code, maybe to make it run faster or use less memory. But before you do anything, ask yourself this, is it even worth optimizing? This is where software profiling comes in. It helps you figure out whether you need to optimize it all, and if yes, where to actually focus your energy. Because honestly, sometimes it's just not worth it. If the code runs once or twice or takes less time to run than it would to rewrite, then why bother? Most of the time, you'll only worry about performance after everything else works well. Now, how can you actually check if everything else is working? You can go through this checklist to make sure everything is working. Testing. Have you tested your code to prove that it actually works as expected and without errors according to your business requirements? Refactoring. Does your code need some cleanup to become more maintainable and Pythonic? And finally, profiling. Have you identified the most inefficient parts of your code? Because just like most things in life, the parita principle makes sense when it comes to making your code run faster. Often 80% of the slowdown comes from 20% of the code. Profiling helps you find that 20%. Okay. So, what is profiling and how can you actually do it in Python? Keep watching. In this lesson, you'll learn about software profiling and then compare it to other things like testing and refactoring. Software profiling is the process of collecting and analyzing various measures of a running program to identify performance bottlenecks, also known as hotspots. To really grasp this better, try to figure out if the following cases are considered profiling or they're something else like refactoring or testing. The first case is you've written a function called find prime numbers and you want to make sure it returns the correct primes in a given range. A quick way to do this is with assert. For example, the first line is checking whether calling the function from 0 to 10 returns exactly a list that contains 2 3 5 and 7. If it does, nothing really happens. If not, Python raises an assertion error telling you the function isn't behaving as expected. Now, do you think this is profiling? No. In fact, it's not. you're not getting any information about how long it takes for your function to run. This is in fact testing. The second case is that you have this function called calc that is a quadratic function that computes the square of x + one. You have def calc that takes in x and then returns x * x + 2 * x + 1. and it does work, but it could be more readable. So, you change its name to calculate quadratic and replace x * x with x to the power of two. And you fix the indentation. Now, your function looks a lot more readable. But is this profiling? No, it is not. This doesn't give you any information about how slow or fast your function is. This is called refactoring. Your third case is you're working on a script that downloads data, processes it, and then saves results to disk through two functions, download data and process data. It works, but it feels slow. So you run something called C profile, which you'll learn about later on, and see this. The column total time shows you that a function called download data is the culprit of your program being slow and process data is innocent. Now, do you think this one is profiling? Yes. Yes, it is. You found out the reason your program is slow. Notice that profiling just tells you what's making your program slow and doesn't show you how to fix it. You have to figure that out on your own. Okay, now you have a better understanding of the core purpose of profiling. It's time to zoom in on it a little bit more. Profilers come in two main types, deterministic and statistical. Deterministic profilers trace every single function call in your program. They measure exactly how long each function runs and how often it was called. You get precise reproducible numbers. A good example of this is C profile, which you'll explore in next lessons. On the other hand, statistical profilers work differently. Instead of tracking every call, they take snapshots of your program at regular intervals. They check which function is running at that moment and use those samples to estimate where time is being spent overall. In the next lessons, you'll start with one of Python's built-in timing tools called Time It to measure the runtime of small code snippets. Then you'll gradually move on to more powerful tools like deterministic and statistical profilers that help you analyze the performance of entire programs, identify slow functions, and pinpoint bottlenecks in practice. In this lesson, you'll use time it on a function to see the average time it takes for it to execute. Time it is a built-in utility module that measures the execution time of small code snippets by running them multiple times to minimize the effects of system noise and provide reliable timing results. To test out time it and other tools, you'll be working with this function. FIB is a recursive function that calculates the nth element of the Fibonacci sequence. The Fibonacci function is a great function to profile because it has recursive depth and repeated calls. Since fib calls itself twice for every value greater than one, the number of calls grows exponentially. So it makes sense to want to profile such a function. Okay, now it's time to write some code. Use time to calculate the average time it takes to execute fib 30. Here you have a Python file called time underline fib.py. You already have your fib function that you want to test out here in lines three and four. The first thing you need to do is to import time it from the time it module. So from time it import time it as you learned before time it executes a function a number of times. So you need to pick a number like iterations equal 100. Now you can actually use time it. So let's create a variable total time equal time it. You need to pass in the function you want to test out. So fib 30. Make sure you're passing it as a string. This is because time it uses strings to run your function in a clean and separate environment. This prevents your existing variables from interfering with timing results and it ensures accurate benchmarks. Now you want to tell time it how many times you wanted to run this function. So number equal iterations and you also need to set globals equal globals open close parenthesis. You need to do this because timemit runs your code in an isolated environment and doesn't automatically have access to your functions or variables. By passing in globals equal globals parenthesis, you give it access to your current global scope so it can find what it needs to run. Finally, you need to print out total time. So let's do that. Print format average time is open close curly brackets total time divided by the number of iterations. You can format with two digits after the decimal point. So, dot2f and your average time is going to be in seconds. So, let's add in seconds. Okay. Now, let's go ahead and run this. So, python 3 time it underline fib.py Okay, it took a few seconds. Time it was running the fib function 100 times. Now you have the results. Average time is 0.09 seconds. Now you know how long your fib function takes to run. But some important questions are still unanswered. For example, what is the total number of function calls? And out of the total number, how many are recursive and how many are primitive meaning non-recursive? So far you know that time it called fib 100 times. But since fib calls itself, you don't know how many function calls actually happen under the hood. How can you answer these questions? Well, one of the tools that answers these questions is CP profile. In the next lesson, you'll be learning what it is and how to use it.