Webhooks & Callbacks For Beginners in Python

What is going on, guys? Welcome back. This video today is going to be very beginner-friendly, focused on beginners actually, and we're going to talk about webhooks and callbacks. So, how to use them, what they are, and why you should not always be using polling, which means asking the server, "Hey, is my task done? What's the status of this? What's the status of that?" But, that you should actually let the server call you back using webhooks, using callbacks. We're going to talk about this. This is, as I said, very beginner-friendly. This is something that you learn quite early when you start learning how to program, but some of you guys might not know about this, so this is what we're going to learn about today. If you like this video, let me know by hitting the like button and subscribing, and now let us get right into it. >> [music] >> All right, so we're tackling the fundamentals of asynchronous programming today. We're going to learn about webhooks and callbacks, and we're going to compare them to the idea of polling. And instead of rambling for too long, I want to start right away with a visual explanation. For this, I'm going to use MyPaint. And we're going to keep this one simple and concise as well, so I'm not even going to use a drawing tablet, just my mouse. So, please excuse the drawings that you're going to see here, but I want to briefly explain what webhooks are and how they work, or what callbacks are in general. Callback is the general pattern. Webhooks are callbacks in HTTP or in the web. And after this, we're going to also implement everything from scratch. We're going to have a server, we're going to have a client, which is actually also going to be a server. Going to explain that here in a second. And we're going to see what webhooks look like from the different perspectives. So, let us start with a visual explanation. I'm going to go full screen here. Let's say you have an application running somewhere. So, this is your web application here. Let's say this is a to-do manager, and you have a bunch of functionality. So, you have, for example, here the server, and then you have some user that is using that application. So, this user will send requests to your server. The server will send responses to the user, and that's the normal interaction, usually done in a synchronous way, sometimes in an asynchronous way. Doesn't really matter, but then let's say you have a third party here, which is going to be handling payment, for example. This could be Stripe, this could be PayPal, something else. This is just a payment provider, let's say. And it doesn't have to be a payment provider. This can be any other service that works with webhooks or with callbacks. So, the idea now is let's say the user wants to subscribe to a plan. What I want to do in this case is I want to create a checkout session which with this payment provider. So, when I say to the payment provider, "Hey, the user wants to buy something. Please give me a workflow, give me a checkout session, give me a web form or something like this where the user can do the payment." So, from this payment provider, I will get something back, which is let's say a link to a web form there where the user can do a checkout, where the user can pay something. So, this will give me something like an ID, let's say. And then this web form exists somewhere. And the user can enter information here in a bunch of text boxes. And then in the end, he can say pay. And that is what the user does here. So, I ask the payment provider, "Give me this form." It gives me an ID and a URL or something like this. I can then give the URL back to the user. The user can enter this form, do the payment, and now the question is, "How do I know that the user actually paid?" So, of course, the payment provider here will get the notification because this happens on the server of the payment provider. Because this here, this component, the payment form here is not part of our application. This is part of the payment provider's application. So, I don't have control over what's happening here. I don't get the information. So, what can I do to know if the user actually paid successfully? Because when the user pays, of course, what I want to do is I want to give the user premium features, for example. So, the moment I know that the user paid, I will say this user is now a premium user. He can use this feature and this feature and that feature. But how do I actually do that? Now, the naive way to do that is I get the URL, I get the ID, I pass everything to the user, and then what I do is I use that ID to ask this payment provider, "Hey, are you done already?" So, did the user actually pay? And then maybe this uh this uh payment provider will answer, "No, not yet." Or it will say, uh "No, it failed because the user provided invalid information." Or it will tell us, "Yes, the user actually paid successfully." And the idea is I can poll this server maybe 200 times, and at some point it will say yes. And then I know, "Okay, now the user actually paid. So, now this actually works, and I can uh enable the premium features." The problem is that this is very inefficient. And if I have many users that subscribe all the time, this is tedious, and this is unprofessional, and this also uh wastes a lot of resources. So, what we actually do is we work with so-called callbacks or webhooks. Webhooks are just callbacks in the web. So, the idea is on my server, I have a route. Let's call this now {slash} callback or something. And this is a public endpoint that everyone can send stuff to. And what I basically do is I say to the payment provider, "I want to create a checkout session." Same as before. Let's call this checkout like this. And I tell it, "Once you're done, once the user is finished with this, please go to {slash} callback of my application to tell me that you're done." This is why it's called callback. Call me back once you're done. So, I'm telling the server, "I want to have a checkout session. Once you're done, once the user's done, maybe with a failure, maybe with a success, I want you to call me back and tell me about it." So, now it gives me a URL. I can pass the URL to the user. The user can finish this form. Once the user is done with this, once the user clicks pay, what happens is this triggers an event on the payment provider's server. The payment provider remembers, "Hey, there's a callback for this. Let me tell him that this actually worked." And it will then just call back on this endpoint and tell me, "Hey, this ID, remember? You wanted a checkout session? It is now completed. Success." Or "It is now completed. Failure." Because the user canceled or something like this. Whatever the status is, we are getting a callback. I don't have to ask a single time if this is already done. I don't have to call myself the payment provider and ask if everything went successfully, if we're already done. I will get a callback. I will This This will use the webhook to tell me that it is actually done. So, this is all very abstract. Let us get into code to see how this actually works. For this, I'm going to go into my tutorial directory, and in here, I'm going to build a simple Flask application that is going to be the payment provider. So, the first thing we're going to build is the so-called payment provider, the thing that actually calls us back, the server. We're going to call this the server. So, UV init, UV add Flask is how I'm going to set up the environment. If you don't know what UV is, you can also just go with pip or pip3 install Flask. Just make sure you have some web framework that you're comfortable working with. FastAPI, Django, Flask, doesn't matter. Pick the one you like the most. I'm going to go with Flask. And in here, I'm going to create a server.py file. This is going to be my Flask application that will get tasks, process tasks, and then call back the user. So, for this, we're going to start with a couple of imports. Import UUID so we can have IDs. Import time so we can delay, artificially delay processes. Import random so we can make the delays random. Import threading so stuff can happen behind the scenes. Import requests so we can actually send information to the callback URL. And then from Flask, import Flask. The application is going to be a Flask application with __name__. And then we're going to keep track of the tasks in memory. Let's keep it simple. No database or anything like that because I just want to illustrate the concept here. So, that's going to be a dictionary and we're going to start by defining a helper function. This helper function will be the process function and all we're going to do here is divide two numbers. So, the user is going to pass two numbers, number one, number two, and we'll do a division. So, for this we want a task ID first to know which division is for which user. We need to be able to return to the correct sources. Then I'm going to say number one, number two, and a callback URL. Then we're going to artificially delay this just to make it seem like we're doing some work. In reality, of course, you would just do the work that takes time, but now we're we're just going to say time sleep and then random rand integer between 5 and 15. So, 5 and 15 seconds is the range. We're just going to wait a random time and then we're going to try to do the division. So, I'm going to say result is equal to number one divided by number two and then I'm going to say tasks, task ID is equal to and then I'm going to say that there is a status. In this case, the status is going to be success and the result is going to be the result of the division. Otherwise, if we get something like, let's say, a zero division error, then I just want to say that there is no result. So, status is going to be fail and result is going to be none. Then we're going to add some more delay. We're going to say time sleep two. That's always going to be there. I'm going to explain why because I want to show you that this delay is not really important to us if we're just using web hooks. It is important to us if we're doing polling. So, I'm going to delay this and I'm also going to delay the polling by two seconds and you're going to see with polling we're going to realize that these two seconds are actually a problem. We're going to feel the two seconds and with web hooks it doesn't really matter. Um, but that's basically it and now here we're going to callback the URL. So, every request that we get here, every processing of the task will have a callback URL that we just need to post to so that the respective sender knows it now it's done. So, I'm going to say if there is a callback URL, if we did provide one, we're going to try to send a request to it. So, requests.post callback URL, JSON is equal to and we're going to say here task {underscore} ID is going to be the task {underscore} ID. Then we're going to unpack tasks task ID. So, that is just an individual task which looks like this or like this. Result essentially and that is what we sent. Otherwise, if there is a problem, so a requests.request exception, if that's the case, we're just going to do nothing. We're just going to pass. That is the processing. We don't have any endpoints yet. That is just the logic of the processing, but here you can see we're calling back some URL. So, this is the magic here. We call back a URL that is specified, that is given to us. Maybe you're only going to understand why this is important once we build the client. So, this is the server. This is so to say the payment provider, in our case the divider. Um, and to this divider we can just pass any URL and what it's going to do once it's done is it's going to post this payload to the endpoint that we provided. Keep that in mind for later when we implement the other server, which is actually the client. Now, next we're going to say that we want to have a Flask route. So, I'm going to call this create task. Since we're working here with asynchronous tasks, this is not going to wait for this task to finish. So, I'm going to say here app.route/task and the methods are going to be equal to post like this, create task. And since this is post, we're going to get the data from the request. And for this, don't forget to import request from Flask. Not requests, but request. This is a different package or a different module. So, request.get_json. Then number one is going to be equal to data number one, actually number one like this. Number two is going to be number two and the callback URL is going to be the data callback URL if one was provided, otherwise it's going to be empty. And the callback URL is going to be equal to data.get callback URL. Why do we do it like this? Because if we don't provide a callback URL, I want it to default to none. This is going to give us a key error. This is going to give us none. Then we're going to just generate a task ID. So UUID.uuid4 is just going to generate a task ID. We want to turn this into a string actually and then I'm going to just say tasks and task ID is going to be equal to status always processing immediately. Result is going to be none if we don't have a result yet. And then we basically just take this task and we try to execute the processing method or the process function here in the background in a separate thread. So thread or threading.thread, target is going to be equal to process. The arguments are going to be equal to task ID, number one, number two and callback URL and this now needs to be started. Then we also need to import jsonify just so we can return JSON objects and with this we're going to say return jsonify task ID is going to be task ID and then again we're just unpacking tasks and then task ID. And we do that now with code 202. That's the proper HTTP code. And again, what do we do here? We have our process function and this here just schedules a task. So this allows a user to submit a number, number two and a callback URL and then it basically just says, "Okay, we're going to do that in the background. It's going to take some time between 5 and 15 seconds then also two more seconds for the callback to actually be executed." But this just now happens behind the scenes. So this is happening in the background. I don't care about this since we're sleeping, this is possible. In reality you course, this was not it would not be done like this with a sleep call. This would be done differently. But, for now, let's just accept that we have a task that is running in the background. We have multiple tasks that are processing in the background. And again, what we do once a task is done is we call back the URL that was passed. However, I do want to support polling, too. I do want to allow a user to ask what the status of a uh task is even if the task is not finished. So, the callback should not be the only thing that we can do. So, I'm going to say {slash} task and then here we're going to have task ID as a parameter in the path. And this is going to allow us to ask for a specific task ID and get the status. Now, here I'm also going to do a delay of time.sleep two of two seconds because I want to have it equal for the callback. We're going to have two second delay and also for the polling. So, think about it this way, we're done and the retrieval of the current status requires two seconds of work. And here to actually retrieve the status, too, and send it back, this also takes two seconds, let's say. So, we just want to keep it fair, but you're going to see that this two sec these two seconds here are much more annoying than the other ones. So, we're going to say task is equal to tasks.get task ID. If we don't have a task, we want to return that the task was not found. So, jsonify error is going to be not found. And we're going to also return code 404, which is not found. Otherwise, we're going to jsonify the task itself and return it. So, this will give us the status, too. So, we have both possibilities here. The user can create a task and then the user can ask what the current status of the task is. I want to know what the the status and the result is. Or we're going to notify the callback URL if there is one, if the user passed one, we're going to be able to call back and say, "Hey, we're done now. This is the status." Finally, let's go with if {underscore}{underscore}name{underscore}{underscore} is equal to {underscore}{underscore}main{underscore}{underscore} app.run debug equals true. And that is basically it. Let's see if we can run this now without any syntax errors. Of course, we need to also add requests, otherwise it's not going to work. But now this is running on port 5000. Now, before we can actually test this, we need to write our client, too. Don't be confused by the word client. It's actually a server, but the client in this case is what I showed you in my paint our application, our web application, and the server is the payment provider or the division provider in this case. But our client is also going to be a Flask application, so we're going to import requests and also from Flask, we're going to import Flask and also requests and also redirect and render templates. Then also our app will be a Flask app with __name__ and we're going to define that our API is hosted at http localhost 5000. Then we're also going to keep track of the task IDs that we already have in our application that we sent, just so we can do the polling, otherwise we don't know what to poll for. We need to ask if the task is already done or what the status of the task is. For that, we need to have the task IDs, so we need to keep track of this. And we're also going to keep track of the statuses, which we're going to initialize here as an empty dictionary. And we're going to keep it simple. We're just going to have an index route and a submit route. So, def index is going to be the function here. We're going to say for TID in task IDs, we want to every time we load the index page, we want to do the polling. So, we want to say is the task ready or not. So, status is of the TID, basically in our statuses dictionary, we're going to have a task ID map to the status. That's going to be requests.get. And here we're going to use our API /task and then TID. And from this, we want to get the resulting JSON object. And for our HTML file that we're going to render, we're going to say tasks is equal to and then we're going to do a triple here. So it's going to be the task ID, it's going to be statuses of this task ID and the status. And it's going to be statuses, task ID and the uh result. And we're going to do that for all the tasks or all the TIDs in task IDs. And then finally, return render template index.html. We don't have this yet, but tasks is going to be equal to tasks. So we're also going to go and create a directory templates. This directory templates will have an index HTML file like this. We're going to um implement it here in a second, but first of all, I also want to have the submit route. I'm going to create a second route here, app route/submit. This one is going to accept methods equal to post. And then def submit. We're going to get here N1 is the float of the form, so request.form. And N2 is going to be number two. And then the callback, uh in this case, we're not going to have a callback URL. We're just going to uh send an empty one. So we're going to say response is going to be equal to request.post. And we're going to post again to the API/task. And the JSON object is just going to be number one, number one, and then number two, number two. The data from this is going to be res.json, response.json. Task IDs. To this we're going to append immediately the data task ID and the statuses. So the dictionary uh also needs to be created. So for this data task ID we also want to set the data dictionary to data. And then we want to just redirect to index. So, to slash. Then we can again wrap this up by saying if name is main and this time we're going to run on a different port. So, app.run port is going to be 5001 and debug is going to be equal to true. And for the HTML stuff, I'm just going to copy-paste this because that's just a basic user interface. We have number one, number two, submit. And here we just iterate over the task that we have and show the status and the result. So, that is basically our client application now. Nothing too fancy. We're just submitting a task. This queues the task in our asynchronous system. And every time we load the index page, we're polling the statuses. So, you will see what this looks like. I will just go ahead and say UV run server.py. Going to open up a second terminal, navigate to the tutorial directory, and say UV run client.py. Where of course we have a syntax error because we don't want to have the colon here. So, that's of course a problem. And also we have to say import request, not requests. And also remove it from here. Then we can open up this in the browser and I can just type now 12 divided by 2 and this will submit the task. So, the submission of the task happens instantly. But when I reload the page, you can see it always takes 2 seconds because it sleeps for 2 seconds to retrieve the status. This is done synchronously. Now I can see that we have a result. So, if I go again, 2 divided by 3, the submission happens. But also we're waiting after the submission because we're running the index page. So, every time I need to ask, is this actually done or is this not done? And we can actually see this better if I even increase the waiting time. So, uh not not this waiting time, not the waiting time for the status retrieval, but for the calculation. So, if I say that this has to be at least 10 seconds and can be up to 25 seconds, um then you will see that when I run this, I can do the same thing. 2 / 1 takes 2 seconds to load the status again. I can reload. 2 seconds. Still processing. Reload. Still processing. And nothing is going to call back my server. Nothing is going to tell my server, "Hey, uh we're done now." At some point it's going to be done. I have to refresh and hope that I can see that it's done. There you go. Success. Uh two is the result. Using webhooks, what we can do is we can now create in our client a uh an additional endpoint that can be called by the server. So, how do we do that? I can go into the client. Let's go full screen. I'm going to say here self is going to be my own address. So, that's going to be HTTP localhost 5001. And of course, pay attention. This needs to be accessible from the other server. So, now both are running on localhost. That's easy. But if you're running something, for example, if you're using Stripe and you pass a Stripe webhook URL, this needs to be accessible. You cannot use localhost. You cannot use your private IP address. You need to have something that Stripe can actually access. So, the host name, the server that you're actually running this on. Um but yeah. So, that's API that's self. Now, for the callback URL here, I'm going to say callback URL is equal to and I'm going to use a formatted string. I'm going to say that that is myself and then {slash} callback. And going to actually pass this here as callback URL. Callback URL. So, this now tells the server, remember we have a server, this server when it gets a callback URL, calls it later. So, it posts to that callback URL. Now, the only thing we need to do is we need to handle this post it. So, we need to create another route, app route, which we're going to call {slash} callback. It's going to take post requests or accept post requests. We're going to call it callback too. And essentially, we're just going to say here data is equal to request. Now, this is flask.request or flask request, not requests the package. Uh get JSON. So, this is going to get the post request and the JSON data from it. And we're going to say statuses and from the data object, I want to get the task ID. So, task ID, that here is going to be equal to data. And then here, we're just going to return nothing and 204. So, this basically means no content. And the 202 that we used before, I think it was on the server, means accepted. So, we accepted the task and we're going to deal with it. So, now we have the callback and this is how the server can reach us. Now, in addition to that, I'm going to remove this status polling. I'm not interested in that. I'm just going to load it from the database. I'm not interested in actually retrieving it manually. So, now we are going to see server is still running. I can restart it. I can also say UV run client. And the client now no longer works with polar. So, I can run this and I can say 1 / 2. Submit. Now, I can instantly reload and you can see I don't have to wait. What I can do now is I can just do nothing and the server, once the calculation is done, once the division is finished, will call back my server. In this case, it won't because I have a syntax error. So, let me take a look at the server.py. That is right here because we didn't use a comma. That's a small error, but of course, it ruins the functionality. So, let's try this again. Let's reload this and let's reload this. Now, let's say 2 / 1 or actually 1 / 2 was what I passed. Submit. Again, I can refresh. Nothing happens. Again, now I can do nothing and basically, the server is going to call back my client. It's going to tell me, "Hey, your task is finished. This is the result." And the next time I refresh, I don't have to wait for it. The waiting time, of course, happens on the server side, but I'm not concerned with that. My server doesn't have to do anything. There you go. We got the callback. So, here we have post callback. And I Now I can just refresh and get it. And I can do that multiple times. Now I can say 2 / 1, 3 / 2, this number divided by this number. I can also say 3 / 0 to see if that works. I can do all of this. And every time I reload, I don't have to wait at all. I don't have to wait 2 seconds for polling. I can just wait here, do nothing, talk to you guys, and at some point the callbacks are going to come back in. And this is exactly how you design applications. There you go. We have one. I can reload. We can see that we have a result. This is how you properly structure and actually you don't see this because my camera is blocking a lot of this. So, let me make myself very, very small. But this is how you build professional applications uh when working with asynchronous tasks. You don't actually repeatedly poll the server. Here we got a fail, as you can see. But you let the server call you back. And this is what we call a callback pattern in general and with HTTP specifically, it's a webhook. So, these are webhooks that we're working with, the webhook URL, the callback URL. This is how you can work with asynchronous tasks professionally. So, that's it for this video today. I hope you enjoyed it and hope you learned something. If so, let me know by hitting a like button and leaving a comment in the comment section down below. Also, in case you're interested on my website, you'll find a services tab and a tutoring tab. There you can contact me if you need help with a project, if you need a freelancer. At the bottom of the two pages, you will find a LinkedIn link and also my email. You can choose any one of the two. Besides that, don't forget to subscribe to this channel and hit the notification bell to not miss a single future video for free. Other than that, thank you very much for watching. See you in the next video and bye. >> Mhm.