Monitor your Cloud Run app

Key Takeaways

Steve in our last episode about reliability for serverless apps you showed me how to implement smoke tests yeah I saw there were some good discussions in the comments I feel like I'm on a roll here I only spent about 15 minutes making my more reliable using smoke tests and it felt great what's the next baby step to make my application even more reliable let's add monitor [Music] Steve you told me the other day that you have experience in reliability engineering that's right I was a site reliability engineer or SRE inside of Google for over a decade I worked on search Android YouTube and Cloud now my job is helping developers understand how to build reliable systems on the cloud excellent now why is monitoring needed well we can't see code running inside a CPU or bytes changing in memory when a user reports an error it's hard to tell what went wrong inside of this opaque box so let's make that box transparent how do we make it transparent well when something breaks our first thought is often I wish I had an x-ray machine so I could see inside the system we can build that x-ray machine ahead of time by adding monitoring I like that idea but Steve you know me I'm all about taking small baby steps how can I start to monitor my system within the next five minutes my code runs on cloudron well Cloud run provides a dashboard with some useful metrics right out of the box let's start with that okay let me go to the cloud run section of the cloud console when I click my service here I get a page full of graphs their chores for request rates request latencies and the number of containers Google has started up for me this gives me a pretty good overview of my cloud run service well that was less than five minutes right yeah it sure was but if my system has a problem will the X-ray machine in other words these charts uh tell me what went wrong well even the best x-ray machine in the world needs a doctor to interpret these results it's the same thing with monitoring when an error is reported monitoring will tell us what metrics changed and that will give us a clue as to what went wrong okay so let's say a user reports an error I would go to this dashboard and look at the graphs yeah let's say the response time graph shows an increase around the same time that a user reported the error we would call that a symptom just like with a doctor it would not tell you exactly what is wrong which is what we call the cause so it could be one of many causes maybe the database queries are taking longer or that application is overwhelmed by a spike in traffic and then I would at least know that I should check the database queries or that I should check the traffic volume that's right the dashboard should point you in the right direction telling you where in the system those symptoms are occurring and the more metrics you track the more clues the dashboard will give you about this cause of the problem one more question I've seen systems where these graphs were kind of spotty because the application only executed when someone visited the website yeah many systems are like that in our previous episode we talked about smoke tests if you run these smoke tests regularly the graphs will have some more data and you'll get a better sense of your system's Health even when users aren't around I like that that would also mean that if something breaks at 5 PM right after people have stopped using the system for the day I would learn about it right away yes you would and you wouldn't have to wait until a user hits the error say the next morning that makes sense but there are a lot of things my application does that's not covered by these default Cloud run metrics for example it calls the YouTube API and I'd like to see those calls in a dashboard too well you wanted the five minute version do you have some more time well you haven't steered me wrong yet Steve I'd be happy to invest some more time building a better x-ray machine for my system okay the next level would be to add some custom metrics that are specific to your application your application code is probably writing logs already we can convert those existing log messages to metrics that we can graph got it every time the YouTube API returns an error my code writes a message like this to the log great we can create a useful metric from that first pull up the log entries that match that text then create a counter metric this will tell us how often this entry appears in the logs if the rate of this counter goes up there may be a problem in the system oh okay I'll enter a name for the metric then I click create metric uh looks like an I can explore the new metric here I'll click that there isn't any data yet but I will save the chart I'll put it in a new dashboard done all right well does your application write any other log messages well every time a user opens the web page a pretty heavy database query runs to find the videos that they subscribe to and the code is triggered by a get request to this endpoint okay we can create a distribution metric from that this will let us create percentile latency graphs for example if we see that the 95th percentile for database queries is 100 milliseconds that means that 95 of queries took 100 milliseconds or less why don't we just measure the average time well using an average would hide a lot of the interesting characteristics of your system which would prevent us from learning more about it response times tend to vary a lot some responses are very fast and others very slow they might have extreme outliers which would be totally hidden from view if we use the average these are actually the cases that we care about the most so let's not hide them and that's where percentiles come in yeah we can use percentiles to better visualize the distribution this gives us a view over time of the shape of the distribution like how slow or the slowest and how fast were the fastest and how does user experience play into these percentiles what we really want to know is how many requests were too slow for this we need a cutoff value let's say one second would be fast enough for a user not to be frustrated in this case we will simply count the number of requests that take more than one second we will use these in something called an SLI a service level indicator which would be used to alert an operator to step in and fix something sounds good I'll pull up the matching log entries like before in the log Explorer then I'll click create metric I'll click here to explore the metric then I will save the chart I'll pick the dashboard I created before there let's look at my new dashboard all right now you have the built-in Cloud run dashboard with all the standard metrics plus this new dashboard with custom metrics for your application let's leave it for a few hours so the graphs can fill up with data and we're back Steve uh I had some use of the dashboard oh what happened a user reported that the data in the application didn't seem fresh and you used your two dashboards yes first I looked in the built-in Cloud run dashboard there were no errors or changes in latency it looked like the cloud run service wasn't throwing exceptions so probably not anything broken in Cloud run so then you checked the dashboard with the custom metrics exactly the application has two major Integrations the YouTube API and the database we created one metric for each and I needed to find out which one was breaking yeah it really helps if you know which part of the system is causing problems because you can't troubleshoot everything at once right the database latency had not changed so I figured it probably wasn't a database problem then I saw this in the YouTube error graph it looks like a problem with the calls to the YouTube API now that I knew what to look for I could check the logs in more detail and it turns out to be the new code that I released that sometimes didn't ask YouTube the YouTube API for all the right Fields it was a quick fix well it's good to hear that your application is working again uh was there something more that you wish you had tracked yes I started tracking this particular error as a new metric so if that metric spikes I know that I started sending bad requests to YouTube again this is often the pattern as new errors appear we learn more about the system and we become better at monitoring it all right you've spent maybe 15 minutes and you already have dashboards that will give you valuable Clues if there is an outage agreed what might I want to monitor next Steve it's always interesting to measure the user experience how long does the user have to wait how many arrows do they see how fresh is the data sent to them and so on the more you can measure the user experience the more you can collaborate with other teams in your organization your dashboard may even become a valuable tool for discussions between engineers and business people ah so to recap a viewers should first familiarize themselves with the built-in metrics they can be useful when troubleshooting problems then they can create custom metrics based on existing logging from the application and finally they may want to add metrics about the user experience that's right if you do all that both your applications and your organization will be in a better place great thank you for that Insight Steve and thank you everyone for watching if you have any questions for Steve or me add them in the comments also let me know if there are other serverless topics you would like to see in future episodes I read every single comment until next time [Music] thank you