The 2020 Google Outage (Detailed Analysis)

On Dec 14 2020 Google across the globe suffered from an outage that lasted 45 minutes nobody could access most of google services. 0:00 TLDR; Summary of the Outage 4:00 Detailed Analysis of the Incident Report Google has released a detailed incident report discussing the outage, what caused it, technical details on their internal service architecture and what did they do to mitigate and prevent this from happening in this in the future In this video I want to take few minutes to summarize the report and then go into detailed analysis. You can find youtube chapters to jump to the interesting part of the video. pick your favorite drink, sit back relax and enjoy. Lets get started. lets start with an overview of how the google id service works, the client connects to google authentication service to get authenticated or retrieve account information The account information is stored in a distributed manner between the different service ids for redundancy. when an update is made to an account on the leader node, the existing data in all nodes are marked as outdated, this is done for security reasons. Let’s say you updated your credit card info, privated your profile or deleted a comment, it is extremely dangerous to serve those outdated information. This was the key to the outage. The updated account is then replicated based on paxos Consensus protocol. The user id service has a storage quota controlled by an automated quota management solution, when the storage usage of the service changes. the quota is maintained accordingly either reduced or increased based on the demand .. So What Exactly Happened that caused the outage? In October 2020, google migrated their quota management to a new system and registered the id service with the new system. however some parts of the old system remained hooked up specifically the parts regarding the reading of the service usage. And because the service is registered to the new system, the old qouta system re