How to solve data quality issues | Data Reliability | Meet the Author

Key Takeaways

are so many issues that can happen with your data like schema changes that are triggered by Engineers not even folks on the data team unknowns are things that you can't really predict for and you didn't know to to set tests for um so in terms of solving for those we really encourage folks to lean on a more automated processes hello everyone thank you so much for joining our book club chat this month uh we're very fortunate to have Molly joining us to talk about her brand new book oh it's not showing up data quality fundamentals yep there we go okay awesome awesome thank you let's get started with a random introduction and we'll dive right into the book my name is Molly Warwick I'm head of content and Communications at Monte Carlo which is a data observability company and what that means is we help data engineers and analysts and scientists understand the health of their data through anomaly detection field level lineage and a bunch of other tools to help improve data reliability um so I prior to this I worked at Uber I manage their engineering blog and their technical brand program so I worked with engineers and data scientists to help them write and communicate about what they were working on um so super excited to be here to talk about the book Thank you for having me Sophia thank you so much Daniel would you like to go next yeah yeah sure my name is uh Daniel uh uh I have been working for quite some time like 17 18 years I work at city it's a research and development Institute in Brazil and I have been working with data in the last one year and a half I'm an engineering manager and I manage a data team so I'm super interested in this book Loop because because we read so many interesting stuff and it's very nice to be able to talk with people that are so knowledgeable like the other authors in like new models so it's I'm very pleased to to be here and very grateful to be able to to chat with you thank you it's actually Daniel's idea to read about this book because she was working on data quality data reliability issues at his own job right that's why like you really really wanted to read this book oh that's fantastic I'd be curious to hear what what specific issues like um I guess how you came across it and what specific issues you were dealing with uh it's not like issues I mean uh I've joined this team uh recently and I have like software engineering background and when I go to a data team and I see all the stuff that we have on the software engineering side and where the data teams uh are you know I just there must be a lot of new things you know we have like okay you need to testing how how do I do units testing data and uh how can I like find more stuff about a quality I mean it's not like okay we're having lots of data down times and my Project's a mess no it's not like that like that I have this it's good to be proactive it's good to be proactive yeah internal thing like okay let's uh try to be excellent let's try to be like better and uh I don't know I think data quality is a topic that it doesn't get a lot of talk you know and I think that's true because people make decisions using data that sometimes isn't is not good data you know and uh I don't know does it get enough respect yeah yeah completely so can I just um kind of goes with my first question what's your motivation of writing this book and it's from your past experience working with the companies and like what's your experience saying what others are doing in terms of data quality control and building a reliability system yeah that's a great question and so obviously I um I don't want to speak too much for for the other authors but um this was this was a joint effort between myself uh bar Moses our CEO and our co-founder of Monte Carlo Jorge so they prior to founding Monte Carlo um worked on teams where data quality kind of was a consistent issue and they would be surfacing dashboards and reports to customers um or internal consumers of the data and um you know they would be getting slacks like the data is wrong what's going on like why why can't I trust this and so it inspired them to kind of launch launch Monte Carlo and create the observability category and a product that serves this but simultaneously Barr was writing a lot about her experiences um with this issue and she wrote she posted to towards uh towards data science uh medium Channel about the rise of data downtime and um how this was a serious problem for companies that were trying to become more data driven and make decisions based on data and you know if the data isn't accurate you can't really do that and you lose trust trust is way easier to lose than it is to gain and that was something that she experienced firsthand during her time at game site and likewisely or was that Barracuda which is a security uh kind of a security software company dealing with similar issues um and so when Barr uh was kind of writing these articles she was talking to hundreds of data practitioners about their experiences um and time and again this problem of data downtime came up and and data quality kind of really is is the root root of it all um and then for my only experience I was at Uber before this and I would work with Engineers to kind of write about the technologies that they were working on um and data quality was something that came up a lot like uh Technical Solutions to solve the problem of data quality so we created a homegrown like anomaly detection tool um that we developed in-house and scaled across the company we had like dedicated data product managers that were responsible for data quality and it was just a problem that teams across the company were facing and so O'Reilly actually reached out to us to write this book and based on my experience lior and bars experience we were like well yes of course this makes a lot of sense uh that they would want someone to write this book and so we partnered with um another person on our team Ryan Kearns who is a data scientist at Monte Carlo now but at the time he he was kind of a uh he was a student at Stanford who was actually working on this problem and kind of the intersection of of of data and philosophy and data reliability and Trust came up a lot in his research and so he's actually responsible for building some of our anomaly detectors at Monte Carlo so um so he partnered with with us on the book and we also interviewed tons and tons of practitioners whether they were customers or just folks in the fields who had spoken to us about data downtime and so we were able to kind of piece together this this narrative and um different Technical and process technical approaches and Technical processes that data practitioners can use to actually start being being serious about data quality or thinking more more intentionally about how to solve it love it love it thank you do you think it's more of a data engineering issue or data science issue or is like end-to-end everyone need to be aware of this data quality daily reliability that's a really good question and something that comes up a lot and we address very early on in the book um so actually before we wrote the book um bar and I wrote this article about the racy Matrix of data Persona so a racy Matrix is a matrix The Matrix that kind of tells you for for any given task who's responsible who's accountable who's consulted and who's informed and data quality is something that affects so many different people at the company and when you're talking about actually being the person to own that equality and to solve it it really depends on your organization and how your organization uh I guess handles data um so um I know Daniel you mentioned that uh you kind of have more of a software engineering background and so um a lot of people who come to data with that perspective and with that experience tend to think of data quality as like a data engineering problem because data quality is something that impacts systems at scale and you can kind of apply like a devops approach or they call it I guess data Ops approach to managing it um but uh there are other companies that have like a very data science heavy uh heavy team um and so the way that they think about data quality is a little more focused on like okay we have a few critical data sets we really need to make sure that the data in these data sets is accurate in that um you know it doesn't inform our ml models or or anything like that without ensuring that the data is reliable so we kind of see it coming from both angles um I would say probably like about 80 percent of the folks that I've spoken to over the past two years and this is hundreds of data teams do think of data quality as more of a data engineering problem in the data scientists are more focused on data cleaning and data um you know data wrangling and kind of like making sure that the specific data is reliable before you put it into the system whereas data Engineers are like okay uh we need to make sure that data in production is reliable because it's fueling all of these disparate products so um so that's kind of the perspective that we took and that's kind of what we've seen in the field I think that makes a lot of sense I I mean as a data scientist I care about data a lot but like where is the data from what's the the root source of the data that's kind of in the data engineering side although even though we need to understand how the data is is passing through different apis and stuff we're not actually working with that and monitor that process closely exactly but yeah yeah I feel just like with the software engineer I mean you don't say like okay they're responsible for quality is the test team you know maybe the data science we're doing their exploratory data analysis they're gonna check okay this day is pretty strange why we did not detect that earlier and maybe the data engineering team kind of create some kind of uh test with an expectation okay this this data should have been alert right or our freshness like you told okay or also volume like I mean okay we received hundreds of uh Rose every day and today we receive only 10 I mean so yeah I can see uh as you told like everyone's responsibility I mean uh in the end of the days now it's not only like creating data Ops stuff uh but it's also not uh only on the exploratory analysis side I mean I think everyone is responsible I mean and exactly yeah and kind of what we found when we were doing our research is that data scientists are very concerned with the field health and the field distribution so like what is kind of in the what what should the data be looking like based on what's feeding it but the data engineering teams are the ones who are actually building the infrastructure that supports the the the data products and the dashboards and things like that so it kind of depends where the data quality issue is coming from um if it's more like a metrics Focus thing it's probably the data scientist is the one who you know is on the line but if it's something where it's like oh our pipeline broker oh we have a freshness anomaly it's often the data engineering team that's responsible speaking of data quality all the tests that we're doing to ensure the data quality the book has mentioned that the traditional tests are hard to scale and we need to manually set up all the tests and when we write tests we only test for like I think the word is no unknowns I've seen the book and there are like a lot more oh no oh no's um what's your advice or suggestion on how to deal with the unknown unknowns and how to skill better to build a better data quality system yeah yeah that makes a lot of sense I mean that that's that's a good question um I think that um so no unknowns are things that you can predict and so for instance it might be like null values or hey I know that this data set needs to be within 50 to a thousand like the the values need to be within 50 to a thousand or whatever it is or I know that the values need to be in CST or Fahrenheit um but there are so many issues that can happen with your data like schema changes that are triggered by Engineers not even folks on the data team um there could be like volume anomalies that are affected by like issues with you know the API api's feeding data um there could be issues kind of with the the operational flow of the data platform like let's say you um you know you uh your Compu compute costs are way way out of you know um off the rails and that's something that affects kind of like um can affect Downstream sources so unknowns are things that you can't really predict for and you didn't know to to set tests for um so in terms of solving for those we really encourage folks to lean on a more automated processes um so there's something called a circuit breaker which you can actually do with an airflow it will actually kind of trigger should there be and you can it's in it's like part of the open source uh uh frame airflow framework and you can find it on their docs and all that stuff but it actually can kind of stop a pipeline for running if there's um if there's an issue uh that you've kind of pre-configured um the one the one kind of downside to doing all this manually is that it takes time and resources and so we really encourage folks to think about this from like um an automated perspective like ways that you can automate things whether that's integrating python with your um your existing like DBT or or great expectation tests or it's investing in something like a data observability solution or a data anomaly detector or something like that so the more you can automate and the more you can kind of ensure that you're focused on the data like you you know what tests to run because that's because those are important and you know what to expect but when it comes to solving for the unknown unknowns it's really valuable to kind of lean into Automation and more data Ops processes that's a great point and the book also mentioned like when you have those automated processes to check your issues there could be a lot of alerts right yes and people can get a lot of alerts and they just like don't care about them anymore yeah how do you deal with the issue where one grade yes the boy who cried well yeah or a girl who cried wolf uh yeah so um so one way to deal with that is by grouping alerts um so if you have like 10 alerts that go off within five minutes of each other you can group them because they're probably related to each other um and that's one way and then grouping the alerts and setting the notifications to Slacker email or wherever you check uh you have check for those um another thing that you can do is um do a bit of so one thing that automated observability Solutions can do out of the box is train on historical data to understand like what looks good and what looks bad or off I guess um but if you were to do this manually there are various ways that you can train uh your anomaly detectors uh kind of using um using different Frameworks and tools and we list several of them in the book um for I think you know uh High spark as one of them ml flow there there are different to open source tools that you can use to help train your anomaly detectors over time and again it does take time and it does take effort from your team but I would say grouping alerts uh training your detectors um as well as kind of setting up only the the detectors that you actually need for the data that that's more critical um that's another thing that you can do that makes sense thank you Daniel do you have more questions yeah yeah I mean uh I think I'm thinking I've been thinking about like we create lots of alerts and yeah sometimes we find stuff and but sometimes the problem is not uh from the data engineering team right sometimes it's something that uh go all the way through the the search systems so yeah yeah I don't know my feeling is that I don't know if we have some kind of decision framework I mean okay I got this error uh yeah what should I do now you know sometimes it seems like simple I don't know I have these iot devices that keep sending like temperature and one temperature is completely off and then say okay I can just discard that it will not yeah any effect but yeah you know sometimes this is very difficult to know okay yeah that was an error what do I do now I mean is this a problem from the search systems uh we'll continue happening I mean should I put this data on quarantine or something like that so yeah I don't know if you guys have any experience or advice or if your platform absolutely if you've been making for that yeah so one thing that teams can do is um and I'm sure this will be familiar to you because you have a software engineering background but applying principles of like site reliability and devops to handling some of these problems and so by that I mean putting together like a strategic Incident Management meant framework as you mentioned a lot of these alerts are not actually issues maybe it's like a scheme unexpected schema change but it triggers an alert and then that you know if you don't know that that was expected then people can unnecessarily freak out or spend or spend Cycles trying to solve a problem that isn't actually a problem and so what you can do is you can actually um if you do group view alerts then you kind of put together a framework for solving these problems you can setup alerts such that you can kind of put together a system for uh saying whether or not the alert is actually something to to care about so a lot of things that something that a lot of devops teams do is they have dedicated emojis and slacker and teams or whatever to indicate like okay this is actually a Sev one or okay this is like uh step five like we don't actually need to look at this and there is some manual effort to kind of putting something like this up and we actually have an anecdote about a t the T the data team at Red Ventures who is putting together kind of like an SLA framework for um for different issues that that surfaced and it does take a lot of time and a lot of training but it can uh it really is effective and you can scale it quite quickly and then it's it can be very meaningful and safe and save time and resources over time so one way to do that is putting together like an incident management process as I mentioned with like slas which are service level agreements basically saying like Okay um if the data is uh you know if if the if 50 uh if 50 if we have like a a row that has um or if we have a table that is 50 rows but it suddenly goes to 50 000 like that's something that we should look at and we should look at it within this amount of time and and um you know let the stakeholders know who need to know um and then their service level objectives which are kind of like the I would say the um kind of the quantifiable way of measuring slas so it's like okay here's the objective so that would be the number that we're going after for the for the amount of rows and then indicators or signals that that will tell us whether or not something looks off so um so tracking those um and then I would just say another way of doing this is also kind of putting together and not doing this but but learning on the job is putting together kind of a process of of blameless postmortem so like understanding like okay something did break here's what we're going to do and learn from next time it could be that we spun our Cycles on something that isn't actually an issue so we're going to kind of train our detector or or kind of like mute that alert uh when it comes up again because it's really just a schema change um and so something that you can do with uh data observability or most data quality monitoring tools whether you build them in-house or or purchase them off the shelf is um you know muting white noise or saying like okay there's a schema change on this table this isn't actually an alert so next time it won't bug the team um so those are a few things that you can do great that they're all uh in the book I was just yeah yeah okay okay yeah so I feel like four eight sorry and then you have another question oh yeah but my question is on another subject so yeah mine too you go ahead yeah I was thinking just about a data catalog yes because I mean uh from uh what I see it's kind of a documentation of the data that you have of the features and all this stuff and I believe it was like all done manually okay I'm gonna provide that data to my data science it has feature ABC and I'm gonna explain everything and you mentioned about uh data discovery which would be some kind of automated way of doing that and it kind of really merges with the data lineage the the field level lineage that that you guys mentioned so I I don't know I was wondering if you can explain like more about that if you have some kind of example where uh this data Discovery I mean worked fine because I don't know I I couldn't figure it out by just reading the book I mean uh how a data Discovery solution would work for data catalogs yeah yeah so um so what I'll say about that is like up until the last like maybe three or four years uh data catalogs were were a pretty manual process where teams had to actually go in and and uh write the documentation themselves about like okay this data source is feeding this this data source uh relates to the marketing analytics team um we can trust it like we validated it like it was a very manual and kind of resource intensive approach to actually maintaining a data catalog but recently there have been a lot of solutions and a lot of developments and knowledge graphs and um kind of AI and ml that have enabled kind of the aggregation of metadata so metadata is data about your data um through traces logs and and other and other uh and other means to kind of put together a holistic map of your data environment so it's how sources connect to each other um like for instance how your your take your I guess your table and snowflake maps to the downstream um dashboards and looker Tableau or whatever you're using and so um data Discovery is a way of understanding the relationships between uh between data and data data assets and data sources um in your environment and how they evolve over time so it's really just kind of a way of applying machine learning to your existing data cataloging approach um through knowledge graphs through SQL parsing through the aggregation of metadata um been kind of these other more novel approaches and there's really cool vendors and in companies that are doing uh interesting things in this space um and building one yourself is a little more complicated you would need to develop like a homegrown SQL parser uh you would need to do a lot of kind of The Upfront work yourself but it's definitely doable and something that we did at Uber we actually had our own kind of like a data Discovery enabled data catalog called Data Book um and it was used by disparate teams at the company um and I think there was like a team of like 10 who were managing it at any given time so as I said it can be a very manual process just because if you have if you have like a snowflake data warehouse and like a couple looker instances and you don't have much going on in the way of like transformation and modeling it's probably not going to be that hard but as your data pipelines get more and more uh mature and you add in deep more DVT models and um airflow or in you know orchestration and you do more complicated things with the data and they're more eyeballs on the data and um it just kind of takes a lot of time and resources and so that's where something like uh data Discovery comes in so so to answer your question more more simplistically I would say data Discovery is just kind of the ability to search for and understand your data in the relationship between data and lineage is a tool to enable that a discovery certainly um as well as other forms of knowledge graphs um and a lot of data cataloging Solutions nowadays will incorporate some elements of data Discovery some of them you can even search for like a specific data asset and it can take you to where that lives um in in your environment um does that answer your question yeah I think so uh maybe the lineage is the input so you can use Ai and generate texts yeah I would say lineage is the the lineage is the technology to enable data Discovery data Discovery is kind of an outcome yeah and you can get not only the lineage but also logs and yeah other stuff to to build that with AI okay exactly it makes more sense to me enough it makes a lot of sense well good on the same topic like different AI Technologies like before we have the SQL parser I mean you mentioned the SQL parser but a lot of the data pipeline is built on python or even other languages not SQL like how are we gonna do about that like there are a lot of airflow jobs for example could transform data do ETL in Python entirely yeah yeah that's a good question so um that's where you might use like an open source solution like Pi spark or um I mentioned uh there's actually one that we Built For This Very purpose called Pi Carlo which is just kind of like a way where you can um it's for it's for our solution specifically but it's a way where you can use Python to build uh kind of the reli to map the relationships between between different sources and there's actually um databricks is building a lot of kind of these open source tools like uh Unity catalog um is an open source solution um that works with uh Pi spark and other things to kind of basically take uh uh go beyond uh the limitations of SQL when you're putting together lineage graphs and other kind of forms of documentation about your your data um data ecosystem um and so um um so I would say that for folks who are are building lineage themselves um to go beyond kind of like the SQL parser uh I would consider using one of those kind of Open Source Frameworks and we we list several in the book and my colleague Ryan Kearns is the one who actually wrote that chapter uh so he would be a great person to reach out to to provide kind of guidance there um but there are definitely several Frameworks and apis that can kind of make uh go beyond SQL if you're trying to to integrate ml models in in your lineage graphs which is something that um that we've done at Monte Carlo with um with pi Spark that's that's so good to know thank you yeah right right yeah no not just equal you can definitely use uh like uh Python and um other Frameworks uh Ryan Kearns and I can share his information later but he's definitely the person to talk to who who is really uh really instrumental in in this chapter in the book awesome awesome thank you thank you um so I want to learn a little more about Monte Carlo if that's okay yeah so from my understanding that data discoverability data observability tool um so so the main technology is it the SQL parser the python parser to provide the data and image and also provide the anomaly detection on top of that is that is that about right yeah so so Monte Carlo is a data observability platform and we leverage um kind of a proprietary uh SQL parser and a data connector that Aggregates uh metadata um so logs traces all that good stuff to kind of put together a holistic picture of what data Health looks like for your organization so that we can trigger and send alerts to your data team when there are anomalies or changes or your pipelines break and one tool that we have within the platform is field level lineage and so that's where our SQL parser comes in we also use the antler um to build our antler as like an open source kind of graphing tool to build our our lineage graphs um there's a few other things that that we use to build it and it's outlined in the book and I didn't build it but uh but uh several folks on my on my team who contributed to that chapter um were responsible for it so um yeah so how we do that is kind of like a proprietary SQL parser and data collector that we've built actually on top of snowflake and databricks and a lot of really really kind of uh uh hot uh hot data tooling um and so what we do is we put together a holistic picture end to end from ingestion in the warehouse or like all the way down to the bi layer of um what your data looks like at any given point in time and give you the tools to to fix data issues as as they arise speaking of the bi layer yeah there could be a lot of bi tools are you all connected to all of those tools 100 we integrate with all of the major um data data stack tools so Tableau looker mode uh Power bi is an integration we announced recently um on kind of the warehousing side um databricks snowflake uh gcp redshift um all the kind of cloud native uh warehouses and lakes and then we also integrate with DBT DBT Cloud um airflow prefect and kind of all of the modeling and orchestration tools too so it really gives you a holistic look at where the errors occurred and how to how to spell them that's amazing that's the duty of the modern native stack right yes everything it's awesome 100 yeah Danielle do you have other questions no Sophie okay I think I asked my questions too uh anything else you would like to share with us I feel like I've learned so much from just communicating chatting with you uh just gives me more context about the book like yeah absolutely together I would say if anyone has oops sorry continue yeah just an observation I think is incredible that Ohio guys went to you and asked for that that book usually is kind of the opposite right the authors go there with a plan and it has to be approved so yeah it is like a statement to the great work that you guys are doing yeah I mean I think it's a testament to the the seriousness and criticality of this problem um and how data quality to your point earlier Danielle really does not get the respect it deserves and the fact that it touches so many different teams from data Engineers to analysts and data scientists all the way you know down the pipeline to the folks that are consuming the data operations teams marketing teams sales teams like data quality is a problem that affects everyone and you you know unfortunately or fortunately it's really kind of up to the data team uh to to solve the problem and make sure that the data is reliable and I think often they kind of get the short end of the stick um I have a lot of friends who are on data teams and are analysts and analytics engineers and they're kind of uh you know glorified SQL uh SQL uh SQL Keynes and queens where they're asked to like query query things and and surface insights and fix problems but they don't get the credit they deserve for for being the ones to really provide the competitive Advantage for their companies so so we really wanted to create something that would could be used as a resource and help them solve this very real problem of data downtime so that they can spend their time focusing on what matters and what actually gives them uh you know what they want to do and enjoy doing uh which is creating these really interesting analyzes and insights and and all that good stuff so um so yeah so the book is called Data quality fundamentals you can order it on Amazon if you do read it please review it um and um and if you have any questions about about the material feel free to reach out to me um you can find me on LinkedIn I have a pretty a pretty unique last name there's not many other boardworks out there um or you can just you know send a note over email and I can share my share my information uh after after the chat yes awesome thank you thank you so much and the book is also free on the yeah website yep all you have to do is search data quality fundamentals Monte Carlo and you can get a free copy of the book yes yes please go get the book it's quite nice and I really appreciate it's not a sales book you don't have sales pitching there no we do not yeah we we're really adamant about helping folks solve this problem and and so we talk about all sorts of Technologies and mention all sorts of uh elements of of the modern data stack um and so uh really really uh hope folks find it valuable awesome awesome thank you so much Molly for chatting with us today uh on a Friday afternoon hope you have a great weekend and thank you the video will be up soon and I will send you the link with that sounds good thank you so much Daniel and Sophia have a great weekend thank you too bye