dataframely: Professional Validation of DataFrames in Python

What is going on, guys? Welcome back. In this video today, we're going to take a look at Data frame-ly. This is an external Python package, which is a little bit like Pydantic for data frames. So, the idea is we have data frames and we want to define certain schemas, data types, rules, consistency checks, and we can do that. And we can also enforce and validate this using this Data frame-ly package. Now, this doesn't work with Polars. Polars is like a Rust version of Pandas, which has a bunch of pros and cons. Depending on the use case, it can be more efficient. And Data frame-ly is like a Pydantic layer on top of Polars, if you want to call it that. It has a bunch of different use cases when validation is important. Now, if you like this video, let me know by hitting the like button and subscribing. But now, let us get right into it. >> [music] >> All right. So, let us get started right away. We're going to take a look at Data frame-ly, which is a validation package for Polars data frames. The idea is we can define schemas, validate them. These schemas can be about data types, but also about certain consistency checks, and certain ranges for specific fields. So, for this, let's go into the tutorial directory here. I'm going to remove everything that I have here, just so we have a clean start here. So, there's nothing in here now. And what you want to do first is you want to install the necessary packages on your system. Now, the basic way to do that is to just use pip. So, either pip or pip three install and then the packages. Obviously, we're going to need Polars today and also the package that we're talking about, Data frame-ly, like this. Data frame, like data frame and then l y in the end. Now, I'm not going to use pip to install this. I'm going to create a UV project. You don't have to do that if you don't want to. Otherwise, you can just follow along and say UV init. For this, you need to UV on your system. I'm not going to cover that in this video today. And here now, I'm going to say UV add instead of pip install Polars and then Data frame-ly. Has the exact same effect. In addition to that, we're also going to install Jupyter Lab, just so we can work with interactive Python notebooks. This is optional. You can also just work in Python files and run them, but I would also recommend running pip or pip three install Jupyter Lab. Or in my case, it's going to be UV add Jupyter Lab. And then, once you have this, you can just say Jupyter Lab like this. Or in my case, UV run Jupyter Lab. Same effect. This is going to open up this interactive Python notebook environment in the browser automatically here on localhost. And for those of you who have never worked with interactive Python notebooks, first of all, don't feel forced to use them. You can also just work with Python scripts. But if you want to know how they work, basically, you create them and you can give them a name and then you can run individual cells. I can say a equals 10 here. Then I can run this cell. I can print a down below. Run this cell. I can run another cell where I set a to 20. Then I can go back up, run this one. So, you can see we don't have a script from top to bottom. We can run individual cells and they function as their own sections, but they share the same kernel. So, if I set a to 20 here, it's also 20 when I print it. That's the inner nutshell explanation. Now, here I want to show you how we can work with Polars data frames and Data frame-ly for validation. So, I'm going to start by saying import Polars as pl and also import Data frame-ly as dy. This is how they call it in the docs. This is the alias, the official one. And now, we can create a class to define our schema for a certain data frame. For example, let's say I want to have a person schema, inherit now from dy.schema. And the basic idea is I now specify every column, what it means and what it's about, and then I also force a data type and later on I can define rules. For example, let's say every person has a social security number, which is going to be the unique identifier. I'm deliberately making this, so I can make a rule later on that it's unique. But for now, I'm just going to say social security number is going to be equal to, let's say it's a string. So, I'm going to say dy.string. Now, the question is, does a person have to have or need to have this social security number? Do I want to make this field nullable? In my case, I want to force it, so it's not going to be nullable. Nullable equals false. There must be a value, a non-null value for this field. Then I can also say same thing is true for this for the name, which is going to be also not nullable, also a string. Then I can do something like age. In this case, it's going to be an integer. We don't just type integer here. We specify the specific type, so it's going to be a uint8. So, an eight byte integer. And this one is going to be nullable. So, we don't want to force a value. We can also deal with not having an age here. Then maybe a job. Let's say this is going to be a dy.string. It's also going to be nullable. It's an optional field. And then maybe finally, we can say years of experience in that specific job. So, dy.uint8. And this one can be null as well. So, nullable equals true. So, this would just be the types. If I want to use the schema, I can already. I don't need to define anything else. This is going to force the social security number to be a string, the age to be an integer. I cannot just put a string there. But it doesn't do much more. In addition to that now, I can define certain rules. What would be an example for a rule? One example would be a unique social security number. A second example would be the age should not be negative. You cannot be negative two years old. Also, years of experience should probably not be larger than the age you have. You cannot have 20 years of experience if you're 10 years old. So, that should also be a consistency rule here. So, let us start with a very simple one. First, let's say I want to have at. This is the decorator here. So, we have to decorate the function for the rule. I'm going to say dy.rule. And then I can just say def years of experience less than age. You can call this whatever you want. I just like to be expressive here. And this rule here is going to ensure that the years of experience we have are less than the age. Now, you can say less or equal to if you want to, but I doubt that anyone starts coding or doing any job at zero years old. So, probably you even want to say it has to be at least um, you know, there has to be a gap of at least 10 or 15 years, depending, I don't know. Like maybe there's some job that you can start pretty early. But here, we're also going to pass cls. It's a class method essentially. And we're going to do the check by doing the following. I'm going to return pl. This is Polars now. So, Polars column and the column is that the years of experience column has to be less than pl.column age. Now, as far as I understand it, and I might be wrong about this, but I think I'm not wrong about this. You don't even need to do checks here for null values. So, if one of them is null, it's it's just not going to fire. So, it's not going to fail just because one of them is null. It's not going to give you an exception for that. It's just going to ignore this rule. But if both of them are actually filled with integers, you're going to check this rule and if it's violated, it's going to fail the validation. Then, second rule, let's go with age not negative or something like this. Also pass the class here. Return pl.column and then age has to be greater than or equal to zero. Now, I should spell negative correctly. Then finally, we can do a more fancy check, which is that the social security number is unique. Now, how do you do that? You cannot just do that with a column value. You need to group and see if you have more than one value when it comes to the social security number. So, I'm going to say unique ssn. And what we do here now is in the rule, we have to specify that we group by. So, group_by is equal to. And then I'm going to pass a list. This list is going to contain ssn. So, that is going to already be grouped in here. I just have to check the length of this grouped result. So, I'm going to say pl.length has to be um, less than one. Less than two. Or actually, let's keep it simple. It has to be one. Because obviously, every row at least contains one entry, so it has to be exactly one. Every ssn has to occur exactly once. And of course, don't forget the cls here. So, that would be a schema with three rules and some type definitions. So, how can I now verify a data frame? First of all, let's create one. Let's say df is equal to Polars data frame. And let's go and create a dictionary here. Now, I copy-pasted the data here to speed this up, but basically, we have the ssn here. We have the name. We have the age. We have the job. We have the years of experience. And this data here is consistent because we have only one entry per ssn. We have ages that are positive. And they're also always larger than the years of experience. And the important fields are not null. So, I can just say this is my data frame. And now, I can use my class to validate it. So, I can say person schema.validate. And then df. And I also want to say cast equals true, so it can automatically cast if necessary. So, this is then going to give me the data frame if it's validated. So, I can say here validated df is equal to that. And this gives me actually the Polars data frame. And I know it's actually working. Now, I can also adjust some stuff. For example, I can go and say I want to use one, two, three twice. So, what happens then? I get a validation error. One one rule was one rule failed the validation here. Unique ssn failed for two rows. Now, I can also do that now for a bunch of different things. Let's go and say John just doesn't have a name anymore. I can set this to none. You can see two validation rules failed now. Then I can say here age is 10, years of experience is 20. What happens then? Three validation errors. And then I could even say something like I have -10. That would give me still three validation errors because I only have three rows, but I have the same row with two different errors. And this is how you can see this is not going to return anything. It's going to basically give you an exception which you need to handle. So you can always force validation. If something is not right, it's not going to proceed with the rest of the code. Now, if you don't want to raise an exception, if you just want to find the mistakes, let me reverse some of these. So maybe let's just have a double um a double SSN here. Now, what I can do is here I'm going to get one validation error. I can also filter. So I can say not validate, but filter. And that is going to return two things. It's going to return good and bad. So in this case, if I look at good, I'm going to get the rows that actually don't have a problem, which in this case Yeah, in this case it's just um John because John has a unique SSN and both of these don't have a unique SSN. If I change this now here to 789, but I say for example that Mike is uh 5 years old. In this case, I would get two rows that are good because these other two are actually consistent. And the bad row would just be the one with the invalid age. Now, as you can see here, bad is a filter result. What I can say now is I can say bad dot counts for example, and it's going to show me how many rows are actually um affected by this. And it's this one here, years of experience less than age effects one row. One row failed this particular row. And if I want to print them, I can also say bad dot invalid, and this is going to give me the row that actually failed. And this is basically the magic that I want to show you. It's not super fancy. It's not uh super complicated. Of course, you can go to the docs. There's more stuff that you can do. But this is just a basic showcasing of a useful package. If you're working with Polars, you can just do your data frames as always, but you can also add a validation layer to it, a Pydantic sort of validation layer uh layer where you can specify the types, but also specific rules. So every time you create a a data frame where you change something about a data frame, you can run the validation and see if everything works correctly. So that's it for this video today. I hope you enjoyed it and hope you learned something. If so, let me know by hitting a like button and leave a comment in the comment section down below. Also, in case you're interested, on my website you will find a services tab and a tutoring tab. There you can contact me if you need help with a project, if you need a freelancer, or if you need consulting, or if you want me to teach you something one-on-one, you can contact me at the bottom via LinkedIn or email. Besides that, don't forget to subscribe to this channel and hit the notification bell to not miss a single future video for free. Other than that, thank you very much for watching. See you in the next video and bye.