What is IDP in Databricks?

Key Takeaways

What's going on everybody? Welcome back to another video. Today we're going to be talking about IDP and data [music] bricks. Now, as a data analyst, I spend a lot of my time on data collection. So, that's going to be getting the data into a usable format for other teams. And a lot of the times it's just in a CSV or it's in a database. It's really easy to use, but it isn't always the case. In fact, a lot of the data that I used to work with was in unstructured formats. Specifically, this was in healthcare when we were working directly with hospitals. And so, we would get doctor's notes, we would get clinician notes, we would get claims documents, and that data is not formatted in columns and rows. It's just free text. And I'm old enough to remember we would use OCR to extract all this data and make it usable and put it into columns and rows. That process was very difficult, filled with a ton of rules and things that would break down a lot, but I don't think that's the way that you should be doing it anymore. In fact, I've been using IDP in data bricks and it is amazing. So, let's dive into IDP and see what it is and why it is so useful. The first thing to look at is just what IDP actually is. It stands for intelligent document processing. It just uses AI to extract data and information from your documents. And I'm sure a lot of you guys have actually tried this at some point. You just pop it into Chat GBT. You say, "Hey, tell me what this document says." And it gives you kind of a right answer. I've done that myself. And for the most part, it's okay, but it's not amazing. You can even see here on the lefth hand side, this is a benchmark test for this type of process. This is going to be accuracy versus pricing. And you can see cha JBT way on the right hand side. It's going to be quite expensive for just okay accuracy right around 76 77%. You'll notice though data bricks is in the top left which is kind of exactly where you want to be. You want it to be the most accurate for the lowest price and data bicks does that. Now why does IDP matter at all? And this goes back to the exact thing I was talking about at the very beginning, which is a lot of data is not actually stored in databases and columns and rows. It's just free text. It's unstructured data. There is a ton of really important data in these unstructured data types and just these free text and documents that are sitting out there that people want to be able to use, but they just can't in its current state. Something that makes IDP really useful is that it is taking into context the document that you're processing. So, it isn't fully rules-based, which OCR pretty much was. When I was back on my data collection team, we had a lot of pipelines that had OCR in it. And I would say at least once a week, they would break. They would just break down because the structure changed or something changed in the documents or the document file type changed. It just would happen all the time. This is what makes IDP really powerful because it isn't completely rules-based like OCR was because if anything changed within that document, it just broke. it just immediately stopped working and it was horrible and we had to fix things all the time. But with IDP, it's really flexible to the document that you're actually using. For example, on this right hand side, this is just a PDF document. This is some image that somebody put out there, but it isn't sitting in columns and rows. There's free text on the lefthand side. There's columns and rows on the right hand side. Now, this data might be really useful that you want to extract even the data in the free text on the lefth hand side. With IDP, you'll be able to write pretty simple code to extract the data in the free text. And it's very easy to extract the data on the right hand side. And if things change, maybe they flip sides or more data is added or there's more free text on the lefth hand side, it's still going to be really flexible with that data type. And in future lessons, we are going to deep dive into this. I'm going to actually show you how to extract data just like this. So, let's take a look at the difference between traditional processing and IDP. Now, traditional processing is the one that I kind of grew up with or the one that I've used the most because this is just what we had when we were doing this type of work. This is going to be using OCR, which stands for optical character recognition. This is what I used and basically everybody else used for unstructured data. And you would set custom rules for each document that you were working with. There were so many different tools and templates that you can use, and those were really helpful. But again, things would break fairly easily, and so it was a lot of maintenance to actually keep up with these pipelines. The most frustrating thing about using this was when a customer would not tell you that they're going to be changing a format to a PDF or whatever document they were sending you. They would just change it and you wouldn't find out until like a week later or 2 weeks later when the data just wasn't coming in anymore. Now, it was still worth maintaining. It was still worth using those OCR systems because we would get a lot of really good data out of it, but it wasn't 100% accurate. We had several data quality analysts who would be on our team who would manually check a lot of these things and run tons of queries and tons of code to make sure that the data is really accurate. Traditional processing with OCR is not a bad system. It's just a lot of work. Now with IDP, we're going to be using an AI agent-based system. So when it processes your document, it's going to understand the layout and the context of your document without you having to create the rules for it. One of the best things about this is just that it doesn't break very easily. It's really hard because it's so flexible. it's really hard to actually get it to break. So with traditional OCR, if you change it from a PDF into like a DOCX or just a Word document, it's going to break, right? It's looking for a specific file type. But with IDP, it's just going to really easily, flexibly change from one document type to another. IDP has a ton of built-in functions, and those are what we're going to be looking at in the next several lessons. They have a bunch of built-in functions that do a lot of the heavy lifting for you. You basically just need to point at it. You need to say here's what you should be looking for and it's going to do most of the work for you. So there aren't a ton of custom rules that you need to be creating. Now in just a second we're going to talk about how we actually use IDP in data bricks but this is just a visual for you to see kind of what it looks like for that transformation. It's super simplified but it can take a raw PDF and in that PDF it could be free text, it could be columns and rows. We'll look at both in the next lessons. But you're going to parse out that data. You're going to use the IDP process which we're going to look at in just a sec. and it's going to put it into columns and rows to make that data usable. This is what I would say 95% of people want when they have these PDF documents that they want to use. They want to just have it in columns and rows that they can query off the data and they can actually use the data. And so how do you do that within data bricks? I had mentioned those AI functions and that's what you're going to use. You're going to use it for parsing, extracting and then classifying your data. If you know how to use those functions as well as just some basic SQL or Python because you can use either one, you can use IDP within data bricks to do this entire process. I personally like using SQL. That's what we're going to use in the next several lessons. But it's just as easy to use Python in the notebooks as well. One of my favorite things about using IDP and data bricks and I've used a lot of tools. I've used a lot of processes. One of my favorite things is that it's all in one place. This isn't some complex pipeline that uses six or seven different tools. It's all within data bricks. And so you have your raw data, your raw PDF files sitting in the catalog. And then once you process it, you're going to have your structured data sitting right next to it. And so you have your raw data, you have your code, and you have your structured output all in one place. And so it makes it really easy to actually do quality checks on your data. And if you've ever run quality checks on your data before, which you absolutely should be doing, when it's all spread out, it's all in these different systems, it is very, very tedious. And so actually having it all in one place makes it 10 times easier. Now in the next several lessons we're going to be diving into these functions of parsing, extracting and classifying our data. And then at the very end we'll have an end to end process where you can build out an entire process where you have your raw documents. We're going to process all of them. It could be 50, could be 100 documents. We're going to process it into structured data. I'm going to show you that it's actually quite easy and extremely accurate. I am a huge data collection nerd. This is like something I really enjoy. So, I can't wait to get hands-on, actually start writing out all the code and showing you how this works. We're going to be using the Data Bricks free edition for all this. So, all this is completely free to use. All you have to do is click on the link in the description. Go ahead and create an account and follow along in the next several lessons. I hope this was really helpful. If it was, be sure to like and subscribe, and I will see you in the next video. [music]