Daft and Unity Catalog: A Multimodal/AI-Native Lakehouse

Key Takeaways

Thank you everyone for coming to our final presentation of the day. Please go ahead, take a seat, silence your phone. The presentation is Daft and Unity catalog, a multimodal AI native lakehouse. I will now be handing it over to your presenter, Jay. Hi. Um, can everyone hear me? Okay, that's good. All right. Yeah, great to meet you all. Thanks for coming by. I'll be giving a presentation on DAT and Unity catalog and kind of extending what we think of today as a uh lakehouse into the multimodal and AI realm like what does it mean to have a multimodal data lakehouse and what does it mean to have an AI native lakehouse? We'll be defining that in a little bit and I'll be showing you all how to kind of couple you know uh the things that we know and love such as Unity Catalog, Delta Lake, Parquet with uh an AI multimodel native engine such as Daft. A little bit about me. Uh I am a co-founder at eventual that's the company behind Daft. Uh previously before this I used to work at biotech and self-driving um at Freom and Lyft respectively. Um I've been building distributed data systems for machine learning and AI for a little bit. So think of, you know, shoveling in large amounts of images, lidar, um, genomics data, things that don't really fit in the table, but you still want to use them for your machine learning and AI. Um, and then yeah, and now I'm a maintainer of Daft. Uh, that's, uh, you know, the the query engine and data engine that we've been working on now for about two years. Um, I'm excited to show all of you the integrations that we now have with data bricks that can en enable you to utilize the entire power of DAFF on your current data infrastructure. Uh, my goals by the end of this talk is number one for everybody in the audience to understand why multimodal data is difficult and why it's different than what we have for tabular data. Number two, I'd love to show you also how the Daft engine makes this much faster and easier than any other data tool out there today. And number three, kind of tie that all back with the lakehouse architecture that you have already in data bricks and just kind of out of the box have that experience with your current data bricks setup. Um, so yeah, first of all, let's talk about what are multimodal and AI workloads. You know, a lot of people come up to me and say, oh, like AI seems very buzzwordy, right? Everything has AI appended appended to it today. I think there is a distinction between a more traditional workload and a multimodal workload. So let's talk a little bit about where we came from as an industry. Um back in the day, you know, in the 2010s, every company was trying to be a datadriven company, right? And and they were making big investments into big data. What does this mean? It actually kind of means that you have raw data. You then run jobs uh we call them ETL jobs, extract, transform, and load. And you then build this nice, you know, this like structured set of data that we usually call clean data or uh in data bricks terms, you call it the the gold tier of your data, right? And then you pipe that into analytics. You use technology such as JDBC, you use uh you know dashboarding software like Tableau, Looker, um and you plot charts and this powers business intelligence, right? BI um and and this was really what all companies were trying to put investment into in order to be a more datadriven company. A few years after that, machine learning started getting really really popular and the the use case for data kind of shifted quite a bit. Instead of producing a chart or a graph at the end of this, the end product of a machine learning data flow is usually some sort of model, right? And so in in this diagram, you know, you start with raw data. Instead of producing just a normal clean data set, you produce what's called a train test split. And so you have your training data set, you have your valid validation data set, and your test data set. And the goal here is to train a model, right? And so if you notice the data flow is quite different. And additionally, there's this one last arrow at the bottom, which is the dream of every MLOps, every machine learning company is that they want to enable this thing called a feedback loop. And the idea is that your model will be outputting data back into your raw data set so that you can then train the model and get it have it be better and better over time. This really only I think happened for a few companies and the majority of use cases ended up actually being very tabular if you think about it. Uh fraud detection is one of those big use cases today, right? Or just general classification problems. Um and really machine learning models I think overall um started becoming for mainly for two main use cases. Number one classification and number two regression. Right? Those were the two big use cases that every company was able to leverage using this model. But the training loop and the training cycles would often take days if not even months sometimes. And it really wasn't until deep learning came around that people started being able to work with multimodal data. Now that you have deep learning, the model everything doesn't really change except for the model architecture. But because you now have access to things like convolutional neuronet networks, recurrent neuronet networks, LSTMs, and you know, in 2017 transformers, you now have the ability to not just work on tabular data sets like uh number of clicks per user or impressions per user. You now can actually work with things like full text, right? Run an LSTM on it. Uh you can work with images, run a convolutional net on it. And that really enabled multimodal data. But the problem with deep learning was that anyone who wanted to do this, you would have to hire a whole team to manage your GPU clusters, right? Uh you would spin up, you know, all these big GPU resources and spend months trying to update your model and trying to get it to work really really well for a very specific use case. And so really most companies weren't able to take advantage of this. I was in a self-driving company and we could do this. But if you weren't in the business of robotics or self-driving, that's extremely difficult to find the right talent uh that you need to build this kind of a system. And you know, most recently, we now have LLMs, right? And specifically multimodal LLMs. What that means is that there's no training involved anymore. You just kind of plug and play. Well, I'll show a demo later of like how crazy this actually is when you apply on your own data. But you just take an LLM and now you can apply that on data whether it's structured or unstructured or multimodal. And you can produce these structured and clean data sets which themselves may contain you know some form of unstructured data like full text right generated text for downstream use cases you can run this for analytics you can train your own models all that but the the interesting thing too is that because LLMs are generative in nature uh the use cases now extend to classification regression and generation now you can generate data and have that kind of feed back into your raw data which is extremely powerful for use case such as synthetic data generation never possible before. And so I would argue that the the data flow today looks a lot more like this implicitly even though people don't know that's what they're doing with it. And the tools that we have today need to evolve to this architecture. Meaning that this raw data that's our data lakeink today and oh you can't really see this but let me make sure that this code is visible. There we go. It's a little dark. uh let me know if you can't see it, but I'll I'll try to explain this as much as possible. Right? So, I'm I'm going to be building a data pipeline in Spark. Uh and the data pipeline, I'll just verbally say it because it's too dark to see on the screen, but number one, we're going to download data from say 100,000 URLs. Number two, we're going to decode that data into uh images, right? The data is just raw bytes. Now, we're going to decode it as JPEG into, you know, images. Number three, we're then going to run it through a model. What does that look like today in Spark? And why is it so hard to do? So the first thing you do in Spark is you read your URLs and then number one you repartition. Why do you have to repartition? That's an interesting question, right? Okay. Back back in the tabular world, this is usually how data flows in a tabular system. You go from a you know say 100 gigabyte data set and then you usually aggregate your data, right? Data becomes smaller over time in a pipeline and it becomes very easy to manage this data at the end because it's just a chart. It's just a graph. It's a histogram, you know, it often fits in like one megabyte. It's so so easy to work with. In a multimodal world, this completely changes, right? I'm getting URLs coming into the system now. And and every URL could potentially be pointing to like 10 megabytes of data. Who knows? And so throughout the process, your data actually inflates quite a bit. rather than going getting smaller because you're running these aggregations and getting it into a form to to present in a chart, you're now actually inflating the data doing things like running models. And when you run a model, the heat memory usage is crazy, right? And Spark has no idea. And so what that means is that when you work with a system like Spark, you have to be very very careful about how large each chunk of data or in Spark terms, each partition of data actually is in order to avoid the issue. If the Spark users in the crowd ever worked with multimodel data, you're your system's going to explode. It's going to be like it's going to be like out of memory, right? That's because one partition of data inflated inflated to be so large that it just crashed your uh entire system. Naively batching your multimodal data just like you do with tabular data is a very bad idea. A thousand integers are is not in any way equivalent to a thousand images. Right? So that's problem number one. Problem number two. Now that we have all the URLs in the system, I'm now going to, you can't see this too easily, but I'm now going to run some Python code, right? I'm going to take these uh URLs. I'm going to download the data using Python, and then I'm going to decode those bytes using Python again into Python data structures, images. What's the problem with this? Well, in tabular workloads, most of the operations you're ever going to run is included with the data engine. The data engine gives you a way to add numbers together, calculate the mean, uh, hash a number, right? Hashing is just about as complicated as it gets in like a traditional database. In a multimodal AI workload, this is the norm, like not the exception, right? Users are going to come to you come to you with uh uh their functionality and their functions and you're expected to run them and you're expected to run them without dying and killing the entire job. That's very very difficult. Um, and so I make the argument here that like it actually the system looks less like a database and more like a web app. In a web app, you know, you're responsible for running user code. You're responsible for potentially fallible user code. Your user code might, you know, explode and like send you an error. You're supposed to handle that, right? User code might make network calls. That doesn't really happen in database code traditionally. And user code might make unpredictably large heap allocations. it might allocate like a 100 gigabytes of uh memory and you as a engine have no control over that and so this is extremely difficult for engines such as you know more traditional data engines like Bitquery, Trino, Spark that are not really built with this idea in mind that like the userdefined functionality is a first class citizen of the engine itself. The last thing we're going to do here is we're going to use uh a model function to run a model say on the images in our data, right? Simply express as like a UDF in in Spark. Why is this challenging? Again, in tabular workloads, the operations are traditionally pretty simple. Uh they just use CPUs, right? You just like stick things into the CPUs and you have CPU instructions to run addition, hashing, all that fun stuff. In a multimodal AI workload, the operations you run are a lot more complex. They may involve making a million parallel calls to OpenAI, right? And hoping that OpenAI doesn't go down and or like tell you to slow down. Uh it may involve GPUs. Maybe now you need to use GPUs in your data engine and run a model. Um and yeah, this means that you have what's called an external dependency. All of a sudden, your data engine can't just do work by itself. It needs to talk to something external. Whether that's an external service like OpenAI, whether that's specialized hardware, a device, a GPU device, it needs to talk to that device now. And these operations potentially may be nonident potent. If you run them more than once, they might, you know, not behave the same way. And they have funky APIs and data types, right? The GPUs want you to send in tensors. Spark has no idea what a tensor is or a token. Like Spark has no idea what a token is either. Um, so it's it's extremely challenging to work with multimodal data as we know it today. And that's really what we're trying to solve with Daft. You can think of Daft as almost this like new generation Spark that's extremely Pythonic, extremely built for uh any data modality. So it's simple and reliable data processing, right? Regardless of how big your data is or how small and also what data modality you're working on. If you're working on integers, numbers, strings, works great. If you're working on HTML, large amounts of text, code, images, uh, you know, video also works great. That's really the core idea behind what we're doing with Daft. Um, here's some Daft code. It's very simply a dataf frame. Looks very similar to what you would see in Spark. Um, you can run, you know, normal dataf framy stuff. Uh, in this case, you can't see the output, but it's basically just calculating the average over a column very SQLy. And in fact, you can also run SQL within Daft. And so we built in SQL support because we know that like multimodal support means you also support tabular workloads meaning you have to do this stuff and then yeah daft runs locally and that will be what I will demo in a bit. Uh but when you install daft it runs on your single machine just like pandas does and in fact this is as fast as something like polers or uh duct db which are kind of the best single machine execution engines today and daft is built to be just as fast as these engines. Or if your data is huge and you're working with multimodal data, uh usually the data is spanning, you know, at least hundreds hundreds of gigabytes and you want to spread this out on the cluster to take advantage of the cluster's memory and also network uh resources. With Daft, all you have to do is kind of point to a cluster and we use a technology called Ray. And so if you can spin up a ray cluster, all you have to do is point daft at it and we will automatically use the resources on that cluster instead of your local machine. Ray is actually available also on data bricks. And so if you have a data bricks account, just spin up a ray cluster and point daft at it. And all of a sudden you now have distributed computing which is pretty pretty powerful. Um and this is Daft working with multimodal data, right? The way it works in Daft is we have custom image types and so you can see we have a image column. Now that's not something you see every day, right? But you now have native support for certain multimodal types such as images, tensors, embeddings, and more as we are building them out. And we also baked in a lot of core functionality around multimodal data such as in this case what we're doing in this example is downloading data from URLs using a URL download baked in expression and then decoding these uh bytes as images. And internally, Daft is built on a Rust core, Rust async core library that makes this all super super fast. To a user, it's almost as if you're using SQL and declaring up front what transformations you want. But under the hood, Daft is doing all sorts of crazy optimizations, native code, Rust, async, Rust to make this all work extremely performantly. Um, yeah. And so what what what's uh what's the DA's story with uh data bricks and unity catalog right so actually last year in 2024 I don't know if folks in this room were here for the for the unveil of unity catalog was pretty dramatic like mate went on stage and I was like I'm going to open source this now and the open source Unity catalog on stage it was pretty cool and so we saw that and then six hours after the launch of Unity catalog uh we actually built Unity catalog support in Daft and we built this with a open- source Python implementation of Unity catalog and we donated that back to the Unity catalog community. Um, and that actually enabled Daft to read tables from Unity catalog, right? And so this is very basic support and we could do this so easily because Daft already understands Delta Lake and under the hood, Unity catalog tables are actually just Delta Lake. And so we built this in like less than six hours and it was pretty cool. Um, and what that looks like is this. You would just create a Unity catalog instance, attach it to Daft Daft attached catalog and then now you can read any table in your Unity catalog just by specifying the table name and DAFF will find it u by making API calls to Unity catalog. This is really cool because prior to this there actually wasn't any real way of reading a data bricks table if you weren't inside of data bricks. You can do this anywhere now. You can do this on your laptop. You can do this on an EC2 machine. You could do this from a Raspberry Pi. You don't have to be in data bricks anymore to read your databicks tables if you have an engine like Daft that understands how to interact with Unity catalog. Now this year we uh did a second tier of integrations because Daft is a multimodal engine. We want to integrate natively with the Unity catalog volumes, right? Volumes are really the story for storing multimodal data. Um, and what that looks like is something like this where your URLs, all you have to do is prepend your URL with DBFS slash. If you're a data bricks user, this is a very, you know, familiar pattern. It just means like hit my volume, right? And Dav now understands this URL and can map that to the appropriate Unity catalog volume and find your data for you and download it. Again, you don't have to be in data bricks to do this. All you have to do is have Unity catalog and you could be running this on a Raspberry Pi and this would work, right? Because we're not actually using the database file system. We're actually hitting the Unity catalog instance and then going outside of uh the entire ecosystem to fetch your data. So what this enables is I would argue a truly multimodal data lickout because now you can read both the tables and the volumes which means both your tabular data is covered and also your uh multimodal unstructured data is covered. Um, and so what Unity catalog really gives you is unified governance over your data. Uh, DAFF provides you still with the tabular data processing that you need with SQL and dataf frame APIs. But now all of a sudden you have multimodal data workloads, right? You can read from Unity catalog the the volumes into uh rich types such as images and tensors, convert them to PyTorch, and that's something that we will now uh demo. Let's uh take a quick look at a live demo. Fingers crossed everything works. Um but we will see that uh in this live demo we're going to be running you know um Daft against Unity catalog. Let me increase the size of my screen. Collapse this. Yeah. So note that this demo right now is actually just running on my laptop. I'm not even in data bricks, right? I just have a database account somewhere and I got some credentials and like that's it. There's no other credentials, just Unity catalog credentials and my laptop. Um, and so I would argue that in 2025, the lakehouse is finally truly open. You can access it from anywhere. Database has made good on its promise to to make this open, which is amazing. So what this demo looks like, uh, just a quick diagram is zoom out a little bit. Um, so this is the lakehouse architecture, right? We have raw storage, which is Amazon S3, and then we have data bricks, which is, you know, hosting Unity catalog. database itself doesn't store the data. They just basically hold pointers to data in S3. And then you have your compute engine. So you have your raw data, you have your catalog, and then the compute engine which is Dafted. And what Daft does in this demo is it's asking Unity catalog for uh credentials. It's asking it for pointers to the data, right? Okay. And all Unity catalog is doing is granting us permission to read this data and then it's going directly to S3 for for your data actual data access which means that like yeah you're actually not going through any data bricks infrastructure to to pull that data out except to get permissions to do this which is really powerful as we'll see in a bit. All right let's get started with the demo. So first of all I've already gone gone ahead to install Daft. Uh and you can import Daft very easily by you know just import Daft just like that. I have a bunch of setup code. Um, and then now we will attach Unity catalog to our current session. And so I'm grabbing a Unity catalog endpoint which is actually my database account uh like URL and I'm grabbing a token which I'm I've hidden from you because that's secret. Uh you can't see that. And now uh I'm just attaching this instance of Unity catalog to DAFF. And so Dav now has access to all of my Unity catalog which is amazing because what we can do now is we can load these uh volumes and tables by name. Right? So here's an example of a volume. This is a very new API in DAFF that we've built to enable interactions with volumes. But now I can just reference a volume by name that's just you know my catalog is called JEST Unity. Um the schema is default and then the volume name is test manage volume. And voila, we now have access to this volume. And we can actually inspect it. Like what is a Unity volume? In fact, a Unity volume just holds a pointer to S3. This like super scary convoluted looking S3 path, right? That's a Unity volume. Um, and all it's doing is vending credentials over that volume for me. And so now that I have access to this uh Unity catalog volume, I can now do all the work that I need to to access data within the S3 bucket without your user or you uh as you're using this system even knowing that S3 is in the picture. We'll see that in a bit. So first thing I'm going to do, I'm going to read a table now from Unity catalog. So all you do is you do dav readad table. Tell it what table you're trying to read. This this is the table identifier in uh in Unity. And voila, this is my table. For this demo, we will be using the Lion data set. Have folks heard of the Lion data set? Okay, one person, two people. Uh Lion is a image data set. It it was very popular I think a couple of years ago and still is as a basically what they did was they crawled the entire internet for images. Um and so what you'll see is that on the left there all these public URLs. It's like random URLs from the public internet. And uh in fact uh they also captioned it. And so you'll see a caption column, right? This this image represents some sort of bedroom. This one is some sort of memorabilia highwaist sleeveless, you know, mini soft jeans and then a public uh uh URL. The problem with Lion is that it's very dirty. A lot of these URLs are down now because like you know these are just public urls, right? And so what I did is I used Daft to curate this data set and get rid of any broken URLs. And additionally what I did was I ingested all the data into Unity catalog volumes. So here we have a new column that I added to the Lion data set that I ingested the the images into a volume. And so you know it's a volume because the the URL starts with DBFS slash, right? This is just like inside of my data bricks account somewhere. Um the volume name is uh this one J test unity default test manage volume. And if that looks familiar it's because it's the volume that I accessed from before. So uh my goal now is like cool you know I have this kind of table of metadata like how do I actually retrieve the images and do some work on it. Right? So let's do that really easily. All we have to do uh oh I guess also to prove to you that I'm not in data bricks. Um I'm going to try to open one of those URLs the DBFS URLs and you know as expected it's going to throw an error right because again I'm not in data bricks. And so this DBFS URL doesn't mean anything to my uh Python um uh uh process right now. Um unless you know you use DAFF because Daft understands what DBFS slash means and it will attempt to map that uh to Unity catalog. So let's run a very simple daff operation which is uh you know daf please create a new column called image. Uh we will now run this URL download operation on the unity volume url column which will download bytes and then we'll take those bytes and decode them into an image column. And there we go. Notice how fast that ran too, right? Uh like daft under the hood does a lot of crazy async.io stuff to make this all work. But yeah, these are like you know full res images being pulled down uh from from your Unity catalog volume. And now we have a new column called image that is if you notice of image type uh it's a native daft column type and that was all in like three lines of code right actually. Yeah like three lines of code which is quite amazing. All right so that that was the big unlock I think. Now you have all your data kind of behind Unity catalog both the tables and the volume and then you have S3 for the raw storage and Daft as your engine and that's you know if you notice a lakehouse next thing we do let's actually make this useful let's run some LLMs right um I built a very simple LLM here uh LLM prompt basically I'm like you know those captions that we saw they were pretty bad I don't know who did them but they're horrible um I wanted the LLM to generate good captions right maybe the use case here is I want to train in a model downstream and I want it to make make better captions and also I want the LLM to score these captions for me like get rid of the really horrible ones like that's just lowquality data I I don't want that in my data set and so I built a very simple prompt here like hey like here's how you define you know caption quality like I want captions that make grammatical and logical sense I want this to be semant I want captions to capture like semantically important focal points not just describe what's in the image but capture important context around the image. This is important because we'll see the LLM actually doing this later. And I've hidden a bunch of code here, but this is where I define my UDF uh for Daft. And that UDF is called score caption. And so all I have to do now is I just run that UDF uh on top of my caption column and my image column. And I'm going to pull out a couple of uh new columns called caption similarity, caption quality. I want a new clean caption. And then I want the reason as to why the LLM made that choice. And let's show the first couple of results. This should run pretty fast. Um, give it a couple of seconds. And so under the hood, it's like fanning out, you know, like requests to OpenAI. There we go. Here are some new results. So, here was the original caption of this one. This is a bedroom, right? Um, home interior storage for kids bedroom design. And the clean caption says, "Minimal uh bedroom interior with extensive wooden storage and large wooden bed." Um the caption quality was only 0.7. And the reason was because um it does not mention that it may be designed for kids. And so if you notice in the original caption, it says like for kids, but if you zoom into the image, it's like not really for kids. And so that's why the model decided to score it that way. And so this is really useful, right? because you can kind of use this unstructured data processing or multimodal data processing to really clean up your data sets uh as I mentioned and harness the power of LLMs within your data processing. Um let's take a look at another fun one. Uh this how about this one? This one has a really low score 0.35. It's kind of sad. Why is that? If you look at this is some sort of album cover, but the caption is just hurt Boston Pops vinyl 12 used, right? And the reason the LLM gave such a low score was um it's saying that hey this caption lacks clarity you know adds ex extra text like vinyl 12 use which distracts from the main content. Uh it doesn't really tell you that this is an this is an album cover and if you notice the clean caption says album cover for Alert and the Boston Pops conducted by Arthur Fudeler or whatever. So it's much better right it's much it's much cleaner data set that you can now use for downstream consumption. Beautiful. So that was so easy in Daft. You know, you just kind of ran this operation, run an LLM on it. Let's go ahead now and um take out the image because images are very expensive to store in a table, right? And we already have a URL, so no need to store the image. And I'm just going to write the data set back out. Let me delete my previous runs. And let's write the data set back out to essentially our data lake, right? So I'm going to write this out as paret as a a new new parquet file. and this like clean captions um uh column. So it's going to run way more than what we just ran. So a lot much larger data set and it's going to hit open AI run all the inference and store store all of our I guess inference results and and all the uh scoring made by the the the model. Uh the scoring by the model interestingly is a lot better than the one provided for by Lion because Lion, if you notice, all the similarity scores are like hovering around 0.35. It doesn't really like score very well, but the the model is very opinionated. It's like oh yeah, this one sucks. You know, it's like uh 0.35 or like this one's really good. It's like 0.8. So there we go. It wrote the data out. And note that all the operations I just ran here ran on my local machine. But if you had way more data and the need to download a lot of these images, what you want to do is actually run this on a cluster and that's just as simple as setting uh DAFF's context set run array and now you're pointing at a cluster instead and we will use cluster resources to run all the operations. Okay, now that we have this clean data set, let's go ahead and load this data into PyTorch for training. Right? That's a very common next step. You're like, let me pull this data in and run this through a model or like in PyTorch. So how you can do that, you know, we do read parquet and then we just do the same thing again. We'll just download the data again from the Unity volume URLs to visualize this and to pipe it into PyTorch. And I'm going to sort this by descending caption quality because I thought that'd be fun to show you the worst captions. So here we have some pretty bad scores. 0.2. Uh let's see what this is. Oak Furniture Land was the original caption. You know, it's saying it's basically saying that like this is a logo, but Oak Furniture Land gives you no information about that. And so the now the LM says this is a Oak Furniture Land logo with a green oak leaf design, which is way better, right? But the original caption quality was like 0.2. Um, so this is great. And I want to just like get rid of these bad caption quality uh uh rows. And so what I can do is I just specify, hey, given, you know, my my data set, I want to filter out uh all rows where caption quality is not at least 0.5. Just get rid of them. Um they're probably like low low quality, high noise data. Um this is how you enhance the data set quality for machine learning and and AI in general. And yeah, now I'll convert it to a PyTorch data set. And now you are in PyTorch land. And this is extremely fast. Daft will run this kind of in a pipeline fashion and feed you data just by iterating on it. And now you get all of your data here along with the metadata, right? That's like reason, you know, caption and this is the stuff that you will use uh to feed into your your GPUs and your and your model training downstream. Um that was the end toend use case, right? Now you have a multimodel data lakeink and you're just kind of loading from it natively into Python and PyTorch. Notice how there was like no cluster involved. You didn't you didn't need to spin up the JVM some like Spark cluster. Notice I didn't install any jars like no like it's all like one notebook one install pip install daff and you're good to go. Pretty amazing. All right, let's continue with the rest of the presentation to round up. Um yeah, so this is your multimodal data lick house. You have daft the engine on top and then you have Unity catalog the catalog layer and then below that you have the raw data storage S3. And it's like super simple. Um these are all the interactions that happen within the lakehouse but it's such a simple architecture. How does Daft do it? So under the hood Daft actually does uh what's called dynamic streaming execution. Remember how I told you before that Spark really struggles with like batching of your data. Um and it's this like static batching and you have to like make sure that your batches are small enough that it doesn't ex explode memory. D has a very sophisticated dynamic streaming execution engine. So in this use case, the batches of data are represented by the size of these pink boxes. Notice how when you are reading from your tables, it's a small like batch. When you're joining the data, it's a pretty big batch. But then as soon as we get to the URL download, the image decode, DA can recognize that this is like a very multimodal operation. And therefore, it should use small batches because when you when you download images from a URL, it's going to like expand your data by quite a bit. And so it's going to stream the data through in much smaller batches to keep your memory usage stable and not and not kind of materialize at every step of the way. And this is what makes uh DAFF really powerful and not go out of memory when working with multimodal data. The second thing is when working with user provided code, right? When when the user gives us code, Daft actually doesn't use any external JVM uh processes like Spark. And so what that means is that everything happens in one process. You don't get pi4j errors or like you know gateway errors or any of that anymore. It's all one process. There's no communication between processes. And we have very nice zerocopy integrations with pi arrow. And so if you have any sort of arrow based system daft actually integrates super super nicely with them. You can kind of move data back and forth really easily. That includes systems like numpy pytorch and that's what makes daft really powerful when working with the python ecosystem. And lastly, Daft has uh I didn't show this, but Daft actually has a really uh good resource aware scheduling uh concept, right? Where you can actually tell Daft, hey, I want to run this operation, but I need a GPU to run it. And Daft will, you know, find the GPU for you. It will it will throw you an error if your GPU doesn't exist. But if you tell it, I need one GPU to run this thing. It will schedule your work with that in mind. And so on the right here I have an example where in addition to traditional database concepts like projections you have special types of projections like active pool projects we call them which is actually you'll find the GPUs and make sure that each GPU has one process that's running on it. So you no longer have to wrangle with Spark and try to get it to use your GPUs, right? You just tell Daft, "Hey, here are my resource requirements and DAFF will handle all of the scheduling for you. Um, including things like GPUs." It's also aware, for example, if you say like URL download, it knows that URL download is very high uh network utilization and so it will attempt to pipeline that with your CPU utilization, which is extremely powerful in a multimodal setting. So here are some of the roadmap items that we have. I'm going to breeze through these so that we can get to Q&A. But yeah, we're going to be trying to integrate also with Unity catalog functions and models. Uh I think a pretty natural transition. Um tighter data bricks integrations. You should be able to work with data bricks regardless of where you are. That's a a key kind of commitment that we have like you should be able to do it from EC2 or your laptop, Raspberry Pi, like you should you should be able to do that. Um and then tighter GPU integrations. That's a lot of people what they want to do is take data from say iceberg delta lake and put it into a GPU so that they can then do work on it and then yeah we actually have a early access kind of managed service uh for daft like we'll figure out what cluster resources are necessary to run your workload um and yeah we'll help you run your multimodel data PDFs images HTML regardless of how big it is right dump us a 100 terabyte thing and we'll be able to run it all the same that's really uh the goal with what we're building there and uh yes we're hiring. So if anyone's interested in working on this kind of stuff uh please get in touch across you know engine roles um product roles uh if you're interested in like pre-training AI if you're experienced there we'd love to chat and if you're working on storage engines uh uh and cloud storage that's also really powerful and yeah send us an email I think that was it um we'd love to take questions now um but yeah pip install give us a try give us a star um yeah happy to take questions [Applause]