How Generative AI Powered Data Pipelines Increase a Data Engineers Productivity

Key Takeaways

hi I'm Stephan Senior Solutions architect from AWS hi I'm Ed Thompson I'm CTO and co-founder of matian and this is my [Music] architecture hey okay thanks for bringing this architecture here today I see you using uh our generative AI service bedrock in this architecture and I would love to learn how you make the life of data and Engineers easier using generative AI in your platform a fantastic so um where all about matian making the data engineer as productive as possible and what our data integration tools does is allow them to build uh AI powered uh data pipelines um built on cloud data warehouses such as Amazon red shift so the way that works and the way that we've Incorporated uh the new AI uh platforms for AWS like Bedrock into the service is uh these data Engineers here they have a new service which is part of our product which is called the co-pilot which we run on Amazon eks hey what does the co-pilot do then so the co-pilot is two kind of main things first of all in the background it's collecting all of the information about how to build pipelines Loco pipelines in matian data productivity Cloud okay so it's doing that by collecting the product documentation by collecting the metadata from Amazon red shift that's the red shift from a customer right this is the customers metadata yeah so it's telling uh it's telling us what tables and uh columns they have SCH and so on and so and more detailed information about that and the product documentation is kind of the knowledge about material as a product exactly just like a human would read the manual to figure out how to use the product the LM needs to read the manual too okay I always try to think of llm as people uh it's a good way to kind of get a good result I often find and the final piece we have this uh component information service that is essentially saying this is exactly what component of the dag that the customer can use and a component of the dag might be something like a filter uh crossthe line calculations a window function something like that got it the DC is kind of the ETL job right BR down data flow yeah it's the transformation yeah absolutely and you're feeding all that into bedro using Titan yeah we're using Titan to do embeddings and then we actually store that data over here in Amazon RDS with its new PG Vector capability and that allows us to vectorize all of that information for the customer per customer got it got the second part of the story is um the the data engineer can then have a conversation with the tool um and that conversation is handled using the context from PG Vector in a uh rag or uh retrieval automated generation pipeline what would be a typical question that one of your data Engineers is asking that's good um so they typically be asking maybe something simple like how do I join these two tables together and we can use the inference and the Deemer and all the uh understanding that the large language model has to do that or it might even be a more complicated multi-step process like I have this Source I have this target table how do I map from one to the other effectively yeah that's usually very time consuming for the data Engineers right it is and it's kind of the bread but work of the data engineer but anything we can do to make that more productive is going to be great for the great for them and great for their company um so the conversation allows them uh the the large language model to modify uh what we call call our DPL or data productivity language this is a yaml based language that's really specifically designed uh for use with large language models okay that's a good information so design it that it's really U you know be able to be integrated with large language models such as Bedrock exactly and like I say it's got to be most human readable language that goes and that makes it work better with the with the large language model um and so what this does is it modifies uh the dag which we store in a in a service called um the uh working tree and that um is like a git service so everything going everything's version everything's going into git so nothing can be lost and then the final part of the process is those dags are then executed on on our workflow engine and the actual data transformation is pushed back down to the data warehouse such as red shift got and it's also presented in your user interface for the data engineer right exactly so they can see step-by-step process they can see the data being modified it's all visual uh and of course the nice thing about um doing rag in a visual tool as opposed to like a codebase C- pilot is it constrains the visual tool into exactly what it can do so you get much better results because you've got very key building blocks for the pipeline go it's a very important learning also that we share here with our customers cool then there say the second perspective can I as a data engineer also use those generative AI service you absolutely can so we have all of this fantastic technology we're using it to run our co-pilot but really what's most exciting to me is getting our customers to use to run their business um so what we want to be able to do and what we've been able to do in the data productivity cloud is essentially build components for all of the key pieces of AI technology so we've got components for loading into uh Vector databases components for doing embeddings and components for prompting and interacting with the large language model those are run through our workflow engine um and when you build a dag like that the really exciting thing is bringing together the customers data MH and the large language model that's the key thing um and when we do that with rag as well uh what you get is fantastic results for really some quite complex data operations so typically what's a customer doing they're doing things like um uh summarizing data uh they're doing sentiment analysis they're asking multiple questions over unstructured data turning unstructured data into structured data got it got it so what you do here is also remove the undifferentiated heavy lifting of tying those Services together right because you have the low code tooling available to your customers absolutely and it all starts with their data of course so it's their data in red shift that we're doing that transformation on got it well thank you for sharing this architecture with us what was your your journey with you know in integrating Bedrock into your platform and also making it available to your customers absolutely so um you know talking to our customers um The Journey with bedrock has been fantastic um customers really were quite Keen to understand what was going on behind the scenes with our co-pilot but also with the large language model they want to use bedrocks I think seen in the industry is very trusted uh way to run a large language model um and uh the kind of progress that we've seen in the platform and having access to really what are kind of Frontier top top tier models uh has been fantastic every time it gets upgraded our product gets better our customers pipeline gets better it's win-win awesome well thank you for sharing this architecture with us oh [Music]