Data Product Development Automation with Strands Agents | Amazon Web Services

Key Takeaways

Do you feel like it takes very long for teams to develop highquality data products in especially in large organizations? Do you wonder if Genai agent can help? My short answer is yes. Today I'm going to show you how to use strength agents to develop data products under 30 minutes. In large companies, it is quite often to take many months to develop data products. from requirement gathering to sending to the analyst team to do initial data analysis and build data model. After that they need to hand the data model to the data engineering team to develop the code. After the data engineering team run through the code and have the data product ready, they will need to have it sent to the quality assurance team. This whole flow takes a lot of communication back and forth. agent here can help. Let's first see this problem set up to mimic a scenario. I have four CSV files sitting on my computer. In reality, these should be coming from four data sources. I have an investor file which contains basic investor information with investor ID, their names, and the date. And I have their portfolio information. That's their internal database with every investor's stock and what's their holdings. I also have a stock profiles. That's just basic information about about the stock. What's their symbol? What's the sector? I also have their transactions. This going to be a huge data set in reality because it has for every transactor for every investors, every stock transaction, what is the transaction date? What's the quality qual quantity and price they transactioned on? So following following data modeling design best practice, the data modeler team will want to design the investor and stock as a type two slow change dimension. And I will want to I will want to see a transaction and portfolio to be designed as type one type one fact tables. And all these four tables need to have a proper surrogate key to link them. Right now they only have the the natural key. Right? That's my expectation. I will first show you what strand agent can do and then we'll go back to see how we did it. So I set up my strength agent and I give it a very simple instruction saying design a data model for this data product that will be used by the fund sales analytics team and then after that build the data product. Um the source data folder is stored here and I want to see the output data model to be saved and the code be saved and the final data product to be saved. When I launch the agent, um, it first going to design the data model and when it start to design the data model, it's calling a CSV extraction tool. Uh, it run into a little error, but that's okay. Uh, it found out it's timed out, but let's try it again. And this time it succeeded and extracted the schema and then designed the data model and saved it out. And let's take a look at that saved data model. See if it's good. So it gave me two data models for data model summaries. One is at the table level. So I'm pretty happy with this. So, it first gave me here's some basic information and then they tell me here's the some fact tables and there's transaction fact table and the portfolio fact table and it's smart enough to tell me what the granularity should be and that's good and then it give me some dimension tables. So first he give me the investor dimension. That's good. I need it to be a type two slow change dimension to keep track of all the historic changes on every investor their marital status change their address change. That's just per regulation purpose. And then you give me a stock table also type two slow change dimension. It's also smart enough to create a dimen date dimension. I didn't ask for that. Um but it knows that if there's a date dimension it makes the connection of these tables very easy. Okay, cool. That's at the table level and it also also need to give me the fields at the the data model at the field level. So this is a source to target field mapping where the data engineering team can take and develop the final data product. So it will tell me this is the target table name, the target field name and their data types and it will tell me the source field. This is interesting. Some of these source are directly copied from the source table. Some of them were generated like this one. It generated the primary key and then these ones are good. They generated these date columns and the current status column and these are used for the type two slow change dimension. Good. So dimension tables good. Let's take a look at the fact table. Um let's take a look at transaction. Transaction table first it generated my primary key. That's good. And then it knows that they need to pull in these surrogate keys from the dimension tables to stamp on my on my fact table so that I can link them. Okay. And then there are additional information to pull in. Okay, this looks good. And the agent started to generate the code based on the data model. And the code generation is very long. And I purposefully have the code generation and code execution set up in two sub agents um to ensure that it generate all the high quality multi-rise code and then execute. If you have it goes together, sometimes the agent will just generate scripting code and then execute and then move on to the next part of the code. The code goes through and in the end it was pretty happy and give me this is very long. in the end gave me oh the the tables are saved. Good. Let's take a look at the final results. See if we're happy with that. I got my data product. Uh I got all the dimensions and the fact and the fact table. So let's pick one dimension to take a look. Okay. My investors good. I got my investor key not just ids. And then I got my uh start date and date and is current or not. In the real database, this will be more interesting because you'll have some outdated record that have a real expiration date. Right now, these are just dummies because nothing has expired yet. And then you will see some record is not current anymore. And that investor probably have a current record that's live. Set the investor. That's good. And then let's take a look at a fact table. Okay, so my transaction fact table looks good too. And I'm pretty happy that it stamped these keys on my fact table. I have my investor key and stock key. Now when I join these two tables, I can just directly link my investor key to my fact transaction table. I don't need to worry about find out the date range whether this transaction happened at this customer's which time range. That's good. I'm really happy. And strength agent is really strong at having the development team put in what they know the best which is what's the exist existing human process. How do we prompt it? How to teach the agent to do the work that human used to do? And it largely minimized the effort to set up the agent. Let's take a look at the code. So behind the notebook, this is the main uh workflow function that I called because this agent these sub agents are obviously working through a flow. You need to have data modeler first and I have the data engineer and I have the code runner to execute the code. So I just set it up as a workflow. That's one typical uh type of agents. Um in some other scenarios you can have multiple agents and have them collaborate basically swarm together to solve one problem. Uh in that case you don't need to define a typical flow. Uh or you can define the graph. Here are a few sub aent paths that you can take. Don't go the other ways and that's a more dictated way. So in my case it's simpler. Just a workflow. Three agents go through three steps. That's it. And you can see in my file there are only three files I imported um that I need to define. The first is the large language models. Uh I happen to define a few because when I was building it I was experimenting what's the cheapest way what's the simplest model to get it done. So we define a data model uh defined models. It's just calling the bedrock model from agent a uh strength models module. So model need to be defined and then after that I need to define the prompts and that is where user can put in all their knowledge uh and the instruction into the agents. So I have for data modeler I need to give an instruction say that you're data modeler you need to follow the data modeling best practice and extract schema to know the source data identify the facts dimensions establish relationships um and then I will give it instruction on how I want the output to look like and this will largely enhance the quality of the agents the accuracy of the agents. Similarly, I defined the prompts for my data engineer um how you write the code modularize it and one is type two slow change dimension watch out for these things and I also have a small prompt for my code runner. So that's it for the code uh for the prompts and the last thing I have to define is the toolkit and if you notice these agents each agent will have a few tools actually a large part of the agent performance is depend on the tools it's like you and me if you have a use easy to use tool it will enhance your productivity so in this example all the tools I defined myself but for strength agents you can define it or you can call MCP very easily. So these tools I defined I put them in the toolkit and the defining the definition of the tool just the Python functions and then you can use the tool decorator from strand agents uh to convert it to a strand tool. And that's it. That's how I build these agents. So the most effort just goes into defining the prompts and defining the tools. I would like to encourage you to think about what are the processes or daily routine work that you want to automate and take a look at strength website and try out building some agent yourself.