Migrate MySQL from Google Cloud to Amazon Aurora using DMS Serverless | Amazon Web Services

Key Takeaways

Hello everyone, my name is Wamipita. I'm a solutions architect at Amazon Web Services. In this video, I'll provide a step-by-step walkthrough of migrating a MySQL database from Google Cloud SQL to Amazon Aurora using AWS database migration service serverless mode. First, let's start by understanding the architecture overview. In this video, I'll be using a custom VPC with two public subnets and two private subnets. I also have a NAD gateway inside the public subnet which is connected to the internet gateway. It is crucial to understand that we have to add the elastic IP address attached to the NAD gateway to the authorized networks in Google Cloud SQL because when we create a a migration task in the serverless mode, the database migration service will create an elastic network interface inside your VBC and we will achieve that by creating a subnet group with only private subnets. When we do that, DMS service will create the ENI inside the private subnet and thus the traffic will flow through the NAD gateway that we have inside the VPC. This way we'll achieve the communication from our custom VPC to the CloudSQL database via the NAD gateways elastic IP address. Note that by default the database migration service will not create a public IP address when you're using a serverless mode and that's why we are achieving the communication through NAT gateways elastic IP address by having the DMS service create ENI inside a private subnet. All right, let's get started. I'm in my Google cloud console and for this walkthrough I have created a cloud SQL database running a sample database of the world data. Uh it has details like countries, cities and uh languages. It's just a sample data I have running in this MySQL uh database. First and foremost, um, what we have to do is go to connections and make sure you have the NAD gateways elastic IP address added to the authorized networks. This way, we will allow the communication coming from the NAD gateway onto the cloud SQL instance. I've connected to the database using CloudShell. Uh, let's run a simple show databases command. And you see the world uh the database here. Let's use that. I have three tables in this database containing city, country, and country language information. Our target in this video is to migrate this data over to Amazon Aurora database. For that let's switch to AWS management console over here. Let's understand the VPC layout that I have in my account. In this walkthrough, I'll be using a custom VPC that I've created in my account which has two public subnets and two private subnets as shown in the architecture diagram. And please note that the private subnets have route tables pointing to the net gateway and the public subnets have route table targeting the internet gateway. If you want to create the same uh custom VPC in your account, I'll leave a link in the description for a cloud for template that you can use to create the same. If you already have a NAD gateway in your custom VPC in your own account, you can add the elastic IP address attached to the net gateway to the authorized networks for the cloud SQL instance. All right, let's start by creating our Aurora instance. Let's click on create a database. We'll select the Aurora MySQL compatible and we'll choose de test for the walk through. Let's provide a cluster identifier. I'll call it MySQL destination instance and leave the username as admin. For this demo, uh for credentials, I'm going to use the self-managed option. You can use the secrets manager which is the recommended option in general for security. For the instance configuration, I'm going to select a burstable class T3 medium instance. Make sure that you're selecting your custom VPC that you want to use for uh this migration and the uh subnet group which has private subnets. If you don't have a default uh subnet group, you can create a subnet group with just the private uh subnets. So this way your instance will be created in the private subnets only. I'm going to select public access as no because I want the instance to be private in this case. And I'll leave the security group as default uh which is good enough for this demo. And I'll be using the same security group for uh the DMS task and also the cloud shell that I'm going to use to connect to this instance. While we are here, um scroll down to the bottom under additional configurations. We are going to provide the same database name that we are targeting to migrate in this scenario. Okay, leave everything else as defaults. Let's go ahead and create the database. The Aurora instance is now uh available. Our next step is to go to database migration service over here. Uh since we will be using serverless mode, we have to create a subnet group which consists of just the private subnets. Just a quick refresher from the architecture diagram. By doing so, the DMS service will create the ENI only in the private subnet and that will make sure the traffic is flowing through the net gateway. Let's go ahead and create the subnet group. Select our VPC and while adding subnets, select just the private subnets. Click on create submit group. All right, we are now ready to create the endpoints and then followed by the task. Click on create endpoint. First, we'll create the source endpoint which is Google Cloud SQL instance in this case. We'll select MySQL as the source engine. Note that even though Google Cloud SQL for MySQL is available as an option, this is not supported in the serverless mode yet. So we'll go ahead and select just the simple MySQL. We'll provide access information manually which is going to be the server IP address here, port number, username and password. You can scroll down and click on create endpoint. Similarly, we'll create an endpoint for the destination. Select target endpoint this time. Select Aurora MySQL as the destination and we'll again provide access information manually. Once information is provided, scroll down and click on create endpoint. Our next step is to create the task which we'll be using these two endpoints for kicking off the migration. Click on tasks. Click on create task. Let's provide a a task identifier. We'll call it DMS serless my SQL migration. Select the source endpoint which we created earlier and target endpoint. Select the serverless option. Observe that by default provisioned task mode is selected but for the purpose of the demo we want to try the serverless option. So let's go ahead and select the serverless option. And for task type we have three options available. Migrate only, migrate and replicate, replicate only. You would use migrate and replicate you if you want to do a full load and then do a continuous replication later. Uh for this scenario, I'm going to choose migrate only because I want the migration to complete after the full load is done. We'll keep the other settings as defaults. We'll turn on Cloudatch logs to observe the whole progress. And under table mappings, expand the selection rules. Click on add a new selection rule. By default, if you leave this as it is with a percentage mark, the DMS will try to copy everything that you have in the cloud SQL instance. Instead, we want only uh the world database to be copied over and all tables within the world. So, we'll leave it as uh schema name world and schema table name as percentage to copy all the tables within the world database. Under connectivity and security, choose the VPC which is custom VPC and you'll see the the DMS private subnet group pop up with a default security group. This is very important um because we want the DMS service to create ENI inside a private subnet. So selecting the private subnet that we created earlier is crucial. We'll select just a single AU option for the demo and we'll put a minimum as one DCU a maximum of uh two DCUs and rest all the settings should be fine. Let's go ahead and click on create task. There's one more step left. Uh we have to select the task and click on actions to start the task. I'm going to start the task without the pre-migration assessment. This will take somewhere around 10 to 15 minutes uh to get uh initialize all the resources. You can track the task status here as it's doing all the multi-steps of spinning up resources, setting up things for you under the hood. Um, depending on the database size, this is going to take a while. So, I'm going to come back when all this is done. All right, the task is now in stopped state. Since we just selected a full load migration, once the migration is done, all the resources have been discarded and then the task status goes into stopped mode. You can view all the cloudwatch logs. You can see that it's has calculated the workload capacity required for the replication and it has spun up two capacity units. Um and then it started a full load mode. Similarly, uh if you go to the other log stream, you can see all the activities that happened behind the scenes for replicating our database world database with uh three different tables. All right, now it's time to check our data. So, let's go back uh to our console and search for cloud show. In this demo, uh, we'll be using a VPC environment in CloudShell to connect to the Aurora instance. I'll leave a link in the description for a tutorial on how to use CloudShell for connecting to either RDS or Aurora instances securely um, using CloudShell. All right, our terminal is now ready. Let's connect to our Aurora instance. We'll run the same uh show commands to look at the databases. You can see the world database here. Let's use that. And let's run a couple of select commands on the tables that we have. Similarly, uh you can see there's some data in the country table. And finally, we can see that there's some data in the country language table as well. As you can see, we were able to seamlessly migrate data from Google Cloud SQL to Aurora MySQL using database migration service serverless mode. Thanks for watching the video. I'll leave uh links to additional resources in the description and a link to the cloud formation template if you want to create the custom VPC as I have in my account.