Top 10 Data Migration Best Practices - AWS Online Tech Talks

hi everyone my name is Chris Rogers I'm a member of the AWS global storage business development team and I'm joined today by Jeff Bartley who's a storage specialist si and we're here to talk to you today about the top 10 data migration best practices Jeff and I have put our heads together and come up with 10 best practices that we feel would be helpful to anyone looking to migrate data to AWS so with that let's jump right into it we're gonna start today by talking about migration basics we're gonna go through some principles of why you'd want to migrate what are the benefits of migrating to AWS and then we'll get into the top 10 best practices and we've listed out 10 and we'll go into some details about each of the best practices and hopefully that gives you some tangible practical advice that you can leave this meeting with and apply in your own environments we'll talk about what it means to choose the right tool for the job there's many ways to migrate data into AWS and into the cloud and we'll will then transition to talking about Jeff we'll talk about how to plan your migration and actually then we'll talk about transferring the data which is the whole point right being able to transfer data from your remote locations or from your premises into your stores in AWS and then finally we'll wrap up and we'll have some time at the end for some online questions and answers or life so we're going to start today by talking about the stages of cloud adoption this is something we talked about quite a bit at AWS and as you look at your journey to the cloud for the typical enterprise we identify various different phases starting with what we call the project phase which is where you're working on various projects and maybe you have projects around your organization that are moving to the cloud the foundation you take it from the project level to more of a enterprise wide at driven towards moving to the cloud across the enterprise and then you get at some point in time and this varies for customers it you get to the migration phase and migration phase is really where all the work happens there's a lot of things that need to be done and that depends on your application environment depends on the amount of data depends on various things location global reach and many organizations will you know need to tailor and customize the migration the good news is that at AWS we have a lot of resource resources that can help you storage experts professional services experts people that know our tools inside now and can walk you through that and guide guide you through those to try to help make that migration process as efficient as possible and then you know I think the real reason that many of our customers tell us that that they want to migrate to AWS is so that they can monetize their applications and ultimately get to the point versus the last phase on the scale where they see continuous reinvention in innovation and and that's really what customers are look like many customers that we talked to do not consider themselves cloud native you can see that cloud native curve their customers that are I guess new enough that are born in the cloud they don't have to migrate because they don't have anything to migrate from but there's a large majority of enterprises out there in the world that do need to consider data migration and application migration and that's what we're here to talk to you about today some of the common migration drivers that we see really around as I mentioned on the previous slide innovation and agility and productivity of development maybe you have data centers that you're consolidating you might be transforming your company or your product so that you can more readily compete and that requires a digital transformation or refresh of technology maybe it's just as simple as a cost reduction certainly cost is important to a lot of enterprises as they look to utilize and make use of the cloud cost reduction you can think of it as layering across all of all types of migration and all efforts in an organization certainly if you can reduce cost and improve your efficiencies that's important there's also some minor more minor drivers that we see you know customers acquire other companies or they divest the business and they need to move facilities or consolidate facilities there might be you know specific large-scale compute intensive workloads in your organization that really bode well to be to to utilize the cloud to to get access to levels of compute that are necessary for those workloads that aren't cost-effective to run on Prem for instance there might be facility real-estate consolidation data center closings things like that you might co-locate or outsource there might be contract changes to those colocation contracts so those outsourcing contracts these are some of the the common migration drivers that we see and really it's all moving towards you know through what we consider three primary business outcomes the reason you want to migrate your data is so that you can increase your agility so that you can build and operate your foundation for innovation so said another way that's really lining up your business so that it even focus on your core competency if you're not in the business of building data centers then why spend any more resources than you need to on building the data centers and that's where you know moving your data into the cloud allows you to focus on the whether it be the applications that you produce or the datasets that you produce whatever your core core competency is allow that infrastructure to be hosted by AWS if you will and managed by AWS operational efficiency along the same lines right you can obtain substantial cost savings freeing up resources to focus on your competitive differentiators focus on what makes your business unique and different and will help your business thrive and then finally reducing risk right migrating to AWS allows you to migrate to a secure and proven architecture that reduces risks allows you to get more resiliency allows you to you don't get increase your levels of security we have customers tell us all the time that their data is more secure with AWS because it's it's automatically backed up it's resilient it's secured they have complete control over their data and they don't have to worry about infrastructure and things like that so again the my the the business outcomes the reasons why customers look to migrate agility operational efficiency and reduce risk so let's jump in to the data migration best practices and Jeff and I sat down and made this list and we came up with ten but we actually came up with eleven and so we felt it was important enough to give you a bonus best best practice as part of this webinar we figure if you're listening to this today we can give you a bonus tip so always good to have a little bit more advice so let's talk about the right tool for the job and like like I mentioned earlier then Jeff will go through the planning phases and also talk about transferring data but selecting the right tool for the job is really about knowing your data right knowing that you what kind of data you're going to move whether it be virtual machines in which case you would think of something like cloud into a war for AWS which we can talk about in a moment maybe you have databases that you need to migrate in which case you can think of something like AWS database migration service which is a standalone service that allows you and put some specialized tools and resources in place to help you move databases or maybe you just have unstructured data file data which is not you know insignificant in many organizations then you can think about there's actually many tools then more than what we list here but tools like AWS data sink in a snowball we'll talk about those today and you know there are many other ways to move that data you could script it you can you there's a lot more methods right so for the purpose of this webinar though we'll we'll talk about a couple of these and we'll talk about it we'll mention some other services but suffice it to say that there's a lot of migration tools out there Jeff and I chose to give you these best practices to give you some general practices that would line you up for success to have a efficient and smooth migration of regardless what kind of data you have so best practice number two let's talk about migrating virtual machines with cotton door so if you have a lot of virtual machines in your environment and you're looking to do either migrate those to the cloud or you're looking to improve your ability to recover from a disaster to your disaster recovery plan then you can think about something like cloud or cloud indoor is a company that AWS acquired in recent past and what Condor does is they continuously replicate any application or database from any source into AWS and this gives you a very self-service quick way to manage your migrations with minimal in business disruption right in the way they do that it all starts with cotton door agents that are installed in your virtual machines that are on the corporate data center side or maybe in another cloud then it really revolves around the Cloudant or user console and there's a handshake and communication between your data center and the cloud indoor console where all of this is managed and on the other side of the console api's are created to to create a staging area and then in that state agent that as a replica of your machines as your virtual machines that is in the cloud kind of temporary at this point but there is management of that throughout using the cloud or user console and then finally if you had a disaster or if you wanted to conclude your replication to the cloud your migration then you would create a target subnet and what was included in the staging area of subnet would then be moved over to the target area so this is a little bit of an eye chart understandable but suffice it to say that cloud N'Dour is a very viable very easy way to migrate virtual machines and they can they're also very good at doing live migrations so again with minimal business source eruption you can have a solution that helps you move over your virtual machines into AWS best practice number three is let's talk about migrating databases using AWS database migration service so the database migration service is a service to migrate between on-prem and AWS you can migrate between databases and it also accounts for an automated schema conversion there's a schema tool that will help you move your database schema so again it's tailored for databases to allow you to move those very effectively and very easily and it's data replication for migration to give you Z downtime and this service has been used to migrate more than a hundred thousand databases to AWS and so if databases is your key component the key component of your data then the AWS database migration service is something worth checking out all right and so now I'm going to turn it over to Jeff to talk about planning your migration well thanks Chris so in this section we're going to go through some best practices you'll want to consider during the planning phase of your migration taking the time to plan out your migration is critical particularly when you're migrating a lot of data making sure all aspects of the migration are fully considered prior to executing the migration will lead to a more successful outcome some of the points we'll cover in this section apply specifically when using snowball and Data Sync services which Chris just mentioned previously while other points will apply more broadly so understanding the network bandwidth available to your migration is critical and will dictate how you approach your migration cloud endure data migration service and data sync our online transfer services and all use the network to transfer data into AWS knowing the bandwidth of your network will help you determine how long a migration will take for example a 1 gigabit network connection may be sufficient for moving a few terabytes of data but for petabytes of data the time to transfer may be unacceptable if the amount of data you need to move is too large or if your network bandwidth is insufficient to move your data in your desired time period then you'll need to look at an offline migration service such as snowball edge which allows you to write data to a physical device and then ship that device back to AWS where your data is written to an s3 bucket also it's important to remember that physical bandwidth does not always equate to available bandwidth network connections particularly those going to the cloud say through an AWS Direct Connect link are often shared across many applications in your organization to avoid impacting running workloads only a portion of the total bandwidth may actually be available to the migration so when you're planning your migration it's important to know how much usable bandwidth can actually be allocated to migration and then use that to plan your transfer time now as you think about what methods you're going to use to migrate your data you'll want to assess what impact the migration will have on your normal operational processes for example if you're using snowball you'll need to order and track the physical units get them integrated into your on-premises infrastructure and deploy workstations to manage the data transfers to the units for migrating unstructured file data to snowball you'll need to build scripts to manage the file migration especially if you're transferring data to multiple snowball devices when you're using snowball edge with the database migration service you'll need to manage access to your database and monitor the DMS process and tools for performing the database migration itself you'll also need to plan for one or more proof-of-concept tests to validate assumptions about the migration process including tools configuration and performance of your transfer all of this requires an additional level of planning and operational management to ensure the migration goes smoothly on the data sync side of things your network can play a big role in affecting how well your migration goes remember that every part of the network is critical whether it's the network between the Data Sync agent and the source storage system or the network connection between the agent and AWS improper network routing slow links or other issues can silently degrade the performance of your data transfers networks are dynamic and can be affected by a multitude of factors often changing on a daily basis and the WAM may not always be the biggest model neck I've worked with customers who have made certain assumptions about their network layout only to discover that those assumptions were incorrect and the communication paths were not as optimized as they had first thought so keep in mind that also high-speed data transfers can highlight issues that may have been imperceptible under lower network loads finally remember that in addition to your network performance the read performance of the store the source storage system can also impact migration speeds during your test runs monitor your network and your storage systems and make sure that both are performing optimally another thing to consider when you're planning your data migration is how much actual data will be migrated you need to know both the total amount of data as measured in gigabytes terabytes or petabytes or maybe even more as well as the number of files to be migrated both points are critical there's a big difference between transferring a hundred terabytes of data consisting of a thousand files and a hundred terabytes of data consisting of 10 million files you're much more likely to saturate your available network bandwidth if you're transferring large files than if you're transferring small files now you don't need to know the exact size of every file to be transferred but you should have a good understanding of average file sizes and how many small files you have and how many large files you have the i/o workload is very different between different file sizes and can significantly change your approach to migration for example snowball devices are best suited for the transfer of large files if you're migrating data that consists of millions of small files you'll want to batch files together and transfer the batches to the snowball rather than transferring the files individually this will dramatically increase the throughput of your snowball migration transfers both on-site and once the snowball reaches a ws on the other hand if you're using Data Sync there's no need to batch files together the Data Sync agent will take care of optimizing the transfer of all file data over the network for you now when you're working with large data sources there are service limitations you'll want to consider as well for example a single snowball edge device has about 80 terabytes of usable capacity if your source data exceeds that amount then you'll need to copy the data to multiple snowball devices and you'll need to figure out how to partition your source data set appropriately data sync on the other hand doesn't store data directly so it has no inherent capacity limitations however today there is a limit of 50 million files per task if you have a data set that exceeds this limit then you'll need to create multiple data sync tasks to migrate your data and again you'll need to figure out how to partition your data set appropriately finally you may need to migrate data sources that have multiple mount points for example you may have an AZ filer with multiple volumes or file systems each presented with its own NFS or SMB share when using data sync keep in mind that a single task can only transfer data from one mount point if you have multiple mount points on your source storage system then you'll need to create one task per mount point if you have a lot of data that needs to be migrated you'll need to scale out the resources you use for that migration both snowball and ad sync were designed to scale out to meet the data transfer needs of the largest workloads but there are a few things you'll want to consider if you're using snowball edge for transfer then you can order multiple devices and transfer data to those devices in parallel but you'll want to keep a few things in mind first depending upon your AWS region there may be a limited number of devices available for immediate shipment if the size of your data set exceeds more than a you snoball devices then you'll want to consider ordering the snowballs in groups you can work with your AWS account team to help streamline and manage the ordering process second keep in mind the operational overhead of managing multiple snowball devices including the stacking of equipment and shipping and receiving and then plan for the necessary number of resources finally consider infrastructure requirements such as power network connections and workstations for data transfer and make sure you have sufficient infrastructure to use the snowball devices as expected if you're using Data Sync for your migration you can have multiple agents accessing a single source and you can distribute your agents across multiple source storage systems you can also have a single task running across multiple agents in general you want to increase the number of agents when you're transferring small files in order to reach your desired throughput levels but again there are a few things you'll want to keep in mind first remember that an agent can only run one task at a time if needed you can queue multiple tasks that use the same agent and data sync will automatically execute the next queued task once the currently running task completes you'll also want to be mindful of the limits of your source data storage systems running multiple agents could impact the performance of your source storage we recommend running tests with different number of agents before committing to a long-running migration finally keep in mind that while Data Sync supports bandwidth throttling this is configured at a cask level if you have multiple tasks running and you need to limit the overall bandwidth the data sync consumes then you'll need to aggregate the bandwidth across all of your tasks now we talked a bit about the importance of a well tuned and properly functioning network but it's equally important to have a well running storage system data can't be transferred faster than it can be read off of your source device whether it's a virtual machine a database or a file system you want to make sure that your source storage systems are healthy rebuilds recoveries and other events might negatively impact the performance of your storage system and you may want to let those recovery events complete before running your full migration now in many migration use cases customers will want to transfer all the data from their source storage system into the cloud in some cases particularly with file systems this may require permissions or privileges beyond those of a normal user in order to read all the data available we also talked about scaling up resources such as using multiple snowball edge devices or multiple data sink agents to maximize migration performance however you'll want to make sure that your source storage system can scale as well to meet the demands being placed upon it for example some storage systems may have volume level or file system level limits that prevent scaling to a significant level of performance you'll want to run some tests ahead of time to make sure your systems can scale as expected you'll also want to consider how often your data is changing getting a consistent view of your data set is important in many migration scenarios if your data is changing often you may want to use a snapshot or other point in time view of your data for your migration source finally if you'll be transferring data from a storage system that is currently in production you'll want to weigh the impact of the migration on your users and processes again run a test prior to proceeding with your full migration to assess how much the transfer will affect your storage systems resources so once you've planned out your migration and consider things like network bandwidth the profile of your data and the impact of your source storage system you're ready to actually transfer your data as you do there are some further things that you'll want to consider one thing to consider is whether you'll need to preserve metadata as part of your transfer metadata is information about your data and is frequently stored with the data itself file systems in particular typically associate various types of metadata with files and folders this is information such as file ownership which defines a user and group that may have created the file or now have full control over the file itself file metadata also defines permissions on files and folders dictating how users and groups of users can actually access the files they may have full read write and execute access or they may only be able to read a file timestamps record when a file was created when it was modified or when it was last accessed and depending upon the file system itself there may be additional attributes flags and features that provide additional settings or behavior there are a number of workloads that benefit from preserving metadata for example making a second copy of your data for protection purposes naturally assumes the possible need to restore that data in the event of a data loss if you do need to recover then you'll want to recover both data and metadata to restore your system to its proper state another example is when migrating data into AWS file systems such as EFS and fsx in these cases you'll want to preserve metadata to do things like making sure data access permissions are preserved a third example is when using another service like AWS Storage Gateway in file mode to provide on-premises access to data stored in the cloud using data sync you can copy your on-premises files to an s3 bucket preserving metadata on the objects and then present the f3 bucket on premises using file gateway file gateway will read the metadata from the s3 object and presented appropriately to your on-premises users finally you may have high performance computing workloads that store data in s3 but process it using fsx for lustre which recently added support for reading and writing POSIX metadata to and from s3 objects in each of these cases and workloads you'll want to preserve the metadata on your source storage when migrating it to the cloud services such as data sync and cloud nor can preserve metadata when used for migrating your data to AWS now we've mentioned this best practice again and again today running a test before you start the full migration is critical to validate your plan and the assumptions you've made a test should transfer a subset of data and finish in a reasonable amount of time allowing you to make adjustments and course-correct is needed a good test run will validate a number of things first you will verify that you can successfully read data from the source system that you have all the necessary network connectivity and permissions in place to access the data if you're using snowball to transfer many small files it will also validate your batching process second your test will verify the performance expectations for your source system you'll need to select a data set large enough to reach expected performance levels this test should also give you insight into how your full migration will impact your source storage system third your test will verify that your network works as expected both from a connectivity and performance standpoint the test should validate every portion of your network to ensure there are no bottlenecks and that traffic is routed appropriately while your test is in progress you'll want to monitor your network looking for issues around drop packets retransmits or other issues that may be leading to lower than expected performance fourth you want to verify that your service level settings are configured appropriately these could be settings such as encryption keys on your snowball or filtering settings on your data sync task or schema conversion settings in DMS or image settings in cloud or your test run should validate that all of your settings are configured properly per your plan and expectations last a good test run will help validate your timeframe expectations you may find that your expectations were too optimistic and you need to allow more time for certain steps in your migration your test run should perform every step that your full migration will perform so as a bonus 11th best practice you want to think about how you verify your data either during the transfer or once the migration is complete verification ensures that the data you migrated matches the source some tools will automatically verify transfer data for you for example data sync performs check sums on all transferred data in flight this protects against any data corruption that could occur in the network itself data sync can also be configured to verify data in the destination after the transfer is completed it can verify only the files that were actually transferred or it can do a full comparison between the source and the destination locations a full verification can be useful when validating a migration prior to cutting over from the old storage system to the new system in other cases you'll need to write tools to verify the data was transferred successfully if you need to write your own validation scripts and tools make sure to plan accordingly creating the tools may take some development effort you'll also want to bake in time for the verification itself on large data sets it may take some time to verify that all data was transferred appropriately and with that I'm gonna turn it back over to Chris to wrap things up all right great thanks Jeff does a great job and now I just want to wrap up you know we've told you a lot about data migration and there's a lot to migration to think about and so you know we've gone through the right tool for the job knowing your data is first and foremost understanding what it is you want to move and we've given you some tools like cloud indoor and DMS and Jeff talked about Data Sync and snowball and a few others around the tools and the services that you can engage to to play in your migration and also transfer that data Jeff went through a lot of details gave you a lot of information and so now that we've walked you through that we would encourage you to go out take a look in more detail at some of these services learn more about them you can see some links here all of these services are listed on AWS website their specific links are included here and we'd encourage you to the best way to get started is to just go out and try it AWS has a free tier where you can go out and try these services learn about them there are resources out there public pricing migration tools resources customer case studies there's a wide array of knowledge that you can gain by taking a look at these services so we'd encourage you to take what you've heard here today and go dive in a little deeper for your particular situation and with that and we're gonna wrap up and thank you very much for joining us today it's been great being able to share this information with you and we wish you the best of luck in your migration