AWS S3 Deep Dive: Vector Buckets, Amazon Q, and CLI Data Management

Key Takeaways

Now can you see here that we have different types of S3 buckets like general purpose bucket, directory bucket, table bucket, vector bucket. These are different categorization. One way to understand is you can ask your queries to Amazon Q. So this is Amazon Q and AI assistant. And when I'm going to ask to Amazon Q, I'm going to ask like differentiate among different S3 bucket types. Okay. So see here what is the reply we are going to get. So I'm writing differentiate among different S3 bucket types. Okay. So what Amazon Q does? Amazon Q browse the AWS documentations. Okay, it is going to browse the blogs and the info whatever it finds relevant and then it is going to provide you the answer or the response to your queries you're asking here. Now every service or every navigation tab will be having Amazon Q integrated. You're going to say you can see the talking about there are two main categorizations bucket types and storage classes under bucket you have different types okay we have general purpose bucket then we have directory bucket then we have table bucket this is new and similarly we have vector bucket now you can see this vector buckets is designed for a IML vector data storage it is optimized for vectors embeddings and similarities search. In a similar manner, if you're talking about table bucket, it is specialized for structured data sources and it is optimized for tabular data format. If you're talking about directory bucket, it is optimized for high performance workloads and general purpose is the original and most common S3 bucket type. Now anybody of you if you want to relate it with real world analogies you can see this particular blog here I have explained bucket types not just storage but I have explained the hidden logic behind bucket type. So I have set general purpose bucket all in one container and then I have taken the pointers about it in a similar manner directory bucket is a organized closet. So basically I have used analogies different analogies here. Okay table bucket it's a spreadsheet in the cloud and if I'm talking about vector buckets it's a brain behind artificial intelligence. Okay. So why I'm giving you an idea about these bucket types because you know based on the use case you are dealing with you can create a bucket of a specific type. Okay. So now we are going to create one bucket and then we are going to explore different features inside that bucket. So it's your choice either you can create a bucket from the management console or you can create a bucket using CLI. So let me open CLI and for that I will be using cloudshell and I'm going to create a bucket today. Let me do that. So let's wait for the cloud shell to be loaded and the terminal to be appeared. So here the command is this AWS S3 MV and then you have to provide the bucket name. Okay. So you can provide any name. The pattern is S3 col double slash. Okay. So let's say I'm giving a bucket exploring DS bucket. Okay. And if you want to append date, you can append a date. For example, I'm pending a date. Okay. And press enter. If everything would be fine, this bucket would be created in your AWS account. And how can you verify this bucket? Go back to S3 dashboard. In the S3 dashboard, if you're going to refresh it and let's say if you want to sort it on the basis of creation date. Let's see that this is the one which we have just now created exploring DS bucket with the date. Okay. Now this DS bucket we are going to use in order to explore different behavior like first behavior we are going to explore that is folder. What do you mean by folder? Okay. So although S3 is an object storage means it follows flat structure. So here folder does not means physical folder. Here folder we are using a keyword for prefixes. Okay, this prefixes makes the organization intuitive and helps with performance partitions. Okay, a very simple idea about understanding the concept of this folders or prefixes is that let's say I'm generating the cost allocation report or cost report every month like for my AWS services or even you can put a cost report on a daily basis. Okay, let's say I'm storing this report in the S3 bucket. If I will be creating the prefixes, prefixes let's say current date or I can create a prefix on the basis of daily date and then I am going to upload the file into that prefix. So these folders are helping you out to create prefixes to create partitions so that it would be easier to retrieve the relevant data you are looking for either over the web or either through the databases queries. Okay. You can also restrict or apply the permissions based on your need. Now I have taken this particular diagram for all of you. If you will be seeing this diagram suppose I have a bucket called my DS project and we have created some prefixes here like one prefix for raw then processed then models then experiments then reports. First of all, these logical prefixes are giving you an idea, an actual overview what exactly you are going to store and where. Let's suppose if you have some unprocessed data, a data in its native format, I will be storing the data in the raw folder or in a prefix starting with raw. Okay. So basically if I have to retrieve the data later like based on the logical names I have given here I will be able to identify I will be able to make my query faster and I will be able to relate from which particular prefix I have to fetch the data or retrieve the data. Okay. So this was the first thing I wanted to explain as a part of a discussion. Don't worry, we are going to talk about few more things as a part of this video lecture. In short, you can track the historical changes whenever you are talking about bucket versioning because what's the default behavior? By default, bucket versioning is disabled. So, you can enable it at the time of creating a bucket or even you can enable it after creating the bucket. So, don't worry as I'm going to show you a demo. I will show you the differences that what happens without bucket versioning and what happens after enabling the bucket versioning and scrolling down you can manage the encryption. Encryption is one of the security setting in the S3 bucket. If you want to explore different advanced settings which includes object lock a simple keyword to understand object lock is warm. write once and read many. Okay, but we are going to create everything default. We are going to have this bucket with the bucket name we have provided and rest of the settings I am keeping default. Okay. So what intelligent taring is it is designed for data with changing or unknown access patterns and everything comes up with a cost means here you can see we have a monitoring and auto tiering fee if you are going to have your data in intelligent taring. Okay. But here you are going to get the smart assistance means AWS is going to monitor your object and based on the access pattern your object will be automatically moved. So this particular section is to choose the storage class based on your workload based on your use case. So by default if you are not providing any storage class while storing the objects in S3 your object is stored in standard storage class. Okay. So let's answer this question without any further delay. So if I'm saying for 30 days immediate access what do you mean by that? It means you are going to store your object in standard storage class. Okay, let me make it bold. And if you are talking about 60 days infrequent access, you have two choices based on how you want your data to be available. Either you can move it to the infrequent access or one zone infrequent access. Let's say I'm choosing S3 standard infrequent access. Okay. Then for 180 days you want to archive your objects. If you want to archive your objects and glacier is a choice and after 365 days you want to delete your object it means you have to perform the delete operation. Okay. So if you are aware about the storage classes, you can take the right decision to move your objects from one storage class to another storage class whenever needed. Instead of keeping your object in a single storage class, you can leverage this feature of S3 and save the cost associated. So come back and we can upload the object using standard storage class for the time being. Let me do that. So I'm going to my cloud shell. You can click inside the terminal window if your cloud shell is asking you to reconnect. Okay. And what we will be doing, we will be using simple commands to upload the file. If you want to do a drag and drop from your local machine, you can do that too. It's your choice. So let me do one thing. I am going to upload the object. So first of all let me list out if I have something here better I can delete everything from here and I am going to create new files. So I have used rm command first of all to delete the two files as I had two files listed down. So I am deleting both the files in a single line. Next time when I am doing again listing in order to see what all objects are present I'm not getting back any response. So simply let me create a simple file. For example, eco I am understanding S3 features. Okay, something like this. And let's append this to a file called test.txt. It is done. If you will be executing ls again, this file is present here. Now I'm going to upload the object to S3 using a command called AWS S3 CP. CP is for copy. Provide the name of the object. I'm providing a name and then I have to provide the name of my bucket. So take the name of the bucket from your console. So let me first of all take these forward slashes. I'm going to provide this forward slash and then I'm going to take a bucket name. Okay, let's take this bucket name. Everybody go back, paste it here. If you want to save the object by a different name, even you can provide the object name. Otherwise, if you're not going to provide any object name, it will be stored with a name called test.txt. Let's press enter and see upload. How can you verify? Come back to your bucket. And if you're going to refresh it, you can see the object got uploaded. Okay. Now one thing all of you can see that storage class of this particular object is standard. So by default if you are not choosing any other storage class this object will be stored in standard storage class. Now one way to change the storage class of this object is go to actions and here you can edit the storage class. So you can change the storage class whenever you want. But this is something you are doing manually. Means whenever you have to change the storage class as per this requirement, you are selecting an object, you are editing a storage class and making a change. This is for single object. What about if you have so many objects stored in the S3 bucket? Are you going to change the storage class of all the object one by one? Is it an ideal way? If you think it is an ideal way, so let me tell you it's not. Then what is the solution for this particular problem? The solution is we are soon going to explore the concept of life cycle policies. So we are going to explore life cycle policies in order to move the objects automatically from one storage class to another storage class. Let me guide you. So if I am clicking on cancel button, I'm not changing the storage class for the time being. I have already shown you how to manually change the storage class. If I'm going to management part, you can see a life cycle configuration here. You can use life cycle rules to define actions you want S3 to take during an object's lifetime such as transitioning object to another class, archiving them or deleting them after a specified period of time. But if you have to use life cycle policies, you also need to understand versioning. Okay. So let's see the demo of versioning first and then we are going to create a life cycle route.