Simple Storage Service (S3) — Part 2
This is a continuation — https://codebykev.medium.com/100-days-of-devops-day-1-87cf192cbfd1 where I took a deeper look at s3 on AWS. Hopefully today I’ll have some practical stuff too.
I’ve had to go slower for a few days to get well with the first cold of 2020. Not COVID, just a headache and mucus drowned brain. 🎉🎉
Today’s topic
Amazon S3 — Simple Storage Service (continued)
Purpose
Unstructured data storage, high-level theory, and some practical stuff via the AWS console.
Accelerating transfer
Jumping right in, we left off yesterday talking about the various classes which offer trade-offs between durability, cost of storage, and in some cases like Glacier, real-time access.
One paid-for option is to accelerate large data transfer. You pay per gigabyte which costs less the more you do.
Practical time
For practical knowledge, I’m going to cover some of the ways you can create and manage S3 buckets:
- Via the AWS console (i assume you have set-up your user in IAMs by this point)
- Via the AWS CLI
- Terraform or another IAC tool
For the sake of time, I’ll just focus on the first one but the latter one I’ll cover when we start playing with Terraform. I currently use the CLI approach to manage a CICD pipeline with decent success.
AWS Console
Create a bucket
A picture is always worth a thousand words… moving pictures even more.
The creation process has changed over the past year or so, so if you’re viewing this later on do not be alarmed if they’ve reshuffled it again.
- Name and Region
- Configure Options
- Set Permissions
- Review
Naming a bucket as discussed requires a globally unique name. The zone you put it in will matter if you’re considering the latency between that region and your user base.
Configurations include versioning and logging options along with some encryption settings for objects in the bucket itself and tagging which is useful if you have themes or related ‘things’ you want to query later. Especially useful if you have a lot of content.
Permissions related to the ownership and access to the bucket. You can set here access for other accounts for example.
Finally, the review stage is as described, a place to review your choices and go back if you need to change anything.
Add an object
An object in a bucket can be ‘any’ file. Once you’ve created a bucket the URL for your bucket and the region it lives is:
https://s3.console.aws.amazon.com/s3/buckets/{bucketname}/?region={region}&tab=overview
An empty bucket would look like so
The creative flow of adding one or more files via the console is pretty easy
- Pick 1 or more files to upload
- Set permissions (similar setup rules to the bucket itself)
- Set properties are where the storage class options are set, along with encryption key settings, metadata, and tag data.
- The review is again where you sanity check before you commit to the settings.
Viewing objects properties and updating them
Selecting an object row lets you view the read-only settings.
Selecting its name lets you go to the object itself where you get more options
The key three sections (overview, properties, permissions) are access to the same things you could set during the wizard. The last one ‘select from’ is an uploading mechanism for settings
With S3 Select you can extract records from a single CSV, JSON or Parquet file using SQL expressions. S3 Select supports GZIP and BZIP2 compressed files and server-side encrypted files. You can use the console to extract up to 40 MB of records from source files up to 128 MB; to work with larger files or more records, use the API.Learn more
Update the object properties
Some notes for the different settings in the properties section of an object.
Deleting the objects
You can delete an object by selecting it, and using the actions menu OR you can access a nice context menu by right-clicking on it.
I think I’ll wrap up there… and finish off the s3 bucket set tomorrow. As usual, if I’ve messed anything up or made an incorrect assumption, take great pleasure in showing me where. Better I know then go on with a misconception.
