An indeterminate #days of DevOps— Day 2

Simple Storage Service (S3) — Part 2

This is a continuation — https://codebykev.medium.com/100-days-of-devops-day-1-87cf192cbfd1 where I took a deeper look at s3 on AWS. Hopefully today I’ll have some practical stuff too.

I’ve had to go slower for a few days to get well with the first cold of 2020. Not COVID, just a headache and mucus drowned brain. 🎉🎉

Today’s topic

Amazon S3 — Simple Storage Service (continued)

Purpose

Unstructured data storage, high-level theory, and some practical stuff via the AWS console.

Accelerating transfer

Jumping right in, we left off yesterday talking about the various classes which offer trade-offs between durability, cost of storage, and in some cases like Glacier, real-time access.

One paid-for option is to accelerate large data transfer. You pay per gigabyte which costs less the more you do.

Practical time

For practical knowledge, I’m going to cover some of the ways you can create and manage S3 buckets:

• Via the AWS console (i assume you have set-up your user in IAMs by this point)
• Via the AWS CLI
• Terraform or another IAC tool

For the sake of time, I’ll just focus on the first one but the latter one I’ll cover when we start playing with Terraform. I currently use the CLI approach to manage a CICD pipeline with decent success.

AWS Console

Create a bucket

A picture is always worth a thousand words… moving pictures even more.

The creation process has changed over the past year or so, so if you’re viewing this later on do not be alarmed if they’ve reshuffled it again.

1. Name and Region
2. Configure Options
3. Set Permissions
4. Review

Naming a bucket as discussed requires a globally unique name. The zone you put it in will matter if you’re considering the latency between that region and your user base.

Configurations include versioning and logging options along with some encryption settings for objects in the bucket itself and tagging which is useful if you have themes or related ‘things’ you want to query later. Especially useful if you have a lot of content.

Permissions related to the ownership and access to the bucket. You can set here access for other accounts for example.

Finally, the review stage is as described, a place to review your choices and go back if you need to change anything.

Add an object

An object in a bucket can be ‘any’ file. Once you’ve created a bucket the URL for your bucket and the region it lives is:

https://s3.console.aws.amazon.com/s3/buckets/{bucketname}/?region={region}&tab=overview

An empty bucket would look like so

New bucket in S3

The creative flow of adding one or more files via the console is pretty easy

1. Pick 1 or more files to upload
2. Set permissions (similar setup rules to the bucket itself)
3. Set properties are where the storage class options are set, along with encryption key settings, metadata, and tag data.
4. The review is again where you sanity check before you commit to the settings.

Viewing objects properties and updating them

preview the object by selecting the row not the name

Selecting an object row lets you view the read-only settings.
Selecting its name lets you go to the object itself where you get more options

Properties of the object can be accessed by the tab menu after you select the objects name

The key three sections (overview, properties, permissions) are access to the same things you could set during the wizard. The last one ‘select from’ is an uploading mechanism for settings

With S3 Select you can extract records from a single CSV, JSON or Parquet file using SQL expressions. S3 Select supports GZIP and BZIP2 compressed files and server-side encrypted files. You can use the console to extract up to 40 MB of records from source files up to 128 MB; to work with larger files or more records, use the API.Learn more

Update the object properties

Some notes for the different settings in the properties section of an object.

Deleting the objects

You can delete an object by selecting it, and using the actions menu OR you can access a nice context menu by right-clicking on it.

Delete the object here

I think I’ll wrap up there… and finish off the s3 bucket set tomorrow. As usual, if I’ve messed anything up or made an incorrect assumption, take great pleasure in showing me where. Better I know then go on with a misconception.