VPC Endpoint to Access S3

Create an S3 Access IAM Role.

IAM roles are a secure way to grant permissions to entities that you trust. For example, an application code running on an EC2 instance that needs to perform actions on AWS resources like s3 might need an IAM role to do that.






1. Goto IAM -> Roles -> Create New Role

2. Select "EC2" and in "Permissions" select AmazonS3FullAccess.

3. Give a Role Name, Description and create a role.

This role helps us to access s3 from Ec2 instance.

Now create a t2 micro ubuntu EC2 instance from an AMI which has awscli ( AWS command line tools ) installed already in a private subnet with the IAM role we created.

The private subnet should be completely private, I mean the subnet should not even have a route to the internet through a NAT instance.

Now connect to the machine using ssh & key and since the machine has already awscli installed, you can try accessing the s3 like below.

$aws s3 ls

This will not work, fails with a timeout.

Why it fails even though we have an s3 access role assigned to that ec2 instance?
Because this instance is in private subnet in which we do not have access to internet and s3 does not reside inside any vpc and its endpoints are public in nature.
If you have to access s3 you have to send the request via internet only.

But how do I access s3 using a completely private machine then?
For that purpose, AWS provides s3 endpoints which can be used to connect a vpc with s3.

Currently, as we do not have a route to s3 through a vpc endpoint in the route table associated with our private subnet it failed.

Let's add a VPC Endpoint.

Select your vpc and s3 and continue.

Select the route table which is associated with your private subnet.

A rule with destination pl-id (com.amazonaws.us-west-2.s3) and a target with this endpoints' ID (e.g. vpce-12345678) will be added to the route tables you selected.

Now that we have a vpc endpoint, try to access the s3 from private ec2 instance again.

$ aws s3 ls

This will also fail with timeout because awscli by default will create request to global s3 url (s3.amazonaws.com)

Add an environment variable to your region.

$ export AWS_DEFAULT_REGION=us-west-2
$ aws s3 ls

This should list your buckets in us-west-2 region (vpc router will route your request to s3.us-east-1.amazonaws.com)

You have now successfully accessed s3 without internet from an ec2 instance residing in vpc's private subnet.

Create Dynamo DB Table using Cloud Formation Template

AWS CloudFormation simplifies provisioning and management on AWS. It gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion. To provision and configure stack resources, we must understand AWS CloudFormation templates, which are formatted text files in JSON or YAML. These templates describe the resources that we want to provision in your AWS CloudFormation stacks. We can use the AWS CloudFormation Designer or any text editor to create and save templates.

Let us how designer works in an another blog entry. For now you can play with designer if you wish.
https://console.aws.amazon.com/cloudformation/designer

Creating Dynamo DB Using AWS Cloud Formation Template.

1. We need to create a custom IAM policy "createstack". This policy hels us to execute aws createstack command.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1449904348000",
            "Effect": "Allow",
            "Action": [
                "cloudformation:CreateStack"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

>

2. Attach the above created policy to user through whom we need to create dynamo db table.

3.install awscli and configure
  a. sudo apt-get install awscli
  b. aws configure
        now give proper details like access key and region. This will create a config file in ~/.aws directory.
  c. test if configured properly using command 'aws s3 ls'. This should list all the s3 buckets if you have any.

4.Create a Cloud formation temlate like below , for creating dynamo db.

{
  "AWSTemplateFormatVersion" : "2010-09-09",
  "Resources" : {
    "myDynamoDBTable" : {
      "Type" : "AWS::DynamoDB::Table",
      "Properties" : {
        "AttributeDefinitions" : [
          {
            "AttributeName" : "Name",
            "AttributeType" : "S"   
          },
          {
            "AttributeName" : "Age",
            "AttributeType" : "S"
          }
        ],
        "KeySchema" : [
          {
            "AttributeName" : "Name",
            "KeyType" : "HASH"
          },
          {
            "AttributeName" : "Age",
            "KeyType" : "RANGE"
          }
        ],
        "ProvisionedThroughput" : {
          "ReadCapacityUnits" : "5",
          "WriteCapacityUnits" : "5"
        },
        "TableName" : "Person"
      }
    }
  }
}

5. Save the above file in a s3 bucket and copy the URL of this file.

6. Now on your command line , you can enter following command to create dynamodb table.

aws cloudformation create-stack --stack-name <stack_name> --template-url <s3_bucket_template_url>

----------------------------------------------------------------------------------------------------------------------------

Now, this is too much of manual process. This can be done using a python code as well. ( or java , node.js etc, lets see python for now ).

Here is how.

1. Create a config file like below. Save it as 'awsconfig'

[default]
aws_access_key_id = xxxxxxxx
aws_secret_access_key = xxxxxxxxxxxxxxxxxxx
region = us-west-2

2. Create a shell script like below.

sudo apt-get install -fy --force-yes awscli python
sudo curl -s 'https://bootstrap.pypa.io/get-pip.py' | python2.7 && pip install boto awscli
sudo mkdir ~/.aws
cp awsconfig ~/.aws/config

The above script should install all the required tools for python to create a dynamo db. Allow execution permission and execute the script.

3. Create a python file using following code.

from __future__ import print_function # Python 2/3 compatibility
import boto3

dynamodb = boto3.resource('dynamodb', region_name='us-west-2')


table = dynamodb.create_table(
    TableName='Person',
    KeySchema=[
        {
            'AttributeName': 'name',
            'KeyType': 'HASH'  #Partition key
        },
        {
            'AttributeName': 'age',
            'KeyType': 'RANGE'  #Sort key
        }
    ],
    AttributeDefinitions=[
        {
            'AttributeName': 'name',
            'AttributeType': 'S'
        },
        {
            'AttributeName': 'age',
            'AttributeType': 'N'
        },

    ],
    ProvisionedThroughput={
        'ReadCapacityUnits': 5,
        'WriteCapacityUnits': 5
    }
)

print("Table status:", table.table_status)

4. Execute python code .

  python create_dynamo_table.py

5. Check your Dynamo DB service , table called person should have been created.