preloader

Advanced AWS CLI with Jupyter Notebooks (Part 3)

illustrations illustrations illustrations illustrations illustrations illustrations illustrations
Advanced AWS CLI with Jupyter Notebooks (Part 3)

Published on Aug 22, 2021 by Raghuveer Varahagiri

Advanced AWS CLI with Jupyter Notebooks (Part 3) [Cross-account S3 Bucket Permissions & working with IAM using AWS CLI]

In a previous posts we saw how to setup Jupyter Notebooks to work with the AWS CLI natively. We also saw how to use AWS CI with Amazon S3 and bring some python and jupyter notebook magic to help us.

In this installment, we will work with IAM service and work through some examples. Let’s start.

Setup Notebook for AWS CLI

First we quickly go through the same steps as we did previously to get this notebook to work with AWS CLI – setting up enviornment variables, iPython shell aliases, and enabling auto-magic.

In []:
%env AWS_PROFILE = handsonaws-demo-a
Out[]:
env: AWS_PROFILE=handsonaws-demo-a
In []:
%rehashx
In []:
%automagic 1
Out[]:
Automagic is ON, % prefix IS NOT needed for line magics.
In []:
aws whoami
Out[]:
{
    "UserId": "AIDAXXREDACTEDXX52WEP",
    "Account": "9REDACTED5541",
    "Arn": "arn:aws:iam::9REDACTED5541:user/iamadmin"
}

As noted in previous post, even though we have enabled auto-magic here same commands will still work better only with the shell magic prefix !. Luckily if you run into one of those, the fix is as simple as inserting the ! prefix for that cell alone. You do not need to turn off auto-magic for the notebook or do anything else that impacts other cells.

Overview

In this demo scenario we have two users Alice and Bob in Account A and Account B respectively. These accounts have nothing in common - they are not part of any AWS Organizations setup.

In our use case Bob and Alice need to exchange some objects securely through S3, without requiring new IAM users or permanent access keys to be created or exchanged.

Architecture

Bob creates a bucket in Account B and applies a bucket policy that grants access to user Alice from Account B.

Also, Alice is part of s3customusers group in Account B that grants S3 permissions to members though a permissions policy named s3custompolicy.

We will go through these steps through the AWS CLI – using our favorite method of using Jupyter notebooks to do so.

IAM Basics

We first run through some basic commands of listing users, groups and their permissions.

One useful reference is the help documentation available through the CLI itself. The subcommand help displays the help documentation for each command and the list of all subcommands available under it.

Here is one place where the regular automagic doesn’t work (At least for me. The aws ... help command seems to internally invoke the cat command to display the help manual pages – which for some reason does not work from within my notebook setup on Win10, even when I alias cat command to type. Need to look deeper into this.)

The cell magic %%bash comes in handy here. The output is verbose – collapsing the output or enabling auto-scroll on the output makes it more manageable in the notebook context. You can scroll-up to this cell whenever you need to refer to it.

In []:
%%bash
aws s3 help
Out[]:

s3
^^

Description
***********

This section explains prominent concepts and notations in the set of
high-level S3 commands provided.


Available Commands
******************

* cp

* ls

* mb

* mv

* presign

* rb

* rm

* sync

* website
In []:
%%bash
aws iam help
Out[]:

iam
^^^


Description
***********

AWS Identity and Access Management (IAM) is a web service for securely
controlling access to AWS services. With IAM, you can centrally manage
users, security credentials such as access keys, and permissions that
control which AWS resources users and applications can access. For
more information about IAM, see AWS Identity and Access Management
(IAM) and the AWS Identity and Access Management User Guide .


Available Commands
******************
...

* create-access-key

* create-account-alias

* create-group

* create-instance-profile

* create-login-profile

* create-open-id-connect-provider

* create-policy
...

Fetching users and groups

List all users in the account as follows – remember we started in Account A. Currently there is only an Admin user and no Alice yet.

In []:
aws iam list-users
Out[]:
{
    "Users": [
        {
            "Path": "/",
            "UserName": "iamadmin",
            "UserId": "AIDAXXREDACTEDXX52WEP",
            "Arn": "arn:aws:iam::9REDACTED5541:user/iamadmin",
            "CreateDate": "2021-04-07T17:37:33+00:00",
            "PasswordLastUsed": "2021-08-14T15:01:57+00:00"
        }
    ]
}
In []:
aws iam list-users --output yaml
Out[]:
Users:
- Arn: arn:aws:iam::9REDACTED5541:user/iamadmin
    CreateDate: '2021-04-07T17:37:33+00:00'
    PasswordLastUsed: '2021-08-14T15:01:57+00:00'
    Path: /
    UserId: AIDAXXREDACTEDXX52WEP
    UserName: iamadmin

Basic operations of getting users and groups –

In []:
aws iam get-user
Out[]:
{
    "User": {
        "Path": "/",
        "UserName": "iamadmin",
        "UserId": "AIDAXXREDACTEDXX52WEP",
        "Arn": "arn:aws:iam::9REDACTED5541:user/iamadmin",
        "CreateDate": "2021-04-07T17:37:33+00:00",
        "PasswordLastUsed": "2021-08-14T15:01:57+00:00"
    }
}
In []:
aws iam list-groups
Out[]:
{
    "Groups": [
        {
            "Path": "/",
            "GroupName": "administrators",
            "GroupId": "AGPA6DW2X6I2ZC53IRU6D",
            "Arn": "arn:aws:iam::9REDACTED5541:group/administrators",
            "CreateDate": "2021-04-07T17:37:24+00:00"
        }
    ]
}
In []:
aws iam list-groups-for-user --user iamadmin
Out[]:
{
    "Groups": [
        {
            "Path": "/",
            "GroupName": "administrators",
            "GroupId": "AGPA6DW2X6I2ZC53IRU6D",
            "Arn": "arn:aws:iam::9REDACTED5541:group/administrators",
            "CreateDate": "2021-04-07T17:37:24+00:00"
        }
    ]
}
In []:
aws iam list-group-policies --group administrators
Out[]:
{
    "PolicyNames": []
}

There are no inline policies applied to this group. But there is a “managed policy” attached which we can retrieve as follows.

Note that retrieving the policy document requires a couple of steps – first you get the PolicyArn(s) attached, then you get policy metadata using that ARN, and finally fetch the current policy version using the default version id returned in the previous response.

Also note that the get-policy family of commands uses the ARN and not the name of the policy as a parameter to uniquely identify the policy resource. This is because there are ‘AWS-managed’ policies and ‘Customer-managed’ policies that could have the same policy name per-se.

ARN for AWS-managed policy –

arn:aws:iam::aws:policy/POLICYNAME

ARN for Customer-managed policy –

arn:aws:iam::ACCOUNT-ID:policy/POLICYNAME

As you can see, using just a policy-name instead of ARN would be ambiguous here. or Users, Roles and Groups, there is no such ambiguity. Therefore the corresponding get-user, get-role, and get-group commands use just the resource name as parameter instead of ARN.

In []:
aws iam get-policy --policy-arn "arn:aws:iam::aws:policy/AdministratorAccess"
Out[]:
{
    "Policy": {
        "PolicyName": "AdministratorAccess",
        "PolicyId": "ANPAIWMBCKSKIEE64ZLYK",
        "Arn": "arn:aws:iam::aws:policy/AdministratorAccess",
        "Path": "/",
        "DefaultVersionId": "v1",
        "AttachmentCount": 1,
        "PermissionsBoundaryUsageCount": 0,
        "IsAttachable": true,
        "Description": "Provides full access to AWS services and resources.",
        "CreateDate": "2015-02-06T18:39:46+00:00",
        "UpdateDate": "2015-02-06T18:39:46+00:00"
    }
}
In []:
aws iam list-attached-group-policies --group administrators
Out[]:
{
    "AttachedPolicies": [
        {
            "PolicyName": "AdministratorAccess",
            "PolicyArn": "arn:aws:iam::aws:policy/AdministratorAccess"
        }
    ]
}
In []:
aws iam get-policy-version --policy-arn "arn:aws:iam::aws:policy/AdministratorAccess" --version-id "v1" --query "PolicyVersion.Document"
Out[]:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "*",
            "Resource": "*"
        }
    ]
}

Creating Users, Groups, and Permissions Policies

Let us use a bit of python to quickly create a JSON document on similar lines – to serve as the permissions policy for the user group we are going to create for Alice.

For simplicity we will create a policy that grants full access to S3. In practice you would trim this down to the bare essentials needed (the principle of least privilege).

In []:
json_policy_s3fullaccess = """{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "*"
        }
    ]
}"""
In []:
print (json_policy_s3fullaccess)
Out[]:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "*"
        }
    ]
}

We write this string to a local text file and use that in the CLI command to create a customer-managed policy.

In []:
policy_filename = "policy.json"
with open(policy_filename, 'w') as the_file:
    the_file.write(json_policy_s3fullaccess)
In []:
!type {policy_filename}
Out[]:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "*"
        }
    ]
}
In []:
aws iam create-policy --policy-name "s3custom" --policy-document file://{policy_filename}
Out[]:
{
    "Policy": {
        "PolicyName": "s3custom",
        "PolicyId": "ANPA6DW2X6I2ZGDMX5FLD",
        "Arn": "arn:aws:iam::9REDACTED5541:policy/s3custom",
        "Path": "/",
        "DefaultVersionId": "v1",
        "AttachmentCount": 0,
        "PermissionsBoundaryUsageCount": 0,
        "IsAttachable": true,
        "CreateDate": "2021-08-14T17:06:41+00:00",
        "UpdateDate": "2021-08-14T17:06:41+00:00"
    }
}

We will create a user group that we will eventually attach the above policy to.

In []:
aws iam create-group --group-name "s3customusers"
Out[]:
{
    "Group": {
        "Path": "/",
        "GroupName": "s3customusers",
        "GroupId": "AGPA6DW2X6I23VD6AX2BM",
        "Arn": "arn:aws:iam::9REDACTED5541:group/s3customusers",
        "CreateDate": "2021-08-14T18:17:59+00:00"
    }
}
In []:
%%bash
aws iam create-user help
Out[]:
create-user
^^^^^^^^^^^


Description
***********

Creates a new IAM user for your AWS account.

...

Synopsis
********

        create-user
    [--path ]
    --user-name 
    [--permissions-boundary ]
    [--tags ]
    [--cli-input-json | --cli-input-yaml]
    [--generate-cli-skeleton ]
    [--cli-auto-prompt ]
In []:
aws iam create-user --user-name "alice"
Out[]:
{
    "User": {
        "Path": "/",
        "UserName": "alice",
        "UserId": "AIDAXXXREDACTEDXXXNUUE",
        "Arn": "arn:aws:iam::9REDACTED5541:user/alice",
        "CreateDate": "2021-08-14T18:29:23+00:00"
    }
}

Now that we have the new user Alice created we are ready to add this user to the group s3customusers that we created above.

In []:
%%bash
aws iam add-user-to-group help
Out[]:
 
add-user-to-group
^^^^^^^^^^^^^^^^^


Description
***********

Adds the specified user to the specified group.

...


Synopsis
********

        add-user-to-group
    --group-name 
    --user-name 
    [--cli-input-json | --cli-input-yaml]
    [--generate-cli-skeleton ]
    [--cli-auto-prompt ]
...
In []:
aws iam add-user-to-group --group-name "s3customusers" --user-name "alice"
In []:
aws iam list-groups-for-user --user-name "alice"
Out[]:
{
    "Groups": [
        {
            "Path": "/",
            "GroupName": "s3customusers",
            "GroupId": "AGPA6DW2X6I23VD6AX2BM",
            "Arn": "arn:aws:iam::9REDACTED5541:group/s3customusers",
            "CreateDate": "2021-08-14T18:17:59+00:00"
        }
    ]
}

We will also attach the s3custom IAM policy to the s3customusers group.

In []:
%%bash
aws iam attach-group-policy help
Out[]:
attach-group-policy
^^^^^^^^^^^^^^^^^^^
Description
***********

Attaches the specified managed policy to the specified IAM group.

...
Synopsis
********

        attach-group-policy
    --group-name 
    --policy-arn 
    [--cli-input-json | --cli-input-yaml]
    [--generate-cli-skeleton ]
    [--cli-auto-prompt ]
In []:
aws iam attach-group-policy --group-name "s3customusers" --policy-arn "arn:aws:iam::9REDACTED5541:policy/s3custom"

We can verify that the new policy got correctly attached as follows –

In []:
aws iam list-attached-group-policies --group-name "s3customusers"
Out[]:
{
    "AttachedPolicies": [
        {
            "PolicyName": "s3custom",
            "PolicyArn": "arn:aws:iam::9REDACTED5541:policy/s3custom"
        }
    ]
}
In []:
aws iam get-policy --policy-arn "arn:aws:iam::9REDACTED5541:policy/s3custom"
Out[]:
{
    "Policy": {
        "PolicyName": "s3custom",
        "PolicyId": "ANPA6DW2X6I2ZGDMX5FLD",
        "Arn": "arn:aws:iam::9REDACTED5541:policy/s3custom",
        "Path": "/",
        "DefaultVersionId": "v1",
        "AttachmentCount": 1,
        "PermissionsBoundaryUsageCount": 0,
        "IsAttachable": true,
        "CreateDate": "2021-08-14T17:06:41+00:00",
        "UpdateDate": "2021-08-14T17:06:41+00:00"
    }
}
In []:
aws iam get-policy-version --policy-arn "arn:aws:iam::9REDACTED5541:policy/s3custom" --version-id "v1" --query "PolicyVersion.Document"
Out[]:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "*"
        }
    ]
}

Creating Access Keys (and Configuring AWS Credentials for CLI)

We will also created a new access key for the new user Alice.

In []:
%%bash  
aws iam create-access-key help
Out[]:

create-access-key
^^^^^^^^^^^^^^^^^

Description
***********
Creates a new AWS secret access key and corresponding AWS access key
ID for the specified user. The default status for new keys is "Active"
.
...


Synopsis
********

        create-access-key
    [--user-name ]
    [--cli-input-json | --cli-input-yaml]
    [--generate-cli-skeleton ]
    [--cli-auto-prompt ]


...
In []:
aws iam create-access-key --user-name alice
Out[]:
{
    "AccessKey": {
        "UserName": "alice",
        "AccessKeyId": "AKIAXXREDACTEDXXLGGA",
        "Status": "Active",
        "SecretAccessKey": "ual+jz9HyxxxxxREDACTEDxxxxxxxdkPaevq",
        "CreateDate": "2021-08-15T05:30:47+00:00"
    }
}

I created a small helper function as shown below to append the new credentials programmatically to the AWS CLI’s local credentials file. this is not necessary especially for adding one or two users but can come in handly should you need to automate it for any reason.

In []:
import os
credentials_file = os.path.expanduser('~/.aws/credentials')
In []:
credentials_file
Out[]:
'C:\\Users\\USERNAME/.aws/credentials'
In []:
def format_aws_profile (profile,accesskey,secret): 
    return "[{profile}]\naws_access_key_id={accesskey}\naws_secret_access_key={secret}\n".format(profile=profile,accesskey=accesskey,secret=secret)

def write_aws_profile (filename,profile,accesskey,secret,comment=""):
    with open(filename, 'a') as f:
        f.write("\n# =====================================")
        if comment !="" : f.write("\n# "+comment)
        f.write("\n\n")
        f.write(format_aws_profile (profile,accesskey,secret))
In []:
write_aws_profile(credentials_file,"demo-alice","AKIAXXREDACTEDXXLGGA","ual+jz9HyxxxxxREDACTEDxxxxxxxdkPaevq","UserName : alice | Demo Account for IAM & S3 Cross-Account Access (S3 custom user in Account A)")
In []:
aws whoami --profile demo-alice
Out[]:
{
    "UserId": "AIDAXXXREDACTEDXXXNUUE",
    "Account": "9REDACTED5541",
    "Arn": "arn:aws:iam::9REDACTED5541:user/alice"
}
In []:
aws s3 ls --profile demo-alice
Out[]:

    2021-07-04 21:32:39 cf-templates-1bwhqi89f3g43-us-east-1
    2021-06-08 09:26:48 xxxxx-training-detailed-billing-reports-legacy
    2021-04-10 08:25:58 s3demo-20210410
    2021-07-13 20:12:37 s3demo-logs

Setup Bucket & Bucket Policy in Account B

Now we switch to Account B and setup the bucket and bucket policy.

In []:
%env AWS_PROFILE = demo-bob
Out[]:
env: AWS_PROFILE=demo-bob
In []:
aws whoami
Out[]:
{
    "UserId": "AIDAXXXREDACTEDXXXVMVJ",
    "Account": "53REDACTED1994",
    "Arn": "arn:aws:iam::53REDACTED1994:user/cliuser"
}
In []:
bucketname = "bobs-demo-bucket"
In []:
aws s3 mb s3://{bucketname} --region us-east-1
Out[]:
make_bucket: bobs-demo-bucket

We use python again to create the bucket policy text file locally and then use the CLI to apply it to the bucket.

In []:
json_bucketpolicy_caa = """{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Sid": "Cross-account S3 permissions on Bucket",
         "Effect": "Allow",
         "Principal": {
            "AWS": "arn:aws:iam::9REDACTED5541:user/alice"
         },
         "Action": [
            "s3:GetBucketLocation",
            "s3:ListBucket"
         ],
         "Resource": [
            "arn:aws:s3:::bobs-demo-bucket"
         ]
      },
      {
         "Sid": "Cross-account S3 permissions on Bucket objects",
         "Effect": "Allow",
         "Principal": {
            "AWS": "arn:aws:iam::9REDACTED5541:user/alice"
         },
         "Action": [
            "s3:GetObject",
            "s3:PutObject",
            "s3:PutObjectAcl"
         ],
         "Resource": [
            "arn:aws:s3:::bobs-demo-bucket/*"
         ]
      }
   ]
}"""
In []:
policy_filename = "bucketpolicy.json"
with open(policy_filename, 'w') as the_file:
    the_file.write(json_bucketpolicy_caa)
In []:
!type {policy_filename}
Out[]:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Cross-account S3 permissions on Bucket",
            "Effect": "Allow",
            "Principal": {
            "AWS": "arn:aws:iam::9REDACTED5541:user/alice"
            },
            "Action": [
            "s3:GetBucketLocation",
            "s3:ListBucket"
            ],
            "Resource": [
            "arn:aws:s3:::bobs-demo-bucket"
            ]
        },
        {
            "Sid": "Cross-account S3 permissions on Bucket objects",
            "Effect": "Allow",
            "Principal": {
            "AWS": "arn:aws:iam::9REDACTED5541:user/alice"
            },
            "Action": [
            "s3:GetObject",
            "s3:PutObject",
            "s3:PutObjectAcl"
            ],
            "Resource": [
            "arn:aws:s3:::bobs-demo-bucket/*"
            ]
        }
    ]
}
In []:
aws s3api put-bucket-policy --bucket {bucketname} --policy file://{policy_filename} --profile demo-bob
We place a test file in this new bucket

In []:
echo "helloworld" > test.txt
In []:
aws s3 cp test.txt s3://{bucketname}
Out[]:
Completed 15 Bytes/15 Bytes (9 Bytes/s) with 1 file(s) remaining
upload: .\test.txt to s3://bobs-demo-bucket/test.txt

Verify Cross-Account Access from Account A

We now switch to user Alice / account A

In []:
%env AWS_PROFILE = demo-alice
Out[]:
env: AWS_PROFILE=demo-alice
In []:
aws whoami
Out[]:
{
    "UserId": "AIDAXXXREDACTEDXXXNUUE",
    "Account": "9REDACTED5541",
    "Arn": "arn:aws:iam::9REDACTED5541:user/alice"
}
In []:
aws s3 ls
Out[]:
2021-07-04 21:32:39 cf-templates-1bwhqi89f3g43-us-east-1
2021-06-08 09:26:48 xxxxxx-training-detailed-billing-reports-legacy
2021-04-10 08:25:58 s3demo-20210410
2021-07-13 20:12:37 s3demo-logs
In []:
aws s3 ls s3://{bucketname}
Out[]:
2021-08-15 21:25:40         15 test.txt
In []:
echo "hello from the other side" > hello.txt
In []:
aws s3 cp hello.txt s3://{bucketname}
Out[]:
Completed 30 Bytes/30 Bytes (19 Bytes/s) with 1 file(s) remaining
upload: .\hello.txt to s3://bobs-demo-bucket/hello.txt

We verified that Alice is able to download as well as upload files to Bob’s bucket. However we still need to test it from Bob’s side.

Testing and Troubleshooting - Bucket & Object Ownership and ACLs

We access bucket as Bob from Account B (the bucket owner) and try to download the object uploaded by Alice.

In []:
aws s3 ls s3://{bucketname} --profile demo-bob
Out[]:
2021-08-15 22:15:47         30 hello.txt
2021-08-15 21:25:40         15 test.txt
In []:
aws s3 cp s3://{bucketname}/hello.txt ./download.txt --profile demo-bob
Out[]:
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

We see that even though Bob / Account B own the bucket they do not seem to have permissions to he newly uploaded object.

At this point Alice (A) still owns the objects they uploaded although Bob (B) owns the bucket itself.

For this to work as expected, Alice should also grant full ACL permissions to the bucket owner on the uploaded objects. This will allow Bob to download those objects.

In []:
aws s3api put-object-acl --bucket {bucketname} --key hello.txt --acl bucket-owner-full-control --profile demo-alice

The above command by Alice puts the bucket-owner-full-control ACL on the speciifc object hello.txt. Note that this operation is not available throuh the higher-level S3 command and you need to use the s3api command from CLI.

With this, Bob (B) should now be able to download that object.

In []:
aws s3 cp s3://{bucketname}/hello.txt ./download.txt --profile demo-bob
Out[]:
Completed 30 Bytes/30 Bytes (26 Bytes/s) with 1 file(s) remaining
download: s3://bobs-demo-bucket/hello.txt to .\download.txt
In []:
!type download.txt
Out[]:
"hello from the other side"

Advanced - Bucket Policy to Enforce bucket-owner-full-control ACL on Upload

We will now update the bucket policy to enforce this specific ACL on every upload. This accomplished by explicitly denying any put object operations that are missing this ACL.

In []:
json_bucketpolicy_caa = """{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Sid": "Cross-account S3 permissions on Bucket",
         "Effect": "Allow",
         "Principal": {
            "AWS": "arn:aws:iam::9REDACTED5541:user/alice"
         },
         "Action": [
            "s3:GetBucketLocation",
            "s3:ListBucket"
         ],
         "Resource": [
            "arn:aws:s3:::bobs-demo-bucket"
         ]
      },
      {
         "Sid": "Cross-account S3 permissions on Bucket objects GET",
         "Effect": "Allow",
         "Principal": {
            "AWS": "arn:aws:iam::9REDACTED5541:user/alice"
         },
         "Action": [
            "s3:GetObject"
         ],
         "Resource": [
            "arn:aws:s3:::bobs-demo-bucket/*"
         ]
      },
      {
         "Sid": "Cross-account S3 permissions on Bucket objects PUT with ACL condition",
         "Effect": "Allow",
         "Principal": {
            "AWS": "arn:aws:iam::9REDACTED5541:user/alice"
         },
         "Condition": {
                "StringEquals": {
                    "s3:x-amz-acl": "bucket-owner-full-control"
                }
            },
         "Action": [
            "s3:PutObject",
            "s3:PutObjectAcl"
         ],
         "Resource": [
            "arn:aws:s3:::bobs-demo-bucket/*"
         ]
      }
   ]
}"""
In []:
policy_filename = "bucketpolicy2.json"
with open(policy_filename, 'w') as the_file:
    the_file.write(json_bucketpolicy_caa)
In []:
!type {policy_filename}
Out[]:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Cross-account S3 permissions on Bucket",
            "Effect": "Allow",
            "Principal": {
            "AWS": "arn:aws:iam::9REDACTED5541:user/alice"
            },
            "Action": [
            "s3:GetBucketLocation",
            "s3:ListBucket"
            ],
            "Resource": [
            "arn:aws:s3:::bobs-demo-bucket"
            ]
        },
        {
            "Sid": "Cross-account S3 permissions on Bucket objects GET",
            "Effect": "Allow",
            "Principal": {
            "AWS": "arn:aws:iam::9REDACTED5541:user/alice"
            },
            "Action": [
            "s3:GetObject"
            ],
            "Resource": [
            "arn:aws:s3:::bobs-demo-bucket/*"
            ]
        },
        {
            "Sid": "Cross-account S3 permissions on Bucket objects PUT with ACL condition",
            "Effect": "Allow",
            "Principal": {
            "AWS": "arn:aws:iam::9REDACTED5541:user/alice"
            },
            "Condition": {
                "StringEquals": {
                    "s3:x-amz-acl": "bucket-owner-full-control"
                }
            },
            "Action": [
            "s3:PutObject",
            "s3:PutObjectAcl"
            ],
            "Resource": [
            "arn:aws:s3:::bobs-demo-bucket/*"
            ]
        }
    ]
}
In []:
aws s3api put-bucket-policy --bucket {bucketname} --policy file://{policy_filename} --profile demo-bob

Testing this new bucket policy, upload an object as Alice first WITHOUT supplying the ACL, and it should fail.

In []:
aws whoami
Out[]:
{
    "UserId": "AIDAXXXREDACTEDXXXNUUE",
    "Account": "9REDACTED5541",
    "Arn": "arn:aws:iam::9REDACTED5541:user/alice"
}
In []:
echo "I must've called a thousand times" > hello2.txt
In []:
aws s3 cp hello2.txt s3://{bucketname}
Out[]:
upload failed: .\hello2.txt to s3://bobs-demo-bucket/hello2.txt An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

Uploading the same object again as Alice but this time with the correct ACL applied (using the s3api command), you see successful upload.

In []:
aws s3api put-object --bucket {bucketname} --key hello2.txt --body hello2.txt --acl bucket-owner-full-control
Out[]:
{
    "ETag": "\"f0993f0398dd72768f46c69b23b1fec7\""
}
In []:
aws s3 ls s3://{bucketname}
Out[]:
2021-08-15 22:15:47         30 hello.txt
2021-08-15 22:24:06         38 hello2.txt
2021-08-15 21:25:40         15 test.txt

Verifying that Bob can download this object as well.

In []:
aws s3 ls s3://{bucketname} --profile demo-bob
Out[]:
2021-08-15 22:15:47         30 hello.txt
2021-08-15 22:24:06         38 hello2.txt
2021-08-15 21:25:40         15 test.txt
In []:
aws s3 cp s3://{bucketname}/hello2.txt ./download2.txt --profile demo-bob
Out[]:
Completed 38 Bytes/38 Bytes (36 Bytes/s) with 1 file(s) remaining
download: s3://bobs-demo-bucket/hello2.txt to .\download2.txt
In []:
!type download2.txt
Out[]:
"I must've called a thousand times"

That brings us to the end of this demo. We will look at a few other use-cases for AWS CLI and explore some other services in subsequent posts.

Additional Tips & Notes

  1. Useful extensions for JupyterLab
  1. GNU Utils for Win32 – https://sourceforge.net/projects/unxutils/
  2. Git-bash for Windows – https://gitforwindows.org/

References

  1. AWS CLI (v2) reference for IAM – https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iam/index.html
  2. S3 Cross-Account Permissions – https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-walkthroughs-managing-access-example2.html
Categories:
Miscellaneous
Security
Tags:
Jupyter
C L I
Developer Tools
Amazon S3
I a M