The Power of Jupyter Notebooks + AWS CLI

One of the best ways to learn the various AWS services hands-on is through the AWS CLI, without a doubt.

The CLI gives you a much deeper appreciation of how the services work under the hood than just clicking through the Management Console. You get to know the individual API methods and their parameters and outputs – which gets abstracted away from you in the web UI. This knowledge of API methods also comes in very handy when you dive into the odccassional development / troubleshooting / scripting you may need – say when you want to automate something using lambda or use the AWS SDK.

At the same time the sheer number of services that AWS has – and keeps adding to every day – can get daunting even for the most dedicated practitioners to keep up with.

As you start to use a new AWS service the initial learning curve can be steep if you choose to do it using the CLI. You spend a lot of time refering to documentation and experimenting with the syntax of the methods, their input parameters, parsing the output format to get the information you need, etc. Like may other things in life, unless you keep using this new knowledge fairly regularly from then onwards, you are likely to forget most of these details. Unlike your python code or bash scripts that encapsulate your learning and you can pick up from your last version, all your “work” on the CLI is ephemeral – as in there is no lasting record of the commands you issued, their sequence and inputs/outputs for your quick reference later.

The processing of having to re-learn some of the details and familiarize yourself with the nitty gritty all over again is often a drain on our productivity and poses a hurdle to our motivation to learn more.

I found that using Jupyter Notebooks (and JupyterLab) to execute the CLI commands overcomes some of these challenges. Having a saved history of all the commands you executed, ability to easily edit or re-run yyour previous commands in any order, ability to collapse verbose outputs, the ability to version and save your work, and very importantly the ability to document your code – all these come in super-handy when you are incrementally skilling up.

We know that Jupyter is not originally designed with the command line in mind. But there are ways in which you can get it to work with the AWS CLI. In this blog post we will walkthrough the steps to setup a Jupyter notebook for such use. I will also show you a few examples of using the CLI once setup this way. For the sake of simlicity I will limit these examples to basic AWS services like S3 and CloudFormation but you can easily extend it to all operations exposed by the CLI. (The notebook used in this blog post is available on github.)

Install and Configure the AWS CLI

Follow the steps in the official AWS docs to install and then configure the AWS CLI.

If you work with more than one AWS account, you can setup multiple named profiles – beyond the default profile.

You can edit the aws credentials file by hand to add additional information and comments to each profile you setup so you can manage them all better.

On a default Windows install the credentials file is stored at this path: C:\Users\<username>\.aws

If you want to be super careful about not executing CLI commands against the wrong account by mistake, I would suggest removing the deafult profile altogether. This forces you to explicitly and consciously supply the profile parameter to every CLI command you issue.

112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137


# =====================================
# xxxx xxxx PRODUCTION
# Account ID: xxxx 
# Account Alias: xxxx
# (Root User: xxxx@gmail.com)
# IAM User: xxxx
# Role: xxxx
# =====================================

[profilename1]
aws_access_key_id=####################
aws_secret_access_key=####################


# =====================================
# HANDSONAWS CLI DEMO
# Account ID: xxxxxxxxxxxxxx 
# Account Alias: xxxxxxxxxxx
# (Root User: xxxxxxxxxxx@gmail.com)
# IAM User: xxxxxxxxx
# Role: Admin
# =====================================

[handsonaws-demo]
aws_access_key_id=##########
aws_secret_access_key=#####################

Install Jupyter Notebooks / Jupyter Lab

If you have not already installed Jupyter notebooks, you can do so now by following the official guide from Anaconda – which is the most popular distribution of Python and comes with an easy way to manage your packages and data science tools.

To make things easier to launch, you can optionally add shortcuts to your Win10 start menu. I have a separate blogpost with downloadable batch files and icons that will walk you through these steps. Click here.

Start Menu Screenshot

Create a Jupyter Notebook for CLI

We will go through a number of steps that will make it easier for us to use CLI through a notebook. You may want to first create a new notebook and create a section at the top to house these setup steps.

Environment Variables & AWS Profile

First we setup the OS environment variable named AWS_PROFILE to point to the profile from the credentials file.

You can access the list of OS environment variables from within a notebook using the magic command %env

In [1]:

%env

Out[1]:

{'ALLUSERSPROFILE': 'C:\\ProgramData',
 'APPDATA': 'C:\\Users\\XXXXXXX\\AppData\\Roaming',
 'COMPUTERNAME': 'XXXXXX',
 'COMSPEC': 'C:\\WINDOWS\\system32\\cmd.exe',
 'DRIVERDATA': 'C:\\Windows\\System32\\Drivers\\DriverData',
 'HOMEDRIVE': 'C:',
 'HOMEPATH': '\\Users\\XXXXXXX',
 ...
 'PATH': 'C:\\WINDOWS\\system32;...;C:\\Program Files\\PuTTY\\',
 ... }

And to set a environment variable you can use the same command, but with parameters, as follows:

In [2]:

%env AWS_PROFILE=handsonaws-demo

Out[2]:

env: AWS_PROFILE=handsonaws-demo

As you might know you can issue shell commands from within the jupyter notebook using the shell magic prefix !

In [3]:

!echo Hello World >> test.txt

In [4]:

!type test.txt

Out[4]:

Hello World

This is very useful, for instance, when you handle local files like CloudFormation templates (YAML/JSON) or IAM policy documents (JSON)as inputs to CLI commands; or upload/download local objects to S3.

With this, you can verify that you have the CLI working with the correct credentials by calling the STS GetCallerIdentity method – from within the notebook as follows.

In [5]:

!aws sts get-caller-identity

Out[5]:

{
    "UserId": "AIDA6XXXXXXXXXXXXX2WEP",
    "Account": "97XXXXXXXX5541",
    "Arn": "arn:aws:iam::97XXXXXXX5541:user/iamadmin"
}

Shell aliases

Another handy magic command in Jupyter notebooks that makes lifes simpler working with shell commands in general and also AWS CLI specifically is %alias.

This lets you specify an alias for a shell command that you can invoke from the notebook using the generic line magic % – instead of the specific shell magic ! (why this distiction matters will become obvious shortly).

You can list the current aliases thus

In [6]:

%alias

Out[6]:

Total number of aliases: 8
[('copy', 'copy'),
 ('ddir', 'dir /ad /on'),
 ('echo', 'echo'),
 ('ldir', 'dir /ad /on'),
 ('ls', 'dir /on'),
 ('mkdir', 'mkdir'),
 ('ren', 'ren'),
 ('rmdir', 'rmdir')]

As you might notice this is a very minimal list and does not include all the recognized commands that you can execute from shell. Specifically we want to be able to invoke AWS CLI using the ‘aws’ command. While there is a way to specifically seup an alias for this command alone – there is a simpler and more robust way : using the %rehashx command.

In [7]:

%rehashx

This command rebuilds the table of aliases for all executables in the system PATH. This should now show a much longer list (depending on the tools installed in your system PATH) and specifically include the alias for aws.

In [8]:

%alias

Out[8]:

Total number of aliases: 995
[ ...
 ('anaconda', 'anaconda'),
 ('anaconda-navigator', 'anaconda-navigator'),
 ('anaconda-project', 'anaconda-project'),
 ...
 ('aws', 'aws'),
 ('bash', 'bash'),
 ...
 ('zstd', 'zstd')]

We will see how to use this to our advantage shortly.

AWS CLI aliases (optional)

In addition to the aliases functionality of Jupyter notebooks, the AWS command-line also has a similar ‘alias’ functionality – where you can substitute one command (within those supported by the AWS CLI) with another that you might prefer.

This is useful if you frequently use a specific command but that is too long to type.

Here is an example –

Create a text file called simply named alias (with no extension) in the AWS CLI path in your local system (C:\Users\<username>\.aws\cli\)

This file follows the INI syntax and you cna create a section and simply list the aliases as follows.

1
2
3


[toplevel]

whoami = sts get-caller-identity

Here in this example alias file, we are substituting the command sts get-caller-identity with the smaller and more intuitive string whoami. Post this, when you say aws whoami at the commandline, it translates to aws sts get-caller-identity.

As you might guess these aws aliases can get confusing if not used consciously – and beginners who want to learn the AWS CLI would be better advised to avoid these shortcuts before they first get a hang of the CLI commands themselves. Thereafter you can create your own shortcuts based on what commands you most frequently use and want to hack your productivity.

Just as an exercise we can verify and list the contents of this alias file for our own reference

In [9]:

!type C:\Users\username\.aws\cli\alias

Out[9]:

[toplevel]

whoami = sts get-caller-identity

From the notebook, using the jupyter alias for aws, you get the following output as you would expect. Note, once again, that we are using the line magic character % and not the shell magic character ! here.

In [10]:

%aws whoami

Out[10]:

{
    "UserId": "AIDA6XXXXXXXXXXXXX2WEP",
    "Account": "97XXXXXXXX5541",
    "Arn": "arn:aws:iam::97XXXXXXX5541:user/iamadmin"
}

iPython Auto Magic

Finally you can turn automagic ON in your notebook so you can invoke magic commands without using the line magic character %. This is very convenient if you plan to work extensively with the commandline – as we intend to do here.

In [11]:

%automagic 1

Out[11]:

Automagic is ON, % prefix IS NOT needed for line magics.

Note however that this doesnt work in 100% of the cases. Especially when you want your commands split across multiple lines – which is another great feature that works great when using notebooks.

Here is an example illustrating everything we have setup so far

In [12]:

aws whoami --query Arn

Out[12]:

"arn:aws:iam::97XXXXXX5541:user/iamadmin"

Using the AWS CLI from Jupyter

With this setup we are finally ready to dive into using the CLI from within our notebook.

List S3 buckets

In [13]:

aws s3 ls

Out[13]:

2021-07-04 21:32:39 cf-templates-1bwhqi89f3g43-us-east-1
2021-06-08 09:26:48 xxxxx-detailed-billing-reports-legacy
2021-04-10 08:25:58 s3demo-xxxx
2021-04-10 12:28:19 s3demo-xxxxyyyy
2021-07-13 20:12:37 s3demo-logs

Create an EC2 security group

In [14]:

aws ec2 create-security-group --group-name demo-sg --description "Hands-on AWS CLI Demo"

Out[14]:

{
    "GroupId": "sg-083f8xxxx71ec46a"
}

Describe the security group we just created

In [15]:

aws ec2 describe-security-groups --group-ids "sg-083f8xxxx71ec46a"

Out[15]:

{
    "SecurityGroups": [
        {
            "Description": "Hands-on AWS CLI Demo",
            "GroupName": "demo-sg",
            "IpPermissions": [],
            "OwnerId": "97XXXX5541",
            "GroupId": "sg-083f8xxxx71ec46a",
            "IpPermissionsEgress": [
                {
                    "IpProtocol": "-1",
                    "IpRanges": [
                        {
                            "CidrIp": "0.0.0.0/0"
                        }
                    ],
                    "Ipv6Ranges": [],
                    "PrefixListIds": [],
                    "UserIdGroupPairs": []
                }
            ],
            "VpcId": "vpc-c1exxxbc"
        }
    ]
}

Invoking Amazon Translate to translate a piece of text on the fly

In [16]:

aws translate translate-text --text "Hello World" --source-language-code en --target-language-code es

Out[16]:

{
    "TranslatedText": "Hola mundo",
    "SourceLanguageCode": "en",
    "TargetLanguageCode": "es"
}

Conclusion

As these examples illustrate, once you go through these steps of setup you can work on any AWS service supported through the CLI – and preserve the record of your interactions with the CLI along with the output for future reference.

We barely scratched the surface with the actual CLI commands here. As we dive deeper we will see more powerful ways in which we leverage the power of Jypyter notebooks and other developer tools and productivity hacks.

References

Aliases in AWS CLI - https://github.com/awslabs/awscli-aliases
Environment variables to configure the AWS CLI - https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html