AWS EC2 Management with Python Boto3 – Create, Monitor & Delete EC2 Instances

Amazon Web Services is one of the world’s leading cloud service provider. among many services, Elastic Compute Cloud (EC2) allows users to rent virtual computers over the AWS.

In this tutorial, you will learn how to monitor, create and manage EC2 instances using Python. AWS has launched the Python library called Boto 3, which is a Python SDK for AWS resources. This tutorial will cover how to install, configure and get started with Boto3 library for your AWS account. This tutorial will also cover how to start, stop, monitor, create and terminate Amazon EC2 instances using Python programs.

Finally, the tutorial provides Python code to easily see EC2 instances and key information in tabular format and ways to query EC2 instances for dynamic access and monitoring.

AWS CLI

Installation

Now that you are going to access AWS resources pragmatically, it is important that AWS CLI is installed and configured.

You can install AWS CLI using pip.

pip install awscli

This would install the AWS CLI on your machine. Refer to docs for more information.

Note: On Linux or Mac, you might need to start with sudo to avoid permission issues.

Configuration

Now that AWS CLI is installed, you need to configure it to represent you and your AWS account.

In order to authenticate credentials for AWS account, head over to IAM Console on AWS.

Click to Users where you will be able to see all existing users.

Create a new user by clicking on Add User .

Give a username and Click on Programmatic access checkbox. Click Next on the bottom of the page.

If there are no groups already in existence, click on Create Group.

Type in “admin” in the Group name search bar and select AdministratorAccess from the results. This gives Admin access to this user and full access to all the AWS services and resources. Ideally, you want to restrict a user  / application to only the resources that it would need.  Click on Create group.

That would lead you to the Add Tags page. Let us skip that for now.

Review the information once, and click on Create user.

Click on Download .csv  and save your Access Key ID and Secret access key in a safe place. You will never be able to see this in future. However you would be able to regenerate / reset it. That is why it is important to back up the credential file and keep it safe.

Open Command Promt / Terminal and type

aws configure

It would prompt you for your AWS  Access Key ID ,  AWS Secret Access Key which you should have saved from the earlier screen. It also asks you for the Default region name. Since I live on the East Cost of the US, I have selected N.Virginia ( us-east-1 ) as my Default region.

Remember, never to share your Secret Access Key to anyone. That it the identity of your user and the application.

Pro Tip: If you were to ever put credentials in the project folder and then share on GitHub, put the credentials file in the ./gitignore  and your credentials will never be pushed.

You are all set with the installation and configuration.

 

Install Boto 3

Just like any other Python library, simply using pip or if you are using anaconda distribution, using conda install would install the Boto 3 library.

pip install boto3

Or to install the package with Conda, run this line:

conda install -c anaconda boto3

 

EC2 Client and Response

Now that the Boto3 Library is all set to use, let us start.

EC2 Client Introduction

Boto3’s client interface allows the user to query against the existing resources and minimal functionality to modify some aspects of these resources.

Getting  and Understanding Response

Let us get the information about all the EC2 instances running currently.

Gives the Output:

That is a long JSON output, which is hard to understand and read.

As you see, at this point, there are 2 EC2 instances running and 1 EC2 instance stopped. Boto3 is a JSON output model. Meaning, it would represent all the information in the JSON structure, which is very complex.

The function below grabs the necessary information and makes a pandas dataframe for us representing the EC2 instances.

Now, invoke the function:

 

Output of data_ec2:

From the JSON output, this dataframe is much more easy to understand and comprehend.

Here all the instances are queried and the output is stored into response. One can always use Filter and ask for specific EC2 instances. For example,

 

returns all the running EC2 instances.

 

Gives output:

 

Monitor EC2 Instances

With the client, you can turn on the CloutWatch monitoring for any EC2 instance by its InstanceId.

Output:

Go to the CloudWatch on the AWS console.

Note: Give 5-10 minutes for Cloud Watch to generate stats and have some movements to the metrics, if EC2 instances were not monitored previously.

To turn off the monitoring, simply use unmonitor()

 

Start EC2 Instances

Continuing the example in the tutorial, we have 3 instances. 2 of them are running while the other one is stopped.

The data of EC2 instances and their status is stored in the Pandas dataframe data_ec2

Start the stopped instance:

the code gives output:

As you can see, 'InstanceId': 'i-03a07cd4a54a74a6c' instance is now requested to start.

Update the status of EC2 instance in the application:

Output:

As we can see, all the instances are now running.

 

Stop EC2 Instances

Similarly, you can stop the running instances:

Output:

Updating the data_ec2 in our application:

Output:

 

Creating EC2 Instances

EC2 Resource Introduction

In order to create and terminate resources, use the resource interface from Boto3 library.

The syntax is very similar to the client interface.

 

AMI for AWS

Amazon Machine Image (AMI) is a software template that contains information about a particular Virtual Machine. In other words, it contains all the information about EBS, OS and Permissions. In order to create an EC2 with the library, you have to know which AMI to create the EC2 instance for.

Go to the EC2 Launch wizard and copy the AMI that you want to resource from AWS arsenal.

Deploy groups EC2 Instances

By design, boto3 library can deploy EC2 instances in groups.

The code creates 3 EC2 instances and stores their generated InstanceId in ec2_created list.

Get the latest EC2 deployment status.

Output:

As seen, now AWS has 3 new EC2 instances running, in addition to 3 stopped ones.

 

Terminate EC2 instances

Terminate the newly created EC2 instances.

Output:

The code requests AWS resources to terminate the 3 EC2 instances, leaving the other 3 instances stopped.

 

Complete Project Code

 

 

 

Rating: 4.8/5. From 5 votes.
Please wait...

Leave a Reply