Amazon Web Services is one of the world’s leading cloud service provider. among many services, Elastic Compute Cloud (EC2) allows users to rent virtual computers over the AWS.
In this tutorial, you will learn how to monitor, create and manage EC2 instances using Python. AWS has launched the Python library called Boto 3, which is a Python SDK for AWS resources. This tutorial will cover how to install, configure and get started with Boto3 library for your AWS account. This tutorial will also cover how to start, stop, monitor, create and terminate Amazon EC2 instances using Python programs.
Finally, the tutorial provides Python code to easily see EC2 instances and key information in tabular format and ways to query EC2 instances for dynamic access and monitoring.
AWS CLI
Installation
Now that you are going to access AWS resources pragmatically, it is important that AWS CLI is installed and configured.
You can install AWS CLI using pip.
pip install awscli
This would install the AWS CLI on your machine. Refer to docs for more information.
Note: On Linux or Mac, you might need to start with sudo to avoid permission issues.
Configuration
Now that AWS CLI is installed, you need to configure it to represent you and your AWS account.
In order to authenticate credentials for AWS account, head over to IAM Console on AWS.
Click to Users where you will be able to see all existing users.
Create a new user by clicking on Add User .
Give a username and Click on Programmatic access checkbox. Click Next on the bottom of the page.
If there are no groups already in existence, click on Create Group.
Type in “admin” in the Group name search bar and select AdministratorAccess from the results. This gives Admin access to this user and full access to all the AWS services and resources. Ideally, you want to restrict a user / application to only the resources that it would need. Click on Create group.
That would lead you to the Add Tags page. Let us skip that for now.
Review the information once, and click on Create user.
Click on Download .csv and save your Access Key ID and Secret access key in a safe place. You will never be able to see this in future. However you would be able to regenerate / reset it. That is why it is important to back up the credential file and keep it safe.
Open Command Promt / Terminal and type
aws configure
It would prompt you for your AWS Access Key ID , AWS Secret Access Key which you should have saved from the earlier screen. It also asks you for the Default region name. Since I live on the East Cost of the US, I have selected N.Virginia ( us-east-1 ) as my Default region.
Remember, never to share your Secret Access Key to anyone. That it the identity of your user and the application.
Pro Tip: If you were to ever put credentials in the project folder and then share on GitHub, put the credentials file in the ./gitignore and your credentials will never be pushed.
You are all set with the installation and configuration.
Install Boto 3
Just like any other Python library, simply using pip or if you are using anaconda distribution, using conda install would install the Boto 3 library.
pip install boto3
Or to install the package with Conda, run this line:
conda install -c anaconda boto3
EC2 Client and Response
Now that the Boto3 Library is all set to use, let us start.
EC2 Client Introduction
Boto3’s client interface allows the user to query against the existing resources and minimal functionality to modify some aspects of these resources.
|
1 2 3 4 |
import boto3 ec2 = boto3.client('ec2') |
Getting and Understanding Response
Let us get the information about all the EC2 instances running currently.
|
1 2 3 4 |
response = ec2.describe_instances() print(response) |
Gives the Output:
That is a long JSON output, which is hard to understand and read.
As you see, at this point, there are 2 EC2 instances running and 1 EC2 instance stopped. Boto3 is a JSON output model. Meaning, it would represent all the information in the JSON structure, which is very complex.
The function below grabs the necessary information and makes a pandas dataframe for us representing the EC2 instances.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
import pandas as pd def get_ec2_status(response): av_zones, instance_ids, state_names= [], [], [] for res in response['Reservations']: for ins in res['Instances']: av_zones.append(ins['Placement']['AvailabilityZone']) instance_ids.append(ins['InstanceId']) state_names.append(ins['State']['Name']) return pd.DataFrame({ 'InstanceId': instance_ids, 'Availibility Zone': av_zones, 'State': state_names }) |
Now, invoke the function:
|
1 2 |
data_ec2 = get_ec2_status(response) |
Output of data_ec2:
From the JSON output, this dataframe is much more easy to understand and comprehend.
Here all the instances are queried and the output is stored into response. One can always use Filter and ask for specific EC2 instances. For example,
|
1 2 |
response_running = ec2.describe_instances(Filters=[{'Name' : 'instance-state-name','Values' : ['running']}]) |
returns all the running EC2 instances.
|
1 2 |
data_ec2_running = get_ec2_status(response_running) |
Gives output:
Monitor EC2 Instances
With the client, you can turn on the CloutWatch monitoring for any EC2 instance by its InstanceId.
|
1 2 |
ec2.monitor_instances(InstanceIds = ['i-0fc36bf8fef1fe9fe']) |
Output:
|
1 2 3 4 5 6 7 8 9 10 |
{'InstanceMonitorings': [{'InstanceId': 'i-0fc36bf8fef1fe9fe', 'Monitoring': {'State': 'pending'}}], 'ResponseMetadata': {'HTTPHeaders': {'content-length': '424', 'content-type': 'text/xml;charset=UTF-8', 'date': 'Mon, 18 Feb 2019 21:59:28 GMT', 'server': 'AmazonEC2'}, 'HTTPStatusCode': 200, 'RequestId': 'd1b762e1-3eca-4621-96e0-1ae73d592553', 'RetryAttempts': 0}} |
Go to the CloudWatch on the AWS console.
Note: Give 5-10 minutes for Cloud Watch to generate stats and have some movements to the metrics, if EC2 instances were not monitored previously.
To turn off the monitoring, simply use unmonitor()
|
1 2 |
ec2.unmonitor_instances(InstanceIds = ['i-0fc36bf8fef1fe9fe']) |
Start EC2 Instances
Continuing the example in the tutorial, we have 3 instances. 2 of them are running while the other one is stopped.
The data of EC2 instances and their status is stored in the Pandas dataframe data_ec2
Start the stopped instance:
|
1 2 |
ec2.start_instances(InstanceIds=list(data_ec2.loc[data_ec2['State'] == 'stopped', 'InstanceId'])) |
the code gives output:
|
1 2 3 4 5 6 7 8 9 10 11 |
{'ResponseMetadata': {'HTTPHeaders': {'content-length': '579', 'content-type': 'text/xml;charset=UTF-8', 'date': 'Mon, 18 Feb 2019 22:18:59 GMT', 'server': 'AmazonEC2'}, 'HTTPStatusCode': 200, 'RequestId': '05353ee5-11bc-4d5e-a760-d24ca94d272e', 'RetryAttempts': 0}, 'StartingInstances': [{'CurrentState': {'Code': 0, 'Name': 'pending'}, 'InstanceId': 'i-03a07cd4a54a74a6c', 'PreviousState': {'Code': 80, 'Name': 'stopped'}}]} |
As you can see, 'InstanceId': 'i-03a07cd4a54a74a6c' instance is now requested to start.
Update the status of EC2 instance in the application:
|
1 2 3 4 5 |
response = ec2.describe_instances() data_ec2 = get_ec2_status(response) print(data_ec2) |
Output:
As we can see, all the instances are now running.
Stop EC2 Instances
Similarly, you can stop the running instances:
|
1 2 |
ec2.stop_instances(InstanceIds=list(data_ec2.loc[data_ec2['State'] == 'running', 'InstanceId'])) |
Output:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
{'ResponseMetadata': {'HTTPHeaders': {'content-length': '1257', 'content-type': 'text/xml;charset=UTF-8', 'date': 'Mon, 18 Feb 2019 22:25:58 GMT', 'server': 'AmazonEC2'}, 'HTTPStatusCode': 200, 'RequestId': '51426fa8-c4ac-4463-84af-b5fd779b905b', 'RetryAttempts': 0}, 'StoppingInstances': [{'CurrentState': {'Code': 64, 'Name': 'stopping'}, 'InstanceId': 'i-0c702a9f1233db36b', 'PreviousState': {'Code': 16, 'Name': 'running'}}, {'CurrentState': {'Code': 64, 'Name': 'stopping'}, 'InstanceId': 'i-03a07cd4a54a74a6c', 'PreviousState': {'Code': 16, 'Name': 'running'}}, {'CurrentState': {'Code': 64, 'Name': 'stopping'}, 'InstanceId': 'i-0fc36bf8fef1fe9fe', 'PreviousState': {'Code': 16, 'Name': 'running'}}]} |
Updating the data_ec2 in our application:
|
1 2 3 4 |
response = ec2.describe_instances() data_ec2 = get_ec2_status(response) print(data_ec2) |
Output:
Creating EC2 Instances
EC2 Resource Introduction
In order to create and terminate resources, use the resource interface from Boto3 library.
The syntax is very similar to the client interface.
|
1 2 |
ec2_resource = boto3.resource('ec2') |
AMI for AWS
Amazon Machine Image (AMI) is a software template that contains information about a particular Virtual Machine. In other words, it contains all the information about EBS, OS and Permissions. In order to create an EC2 with the library, you have to know which AMI to create the EC2 instance for.
Go to the EC2 Launch wizard and copy the AMI that you want to resource from AWS arsenal.
Deploy groups EC2 Instances
By design, boto3 library can deploy EC2 instances in groups.
|
1 2 3 4 |
list_ec2 = ec2_resource.create_instances(ImageId='ami-0a483c9901a5df3b7', MinCount=1, MaxCount=3) ec2_created = [ec2_instance.id for ec2_instance in list_ec2] |
The code creates 3 EC2 instances and stores their generated InstanceId in ec2_created list.
Get the latest EC2 deployment status.
|
1 2 3 |
data_ec2 = get_ec2_status(response) print(data_ec2) |
Output:
As seen, now AWS has 3 new EC2 instances running, in addition to 3 stopped ones.
Terminate EC2 instances
Terminate the newly created EC2 instances.
|
1 2 |
ec2.terminate_instances(InstanceIds=ec2_created) |
Output:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
{'ResponseMetadata': {'HTTPHeaders': {'content-type': 'text/xml;charset=UTF-8', 'date': 'Mon, 18 Feb 2019 22:47:57 GMT', 'server': 'AmazonEC2', 'transfer-encoding': 'chunked', 'vary': 'Accept-Encoding'}, 'HTTPStatusCode': 200, 'RequestId': 'ca449800-58e0-47e9-b41f-ba5424d298b8', 'RetryAttempts': 0}, 'TerminatingInstances': [{'CurrentState': {'Code': 32, 'Name': 'shutting-down'}, 'InstanceId': 'i-06dd66a3e980fe96e', 'PreviousState': {'Code': 16, 'Name': 'running'}}, {'CurrentState': {'Code': 32, 'Name': 'shutting-down'}, 'InstanceId': 'i-00d6f877aa0a91ceb', 'PreviousState': {'Code': 16, 'Name': 'running'}}, {'CurrentState': {'Code': 32, 'Name': 'shutting-down'}, 'InstanceId': 'i-03a5c222e1681e385', 'PreviousState': {'Code': 16, 'Name': 'running'}}]} |
The code requests AWS resources to terminate the 3 EC2 instances, leaving the other 3 instances stopped.
Complete Project Code
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 |
import boto3 import pprint as pprint import pandas as pd # EC2 client and Response ec2 = boto3.client('ec2') response = ec2.describe_instances() print(response) # Function for status reporting def get_ec2_status(response): av_zones, instance_ids, state_names= [], [], [] for res in response['Reservations']: for ins in res['Instances']: av_zones.append(ins['Placement']['AvailabilityZone']) instance_ids.append(ins['InstanceId']) state_names.append(ins['State']['Name']) return pd.DataFrame({ 'InstanceId': instance_ids, 'Availibility Zone': av_zones, 'State': state_names }) response = ec2.describe_instances() data_ec2 = get_ec2_status(response) print(data_ec2) # EC2 Monitoring response_running = ec2.describe_instances(Filters=[{'Name' : 'instance-state-name','Values' : ['running']}]) data_ec2_running = get_ec2_status(response) print(data_ec2_running) ec2.monitor_instances(InstanceIds = ['i-0fc36bf8fef1fe9fe']) # Start EC2 Instances ec2.start_instances(InstanceIds=list(data_ec2.loc[data_ec2['State'] == 'stopped', 'InstanceId'])) response = ec2.describe_instances() data_ec2 = get_ec2_status(response) print(data_ec2) # Stop EC2 Instances ec2.stop_instances(InstanceIds=list(data_ec2.loc[data_ec2['State'] == 'running', 'InstanceId'])) response = ec2.describe_instances() data_ec2 = get_ec2_status(response) print(data_ec2) # Create EC2 Instances ec2_resource = boto3.resource('ec2') list_ec2 = ec2_resource.create_instances(ImageId='ami-0a483c9901a5df3b7', MinCount=1, MaxCount=3) ec2_created = [ec2_instance.id for ec2_instance in list_ec2] print(ec2_created) response = ec2.describe_instances() data_ec2 = get_ec2_status(response) print(data_ec2) # Terminate EC2 Instances ec2_client.terminate_instances(InstanceIds=ec2_created) |
I am a Data Scientist in the Manufacturing / IoT domain and a ML enthusiast. I like to work on data analysis and data infrastructure projects as well. Looking forward to connect.