I am trying to create a Python program using boto3 that creates an Amazon AutoScaling group. Define scale up and scale down policies and corresponding CloudWatch alarms. Supply the startup script created in previous step while launching a new instance from the program. Using a security group port 80 open,here is the program ,
# Check if the user has the Access & Secret key configuredimport boto3from boto3 import Sessionsession = Session()credentials = session.get_credentials()current_credentials = credentials.get_frozen_credentials()# Break & Exit if any of the key is not presentif current_credentials.access_key is None:print("Access Key missing, use `aws configure` to setup")exit()if current_credentials.secret_key is None:print("Secret Key missing, use `aws configure` to setup")exit()# VPC design for multi az deploymentsglobalVars = {}globalVars['REGION_NAME'] = "ap-south-1"globalVars['AZ1'] = "ap-south-1a"globalVars['AZ2'] = "ap-south-1b"globalVars['CIDRange'] = "10.240.0.0/23"globalVars['az1_pvtsubnet_CIDRange'] = "10.240.0.0/25"globalVars['az1_pubsubnet_CIDRange'] = "10.240.0.128/26"globalVars['az1_sparesubnet_CIDRange'] = "10.240.0.192/26"globalVars['az2_pvtsubnet_CIDRange'] = "10.240.1.0/25"globalVars['az2_pubsubnet_CIDRange'] = "10.240.1.128/26"globalVars['az2_sparesubnet_CIDRange'] = "10.240.1.192/26"globalVars['Project'] = { 'Key': 'Name', 'Value': 'test1'}globalVars['tags'] = [{'Key': 'Owner', 'Value': 'test1'},{'Key': 'Environment', 'Value': 'Test'},{'Key': 'Department', 'Value': 'TestD'}]# EC2 ParametersglobalVars['EC2-Amazon-AMI-ID'] = "ami-d783a9b8"globalVars['EC2-InstanceType'] = "t2.micro"globalVars['EC2-KeyName'] = "datastructutre key"# AutoScaling ParametersglobalVars['ASG-LaunchConfigName'] = "ASG-Demo-LaunchConfig"globalVars['ASG-AutoScalingGroupName'] = "ASG-Demo-AutoScalingGrp"# Creating a VPC, Subnet, and Gatewayec2 = boto3.resource('ec2', region_name=globalVars['REGION_NAME'])ec2Client = boto3.client('ec2', region_name=globalVars['REGION_NAME'])vpc = ec2.create_vpc(CidrBlock=globalVars['CIDRange'])asgClient = boto3.client('autoscaling', region_name=globalVars['REGION_NAME'])rds = boto3.client('rds', region_name=globalVars['REGION_NAME'])# AZ1 Subnetsaz1_pvtsubnet = vpc.create_subnet(CidrBlock=globalVars['az1_pvtsubnet_CIDRange'], AvailabilityZone=globalVars['AZ1'])az1_pubsubnet = vpc.create_subnet(CidrBlock=globalVars['az1_pubsubnet_CIDRange'], AvailabilityZone=globalVars['AZ1'])az1_sparesubnet = vpc.create_subnet(CidrBlock=globalVars['az1_sparesubnet_CIDRange'], AvailabilityZone=globalVars['AZ1'])# AZ2 Subnetaz2_pvtsubnet = vpc.create_subnet(CidrBlock=globalVars['az2_pvtsubnet_CIDRange'], AvailabilityZone=globalVars['AZ2'])az2_pubsubnet = vpc.create_subnet(CidrBlock=globalVars['az2_pubsubnet_CIDRange'], AvailabilityZone=globalVars['AZ2'])az2_sparesubnet = vpc.create_subnet(CidrBlock=globalVars['az2_sparesubnet_CIDRange'], AvailabilityZone=globalVars['AZ2'])# Enable DNS Hostnames in the VPCvpc.modify_attribute(EnableDnsSupport={'Value': True})vpc.modify_attribute(EnableDnsHostnames={'Value': True})# Create the Internet Gatway & Attach to the VPCintGateway = ec2.create_internet_gateway()intGateway.attach_to_vpc(VpcId=vpc.id)# Create another route table for Public & Private trafficrouteTable = ec2.create_route_table(VpcId=vpc.id)rtbAssn=[]rtbAssn.append(routeTable.associate_with_subnet(SubnetId=az1_pubsubnet.id))rtbAssn.append(routeTable.associate_with_subnet(SubnetId=az1_pvtsubnet.id))rtbAssn.append(routeTable.associate_with_subnet(SubnetId=az2_pubsubnet.id))rtbAssn.append(routeTable.associate_with_subnet(SubnetId=az2_pvtsubnet.id))# Create a route for internet traffic to flow outintRoute = ec2Client.create_route(RouteTableId=routeTable.id, DestinationCidrBlock='0.0.0.0/0', GatewayId=intGateway.id)# Tag the resourcesvpc.create_tags (Tags=globalVars['tags'])az1_pvtsubnet.create_tags (Tags=globalVars['tags'])az1_pubsubnet.create_tags (Tags=globalVars['tags'])az1_sparesubnet.create_tags(Tags=globalVars['tags'])az2_pvtsubnet.create_tags (Tags=globalVars['tags'])az2_pubsubnet.create_tags (Tags=globalVars['tags'])az2_sparesubnet.create_tags(Tags=globalVars['tags'])intGateway.create_tags (Tags=globalVars['tags'])routeTable.create_tags (Tags=globalVars['tags'])vpc.create_tags (Tags=[{'Key': 'Name', 'Value': globalVars['Project']['Value'] + '-vpc'}])az1_pvtsubnet.create_tags (Tags=[{'Key': 'Name', 'Value': globalVars['Project']['Value'] + '-az1-private-subnet'}])az1_pubsubnet.create_tags (Tags=[{'Key': 'Name', 'Value': globalVars['Project']['Value'] + '-az1-public-subnet'}])az1_sparesubnet.create_tags(Tags=[{'Key': 'Name', 'Value': globalVars['Project']['Value'] + '-az1-spare-subnet'}])az2_pvtsubnet.create_tags (Tags=[{'Key': 'Name', 'Value': globalVars['Project']['Value'] + '-az2-private-subnet'}])az2_pubsubnet.create_tags (Tags=[{'Key': 'Name', 'Value': globalVars['Project']['Value'] + '-az2-public-subnet'}])az2_sparesubnet.create_tags(Tags=[{'Key': 'Name', 'Value': globalVars['Project']['Value'] + '-az2-spare-subnet'}])intGateway.create_tags (Tags=[{'Key': 'Name', 'Value': globalVars['Project']['Value'] + '-igw'}])routeTable.create_tags (Tags=[{'Key': 'Name', 'Value': globalVars['Project']['Value'] + '-rtb'}])# Let create the Public & Private Security GroupselbSecGrp = ec2.create_security_group(DryRun=False,GroupName='elbSecGrp',Description='ElasticLoadBalancer_Security_Group',VpcId=vpc.id)pubSecGrp = ec2.create_security_group(DryRun=False,GroupName='pubSecGrp',Description='Public_Security_Group',VpcId=vpc.id)pvtSecGrp = ec2.create_security_group(DryRun=False,GroupName='pvtSecGrp',Description='Private_Security_Group',VpcId=vpc.id)elbSecGrp.create_tags(Tags=globalVars['tags'])pubSecGrp.create_tags(Tags=globalVars['tags'])pvtSecGrp.create_tags(Tags=globalVars['tags'])elbSecGrp.create_tags(Tags=[{'Key': 'Name', 'Value': globalVars['Project']['Value'] + '-elb-security-group'}])pubSecGrp.create_tags(Tags=[{'Key': 'Name', 'Value': globalVars['Project']['Value'] + '-public-security-group'}])pvtSecGrp.create_tags(Tags=[{'Key': 'Name', 'Value': globalVars['Project']['Value'] + '-private-security-group'}])# Add a rule that allows inbound SSH, HTTP, HTTPS traffic ( from any source )ec2Client.authorize_security_group_ingress(GroupId=elbSecGrp.id,IpProtocol='tcp',FromPort=80,ToPort=80,CidrIp='0.0.0.0/0')# Allow Public Security Group to receive traffic from ELB Security groupec2Client.authorize_security_group_ingress(GroupId=pubSecGrp.id,IpPermissions=[{'IpProtocol': 'tcp','FromPort': 80,'ToPort': 80,'UserIdGroupPairs': [{'GroupId': elbSecGrp.id}]}])# Allow Private Security Group to receive traffic from Application Security groupec2Client.authorize_security_group_ingress(GroupId=pvtSecGrp.id,IpPermissions=[{'IpProtocol': 'tcp','FromPort': 3306,'ToPort': 3306,'UserIdGroupPairs': [{'GroupId': pubSecGrp.id}]}])ec2Client.authorize_security_group_ingress(GroupId=pubSecGrp.id,IpProtocol='tcp',FromPort=80,ToPort=80,CidrIp='0.0.0.0/0')ec2Client.authorize_security_group_ingress(GroupId=pubSecGrp.id,IpProtocol='tcp',FromPort=443,ToPort=443,CidrIp='0.0.0.0/0')ec2Client.authorize_security_group_ingress(GroupId=pubSecGrp.id,IpProtocol='tcp',FromPort=22,ToPort=22,CidrIp='0.0.0.0/0')# The user defined code to install WebServer & Configure themuserDataCode = """#!/bin/bashset -e -x# Setting up the HTTP server yum install -y httpd service httpd startchkconfig httpd ongroupadd wwwusermod -a -G www ec2-usercd /var/www/# Set the permissionschown -R root:www /var/wwwchmod 2775 /var/wwwfind /var/www -type d -exec chmod 2775 {} +find /var/www -type f -exec chmod 0664 {} +# SE Linux permissive# setsebool -P httpd_can_network_connect=1service httpd restart# Remove below file after testingecho "<?php phpinfo(); ?>" > /var/www/html/phptestinfo.php"""# Create the Public Instance##### **DeviceIndex**:The network interface's position in the attachment order. For example, the first attached network interface has a DeviceIndex of 0instanceLst = ec2.create_instances(ImageId=globalVars['EC2-Amazon-AMI-ID'],MinCount=1,MaxCount=1,KeyName="datastructutre key",UserData=userDataCode,InstanceType=globalVars['EC2-InstanceType'],NetworkInterfaces=[{'SubnetId': az1_pubsubnet.id,'Groups': [pubSecGrp.id],'DeviceIndex': 0,'DeleteOnTermination': True,'AssociatePublicIpAddress': True,}])# Create the Launch Configuration# InstanceId = 'string'asgLaunchConfig = asgClient.create_launch_configuration(LaunchConfigurationName=globalVars['ASG-LaunchConfigName'],ImageId=globalVars['EC2-Amazon-AMI-ID'],KeyName=globalVars['EC2-KeyName'],SecurityGroups=[pubSecGrp.id],UserData=userDataCode,InstanceType=globalVars['EC2-InstanceType'],InstanceMonitoring={'Enabled': False },EbsOptimized=False,AssociatePublicIpAddress=False)# create Auto-Scaling GroupASGSubnets = az1_pubsubnet.id + "," +az2_pubsubnet.idasGroup=asgClient.create_auto_scaling_group(AutoScalingGroupName=globalVars['ASG-AutoScalingGroupName'],LaunchConfigurationName=globalVars['ASG-LaunchConfigName'],MinSize=1,MaxSize=3,DesiredCapacity=2,DefaultCooldown=120,HealthCheckType='EC2',HealthCheckGracePeriod=60,Tags=globalVars['tags'],VPCZoneIdentifier=ASGSubnets)asgClient.create_or_update_tags(Tags=[{'ResourceId': globalVars['ASG-AutoScalingGroupName'],'ResourceType': 'auto-scaling-group','Key': 'Name','Value': globalVars['Project']['Value'] + '-ASG-Group','PropagateAtLaunch': True},])###### Print to Screen ########print("VPC ID : {0}".format(vpc.id))print("AZ1 Public Subnet ID : {0}".format(az1_pubsubnet.id))print("AZ1 Private Subnet ID : {0}".format(az1_pvtsubnet.id))print("AZ1 Spare Subnet ID : {0}".format(az1_sparesubnet.id))print("Internet Gateway ID : {0}".format(intGateway.id))print("Route Table ID : {0}".format(routeTable.id))print("Public Security Group ID : {0}".format(pubSecGrp.id))print("Private Security Group ID : {0}".format(pvtSecGrp.id))print("EC2 Key Pair : {0}".format(globalVars['EC2-KeyName']))print("EC2 PublicIP : {0}".format(globalVars['EC2-KeyName']))print("RDS Endpoint : {0}".format(globalVars['Endpoint']))###### Print to Screen ########"""Function to clean up all the resources"""def cleanAll(resourcesDict=None):# Delete the instancesids = []for i in instanceLst:ids.append(i.id)ec2.instances.filter(InstanceIds=ids).terminate()# Wait for the instance to be terminatedwaiter = ec2Client.get_waiter('instance_terminated')waiter.wait(InstanceIds=[ids])ec2Client.delete_key_pair(KeyName=globalVars['EC2-KeyName'])# Delete Routes & Routing Tablefor assn in rtbAssn:ec2Client.disassociate_route_table(AssociationId=assn.id)routeTable.delete()# Delete Subnetsaz1_pvtsubnet.delete()az1_pubsubnet.delete()az1_sparesubnet.delete()# Detach & Delete internet Gatewayec2Client.detach_internet_gateway(InternetGatewayId=intGateway.id, VpcId=vpc.id)intGateway.delete()# Delete Security GroupspubSecGrp.delete()pvtSecGrp.delete()vpc.delete()
how ever I have not reached that stage to write scale up and scale down policies for cloud watch alarms in the above code, it is just that while executing it before proceeding further I got following errors,
autoscaling.py", line 51, in <module>vpc = ec2.create_vpc(CidrBlock=globalVars['CIDRange'])File "E:\installation2\python3\lib\site-packages\boto3\resources\factory.py", line 520, in do_actionresponse = action(self, *args, **kwargs)File "E:\installation2\python3\lib\site-packages\boto3\resources\action.py", line 83, in __call__response = getattr(parent.meta.client, operation_name)(**params)File "E:\installation2\python3\lib\site-packages\botocore\client.py", line 314, in _api_callreturn self._make_api_call(operation_name, kwargs)File "E:\installation2\python3\lib\site-packages\botocore\client.py", line 612, in _make_api_callraise error_class(parsed_response, operation_name)botocore.exceptions.ClientError: An error occurred (VpcLimitExceeded) when calling the CreateVpc operation: The maximum number of VPCs has been reached.
What should I do to get rid of this error :The maximum number of VPCs has been reached.
, I use free tier services of amazon.
The output of aws ec2 describe-security-groups is following
{"SecurityGroups": [{"Description": "default VPC security group","GroupName": "default","IpPermissions": [{"IpProtocol": "-1","IpRanges": [],"Ipv6Ranges": [],"PrefixListIds": [],"UserIdGroupPairs": [{"GroupId": "sg-03c0e0d31aca5827b","UserId": "101010101010"}]}],"OwnerId": "101010101010","GroupId": "sg-03c0e0d31aca5827b","IpPermissionsEgress": [{"IpProtocol": "-1","IpRanges": [{"CidrIp": "0.0.0.0/0"}],"Ipv6Ranges": [],"PrefixListIds": [],"UserIdGroupPairs": []}],"VpcId": "vpc-06eedbb5dc8c8e20b"},{"Description": "default VPC security group","GroupName": "default","IpPermissions": [{"IpProtocol": "-1","IpRanges": [],"Ipv6Ranges": [],"PrefixListIds": [],"UserIdGroupPairs": [{"GroupId": "sg-080f42f6c90e60956","UserId": "101010101010"}]}],"OwnerId": "101010101010","GroupId": "sg-080f42f6c90e60956","IpPermissionsEgress": [{"IpProtocol": "-1","IpRanges": [{"CidrIp": "0.0.0.0/0"}],"Ipv6Ranges": [],"PrefixListIds": [],"UserIdGroupPairs": []}],"VpcId": "vpc-0a0a0699b064d3382"},{"Description": "default VPC security group","GroupName": "default","IpPermissions": [{"IpProtocol": "-1","IpRanges": [],"Ipv6Ranges": [],"PrefixListIds": [],"UserIdGroupPairs": [{"GroupId": "sg-096d48a3a161a98cc","UserId": "101010101010"}]}],"OwnerId": "101010101010","GroupId": "sg-096d48a3a161a98cc","IpPermissionsEgress": [{"IpProtocol": "-1","IpRanges": [{"CidrIp": "0.0.0.0/0"}],"Ipv6Ranges": [],"PrefixListIds": [],"UserIdGroupPairs": []}],"VpcId": "vpc-090b6525d5e4166bd"},{"Description": "default VPC security group","GroupName": "default","IpPermissions": [{"IpProtocol": "-1","IpRanges": [],"Ipv6Ranges": [],"PrefixListIds": [],"UserIdGroupPairs": [{"GroupId": "sg-0aea8c83e780f8bca","UserId": "101010101010"}]}],"OwnerId": "101010101010","GroupId": "sg-0aea8c83e780f8bca","IpPermissionsEgress": [{"IpProtocol": "-1","IpRanges": [{"CidrIp": "0.0.0.0/0"}],"Ipv6Ranges": [],"PrefixListIds": [],"UserIdGroupPairs": []}],"VpcId": "vpc-0cf43d41666c21f82"},{"Description": "launch-wizard-1 created 2018-08-01T09:57:51.176+05:30","GroupName": "launch-wizard-1","IpPermissions": [{"FromPort": 80,"IpProtocol": "tcp","IpRanges": [{"CidrIp": "0.0.0.0/0"}],"Ipv6Ranges": [{"CidrIpv6": "::/0"}],"PrefixListIds": [],"ToPort": 80,"UserIdGroupPairs": []},{"FromPort": 22,"IpProtocol": "tcp","IpRanges": [{"CidrIp": "0.0.0.0/0"}],"Ipv6Ranges": [],"PrefixListIds": [],"ToPort": 22,"UserIdGroupPairs": []}],"OwnerId": "101010101010","GroupId": "sg-0e81c2a33e1039f58","IpPermissionsEgress": [{"IpProtocol": "-1","IpRanges": [{"CidrIp": "0.0.0.0/0"}],"Ipv6Ranges": [],"PrefixListIds": [],"UserIdGroupPairs": []}],"VpcId": "vpc-08356c60"},{"Description": "default VPC security group","GroupName": "default","IpPermissions": [{"FromPort": 80,"IpProtocol": "tcp","IpRanges": [{"CidrIp": "0.0.0.0/0"}],"Ipv6Ranges": [{"CidrIpv6": "::/0"}],"PrefixListIds": [],"ToPort": 80,"UserIdGroupPairs": []},{"FromPort": 22,"IpProtocol": "tcp","IpRanges": [{"CidrIp": "0.0.0.0/0"}],"Ipv6Ranges": [],"PrefixListIds": [],"ToPort": 22,"UserIdGroupPairs": []}],"OwnerId": "101010101010","GroupId": "sg-40e5492a","IpPermissionsEgress": [{"IpProtocol": "-1","IpRanges": [{"CidrIp": "0.0.0.0/0"}],"Ipv6Ranges": [],"PrefixListIds": [],"UserIdGroupPairs": []}],"VpcId": "vpc-08356c60"}]}
I have 6 vpc all mentioning defalt logged in to web console don't know command line output to find themhere are they
vpc-06eedbb5dc8c8e20bvpc-0a0a0699b064d3382vpc-090b6525d5e4166bdvpc-0cf43d41666c21f82launch-wizard-1 created 2018-08-01T09:57:51.176+05:30vpc-0a0a0699b064d3382
the description of all of them reads Default VPC security group.Basically what I am trying to achieve is to create an auto-scaling configuration for the web tier of a cloud application. Following the steps below. Create a static website with one or two HTML pages and copy the pages and dependent files in an Amazon S3 bucket. Create a startup script to install Apache server and copy the website files from S3 to the instance. Create a Python program using boto that creates an Amazon AutoScaling group. Define scale up and scale down policies and corresponding CloudWatch alarms. Supply the startup script you created in previous step while launching a new instance from the program. Use a security group port 80 open. Open the public DNS of the newly launched instance in a browser and verify if the static website works.What I have not been able to do above is set cloudwatch alarms and metrics and see how they run in my code.
Best Answer
You don't want to create a new VPC each time. So get rid of your ec2.create_vpc call. Instead just define your vpc variable from a describe vpcs call (don't know what that is off the top of my head in boto3) - probably something like ec2.describe_vpcs -> then select whatever one you want to use. Or just hard code the vpc id in your script.
@Kush is correct - there is a soft limit of 5 VPCs per region but you don't really need to create new VPCs for everything you do. If you need to segment different aspects of you AWS infrastructure for security reasons you can rely on subnet ACLs and security groups to accomplish this.