AWS Yarn AWS Yarn

What are Cloudwatch Metrics?

Cloudwatch metrics are data about the performance of our systems. By default, many services provide free metrics for resources (such as Amazon EC2 instances, Amazon EBS volumes, and Amazon RDS DB instances). We can also enable detailed monitoring for some resources, such as our Amazon EC2 instances, or publish our own application custom metrics. Amazon CloudWatch can load all the metrics in our account (both AWS resource metrics and application metrics that you provide) for search, graphing, and alarms.

Use Case – I have been tasked to develop a solution, to have a consolidated view of all the EC2 instances utilization across multiple AWS accounts without deploying or enabling cross-account access. 

Amazon CloudWatch is the way to got to monitor AWS resources including EC2 instances. Cloudwatch collects and processes raw data from Amazon EC2 into readable, near real-time metrics.

The best way to monitor all the Cloudwatch resources is AWS provided Cross-Account and Cross-region cloudwatch console solution. As the AWS suggested solution requires AWS account level configuration changes like deploying cross-account IAM roles, enabling cross-account access, etc, which triggers the tedious process of change management requests and approvals.

To avoid any deployment and configuration changes to the existing accounts and using our existing AWS access keys and roles the solution I came up with is a python script to

1. Filter out all the running EC2 instances in the Client AWS account.
2. Make get_metric_statistics call to each instance. (Fetch)
3. Process and transform the response. 
4. Push custom AWS/EC2 metrics to the Central (Master) AWS account.

Architecture

Code and Usage Instructions.

Github Repository

https://github.com/avasisht/cloudwatch-metrics

Code Explanation

To store the client and master account AWS credentials, we are using boto3 custom sessions.

sessionClient = boto3.session.Session(profile_name = cProfile)

sessionMaster = boto3.session.Session(profile_name = mProfile)

instances = ec2.instances.filter(

    Filters=[{‘Name’: ‘instance-state-name’, ‘Values’: [‘running’]}])

To get the CPU Utilization CloudWatch Metrics for EC2 Instances from the Client Account.

for instance in instances:

    cloudwatchClient = sessionClient.client(‘cloudwatch’)

    getRequest = cloudwatchClient.get_metric_statistics(

            Namespace=”AWS/EC2″,

            MetricName=”CPUUtilization”,

            Dimensions=[

            {

                ‘Name’: ‘InstanceId’,

                ‘Value’: instance.id

            }

            ],

            StartTime=datetime.utcnow() – timedelta(days = 14),

            EndTime=datetime.utcnow(),

            Period=900,

            Statistics=[‘Average’]

            )

    filterResponse = getRequest.get(‘Datapoints’,[])

    cloudwatchMaster = sessionMaster.client(‘cloudwatch’)

    for dic in filterResponse:

        putResponse = cloudwatchMaster.put_metric_data(

            MetricData=[

                {

                    ‘MetricName’: ‘CPUUtilization’,

                    ‘Dimensions’: [

                        {

                        ‘Name’:’InstanceId’,

                        ‘Value’: instance.id

                        }

                    ],

                    ‘Value’: dic[‘Average’],

                    ‘Timestamp’: dic[‘Timestamp’]

                }

            ],

            Namespace = ‘Custom/EC2’

        )

Aggregate view of EC2 instances custom CPUUtilization metrics.

Conclusion

We can use AWS Cloudwatch get_metric_statistics or get_metric_data API to pull the Cloudwatch metrics from Multiple AWS accounts and to have an aggregate view of all the metrics publish the data using put_metric_data API to a central AWS Cloudwatch Dashboard for visualization.

References

• https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Cross-Account-Cross-Region.html
• https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricStatistics.html