Recently, I observed a notable increase in our AWS invoice, amounting to $8,000 compared to the previous month. To identify the root causes of this spike, we conducted a thorough investigation utilizing AWS Cost Explorer and AWS billing reports. My analysis revealed several key contributors to the cost escalation, including EC2-others, Saving Plan, EC2 instances, Tax, EFS, S3, CloudTrail, and CloudWatch.

Through a detailed examination, I uncovered that “regional data transfer – in/out/between EC2 AZs” emerged as a significant factor amplifying our workloads’ costs. 

To understand the regional data transfer cost, I deep-dived into our platform backend architecture, which includes Parse Server application nodes, MongoDB self managed database nodes, and Dremio as data lake layer.

To better understand and analyze the inter-AZ data transfer patterns, we enabled custom VPC flow logs and stored them in an Amazon S3 bucket. I then queried the logs using Amazon Athena to identify the top inter AZ data transfer contributors.

Key differences between AWS Cost Explorer and AWS Billing reports

What is regional Data Transfer in AWS?

Regional Data Transfer refers to the movement of data within the same AWS region across different resources and Availability Zones (AZs). AWS charges for some types of data transfer depending on the direction and scope of the data flow.

Categories of Regional Data Transfer

  1. Data Transfer In (Inbound)

    • Data transferred into EC2 instances or other services within a region.
    • Typically free of charge.

  2. Data Transfer Out (Outbound)

    • Data transferred out of EC2 instances or other services within a region to external services or users.
    • Charged based on the volume of data.

  3. Data Transfer Between EC2 Instances Across AZs

    • Data transferred between different Availability Zones (AZs) in the same region.
    • Charged per GB because data must traverse AWS networking infrastructure.

  4. Data Transfer Within the Same AZ (Same Subnet)

    • Data transfer within the same Availability Zone is typically free if instances are in the same VPC.

Example Scenarios

1. Data Transfer In (Free)

  • We upload data from an external system (on-premises or the internet) to an S3 bucket in the same AWS region. There are no charges for data transfer IN to S3 or EC2.

2. Data Transfer Out (Charged)

  • Suppose we have an application running on an EC2 instance in the us-east-1 region.
  • When this application serves content to an external user (say a client in a browser), the data transfer OUT is charged based on the volume.

3. Data Transfer Between EC2 Instances in Different AZs (Charged)

  • Instance A in us-east-1a sends data to Instance B in us-east-1b.
  • AWS charges for this cross-AZ data transfer at a per-GB rate because the traffic uses AWS’s backbone infrastructure.

4. Data Transfer Within the Same AZ (Free)

  • Instance X and Instance Y are both in us-east-1a within the same VPC.
  • They communicate over a private IP address.
  • AWS does not charge for this intra-AZ data transfer.

What is AWS VPC flow logs?

AWS VPC (Virtual Private Cloud) Flow Logs capture information about network traffic flowing to and from interfaces within a VPC. They provide detailed insights into the source, destination, and volume of traffic within AWS.

Flow logs can be enabled for:

  • A specific VPC
  • A specific Subnet
  • A specific Network Interface (ENI)

The logs are stored in Amazon CloudWatch Logs, Amazon S3, or other destinations, where they can be analyzed for network behavior, security monitoring, and cost optimization.

How VPC Flow Logs Assist in Identifying Regional Data Transfer Costs

  1. Detect Cross-AZ Traffic (Chargeable)

    • By analyzing source and destination IPs, we can determine whether traffic flows between different Availability Zones (AZs) within a region, which incurs costs.

    Example:

    • Source IP: 10.0.1.10 in AZ us-east-1a
    • Destination IP: 10.0.2.20 in AZ us-east-1b
      If this traffic is significant, it indicates a design pattern leading to cross-AZ charges.

What is Amazon Athena?

Amazon Athena is a serverless, interactive query service that allows us to analyze data stored in Amazon S3 using standard SQL. It eliminates the need to set up and manage data warehouses, making it ideal for ad-hoc querying of structured or semi-structured data formats (like JSON, Parquet, or CSV).

Why Use Athena for VPC Flow Logs Analysis?

VPC Flow Logs generate large amounts of detailed data about network traffic. Storing these logs in S3 and analyzing them directly using Athena provides a scalable and cost-effective solution for gaining insights into our VPC traffic patterns.

Next Steps

To mitigate the costs with regional Data Transfer, Option is to consolidate application server and data lake  nodes into single Availability Zones. Additionally, we could operate MongoDB nodes in two Availability Zones instead of three and manage the elastic file system from a single AZ. These adjustments had the potential to reduce regional data transfer costs without compromising the integrity of our solution.

Stay tuned for my next blog, how I have consolidated the application server and data lake nodes to single AZ and using EC2 autoscaling and Amazon Cloudwatch service to maintain the resilient architecture.

(Visited 3 times, 1 visits today)