Read metrics from AWS CloudWatch

​

When NOT to use this

• The metric is already in Prometheus via a CloudWatch exporter (yace, cloudwatch_exporter). Use the prometheus source. You get one query against your Prometheus instead of one CloudWatch API call per tick, and the CloudWatch billing surface stays at your exporter.
• You need sub-minute granularity. CloudWatch periods are 60s, 300s, 900s, or 3600s. If you need 10-second resolution, send the metric over OTLP instead.
• You need to alert on the absence of a metric. CloudWatch can take 1-3 periods to publish; the agent looks back 5 periods or 10 minutes, whichever is longer, to absorb that lag. If a metric stops emitting, cloudwatch_no_data surfaces only after that window, not immediately.

​

AWS credentials

1. Environment variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, optional AWS_SESSION_TOKEN.
2. AWS shared credentials file (~/.aws/credentials) with an optional AWS_PROFILE env var to pick a profile.
3. EC2 instance metadata (the agent runs on an EC2 instance with an attached IAM role).
4. ECS task role (the agent runs as an ECS task with a task role).
5. EKS pod identity / IRSA (the agent runs as a Kubernetes pod with an associated service account).

​

Minimum-permissions IAM policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ObserverAgentReadMetrics",
      "Effect": "Allow",
      "Action": [
        "cloudwatch:GetMetricData",
        "cloudwatch:ListMetrics"
      ],
      "Resource": "*"
    }
  ]
}

​

Cross-account access

1. In account B, create a role (e.g. observer-cloudwatch-read) with the policy above. Add a trust policy permitting account A’s role / user to assume it:
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": "arn:aws:iam::AAAAAAAAAAAA:role/observer-agent"
         },
         "Action": "sts:AssumeRole",
         "Condition": {
           "StringEquals": { "sts:ExternalId": "your-external-id" }
         }
       }
     ]
   }
   
2. In Observer, set the metric definition’s Role ARN to arn:aws:iam::BBBBBBBBBBBB:role/observer-cloudwatch-read and External ID to your-external-id. The agent calls sts:AssumeRole with its ambient credentials. The agent assumes the role once and reuses the session, refreshing it before it expires.
3. In account A, attach this inline policy to the role the agent uses (the same identity named in the Principal of step 1’s trust policy). That’s usually the IRSA service-account role, the EC2 instance role, or the ECS task role:
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": "sts:AssumeRole",
         "Resource": "arn:aws:iam::BBBBBBBBBBBB:role/observer-cloudwatch-read"
       }
     ]
   }
   

​

Configure a metric in the Observer console

• Region: AWS region code (us-east-1, eu-west-2, etc.).
• Period: 60s, 300s, 900s, or 3600s. Lower = more API calls per hour (one per cron tick) and finer-grained alerting. See Cost considerations.
• Namespace: AWS/RDS, AWS/Lambda, AWS/ApplicationELB, or your custom namespace. See Common namespaces and metrics.
• Metric name: e.g. CPUUtilization. Case-sensitive.
• Dimensions: Key=Value lines that scope to a single resource (e.g. DBInstanceIdentifier=prod-db).
• Statistic: Average, Sum, Minimum, Maximum, SampleCount, or a percentile (p50, p95, p99.9). See Statistic reference.
• Role ARN / External ID (optional): for cross-account reads. See Cross-account access.

​

Statistic reference

​

Common namespaces and metrics

​

Cost considerations

1. Raise the period for metrics that don’t need 1-minute granularity. A 300s period cuts the call rate by 5×.
2. Move stable metrics to Prometheus via a CloudWatch exporter. The exporter consolidates many CloudWatch metrics into one exporter scrape; Observer reads the exporter via Prometheus without per-metric CloudWatch billing.

​

Reason codes specific to CloudWatch

• cloudwatch_no_data: GetMetricData returned an empty value list. The metric is not publishing, or no data point exists in the lookback (5 periods or 10 minutes, whichever is longer).
• cloudwatch_access_denied: the agent’s credentials cannot call GetMetricData against this metric. Check the IAM policy on the role / user the agent assumes.
• cloudwatch_throttled: AWS is rate-limiting the agent. Raise the period, or split the metric across multiple agents in different AWS accounts.
• cloudwatch_invalid_parameter: the request was malformed. Common causes: a dimension name CloudWatch doesn’t recognize, or a statistic the metric doesn’t support.
• cloudwatch_resource_not_found: the namespace, metric name, and dimension combination doesn’t exist (and never has) in this region and account.
• cloudwatch_expired_credentials: an STS session expired. The agent refreshes automatically; this should self-heal on the next tick.
• cloudwatch_server_error: AWS returned 5xx. Transient; usually clears within minutes.
• cloudwatch_error: an uncategorized error. Check the AWS service health dashboard.

​

Troubleshooting

• cloudwatch_access_denied and IAM policy looks right. Check the role’s trust policy. The agent’s identity must be a principal named in the target role’s AssumeRolePolicyDocument. Run aws sts get-caller-identity from the agent’s host to confirm which identity it’s using.
• cloudwatch_no_data but the metric is visible in the AWS Console. Check the dimensions exactly. CloudWatch matches on the full dimension set: a metric published with {DBInstanceIdentifier=prod-db} is not the same metric as {DBInstanceIdentifier=prod-db, EngineName=postgres}. The Console shows you the dimensions when you click a metric.
• cloudwatch_throttled repeatedly. Multiple metric definitions on the same agent share an account-wide TPS limit. Either raise the period for non-critical metrics or split agents per account.
• Wrong region. A metric in eu-west-1 is invisible from a us-east-1 query. Each region has its own metric definition.
• Cross-account read returns cloudwatch_no_data but works for the IAM user directly. The assumed-role session inherits the role’s policy, not the trust-policy principal’s. Add cloudwatch:GetMetricData to the target role’s policy, not the source role’s.

​

Known limits

• One metric per definition. This source does not batch across metric defs. If you need 500 metrics in one API call, write a custom dashboard against GetMetricData directly and forward aggregates as OTLP.
• Static credentials are not stored. If you cannot use the AWS credential chain (env vars / instance role / IRSA), set the env vars on the agent process. Per-metric access keys would require an encryption + rotation surface that v1 deliberately omits.
• Two ways to discover metric names. The namespace + metric-name inputs are pre-populated from a curated catalog covering the AWS services most operators instrument (RDS, Lambda, ApplicationELB, EC2, SQS, ApiGateway, ECS, DynamoDB, S3, CloudFront, SNS, Step Functions, Kinesis, Network LB). For custom namespaces or AWS services outside the catalog, click Fetch from AWS next to the metric name input: the agent runs cloudwatch:ListMetrics against the configured region (using its own AWS credentials, including any cross-account role ARN you set) and returns the live list within ~5 seconds. Click a row to fill the metric name + dimensions in one step.