Use p95 to Tune Autoscaling and Cloud Cost (No Rewrite)

Many teams scale only on average CPU. That is easy, but it may miss queues and tail latency. Use demand signals such as queue depth or request rate to scale, then use your p95 latency SLO to validate and tune the policy.

Why CPU avg wastes money

• CPU utilization is not a direct measure of user experience
• CPU alone does not capture request queuing, downstream waits, or tail latency
• Burst patterns can make a policy scale late and stay high for too long
• You pay more while still missing p95 targets

The p95-aware approach (6 steps)

1) Define SLO (e.g., 95% of requests < 250 ms) 2) Add instrumentation for p95 latency (per service/endpoint) 3) Configure autoscaling on an actionable demand metric such as request rate or queue depth 4) Add hysteresis and cool-down to avoid flapping 5) Protect queues with lag thresholds and exponential backoff 6) Set cost guardrails (budgets, quotas, anomaly detection)

What changes in practice

• Scale before queue explosion; scale down when tails calm
• Fewer retry storms and less error amplification
• Better cost/perf ratio with the same infrastructure

Quick wins (1-2 weeks)

• Export p95 via OTel/Datadog; create a custom metric
• Tune HPA policies with demand metrics; keep p95 as the SLO guardrail
• Right-size instances; fix one noisy endpoint
• Add daily budget alerts + lifecycle rules for logs

Example: Kubernetes HPA (request-rate metric)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 20
  behavior:
    scaleDown: { stabilizationWindowSeconds: 300 }
  metrics:
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "50"

Example: Queue scaling with KEDA (Pub/Sub/Rabbit/Kafka)

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: worker-queue
spec:
  scaleTargetRef:
    kind: Deployment
    name: worker
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus:9090
      metricName: queue_lag
      threshold: "1000"
      query: sum(queue_lag{consumer_group="worker-queue"})

What to measure

• p95 during burst and steady windows
• Queue lag vs. consumer throughput
• Spend / 1000 requests (cost-to-serve)

Possible result: lower cost and fewer latency incidents without an application rewrite. The outcome depends on workload shape, metric delay, and the original scaling policy, so measure against a baseline rather than assuming a fixed saving.

Illustrative Example: Before and After

Before (CPU-based scaling):

• Average CPU: 45%
• p95 latency: 450ms (SLO: 250ms)
• Instances: 10-25 (thrashing)
• Monthly cost: €2,400
• Incidents: 3 per month (latency spikes)

After (demand-based scaling, validated against p95):

• Average CPU: 60% (higher utilization)
• p95 latency: 180ms (within SLO)
• Instances: 5-12 (stable)
• Monthly cost: €1,680 (30% reduction)
• Incidents: 0 per month

Key insight: CPU doesn’t reflect user experience. p95 latency does.

These numbers illustrate how to compare policies; they are not a benchmark or a guaranteed outcome.

Implementation Guide

Step 1: Instrument p95 Latency

For Kubernetes (Prometheus):

# Add to your deployment
apiVersion: v1
kind: Service
metadata:
  name: api
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9090"
    prometheus.io/path: "/metrics"
spec:
  ports:
  - port: 80
    targetPort: 8080
---

In your application code (Go example):

package metrics

import (
  "net/http"
  "strconv"
  "time"

  "github.com/prometheus/client_golang/prometheus"
  "github.com/prometheus/client_golang/prometheus/promauto"
)

var httpRequestDuration = promauto.NewHistogramVec(
  prometheus.HistogramOpts{
    Name:    "http_request_duration_seconds",
    Help:    "HTTP request duration in seconds",
    Buckets: []float64{0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0},
  },
  []string{"method", "endpoint", "status"},
)

func handler(w http.ResponseWriter, r *http.Request) {
  start := time.Now()
  statusCode := http.StatusOK

  // ... your handler logic ...

  duration := time.Since(start).Seconds()
  httpRequestDuration.WithLabelValues(
    r.Method,
    r.URL.Path,
    strconv.Itoa(statusCode),
  ).Observe(duration)
}

For Cloud Run / Cloud Functions:

• Use OpenTelemetry or Cloud Monitoring client libraries
• Export custom metrics to Cloud Monitoring
• Create custom metrics from request logs

Step 2: Create Scaling and SLO Metrics

Prometheus recording rule:

groups:
- name: slo
  interval: 30s
  rules:
  - record: http_requests_per_second
    expr: |
      sum(rate(http_requests_total[5m])) by (namespace, pod)
  - record: http_p95_latency_ms
    expr: |
      histogram_quantile(0.95,
        sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
      ) * 1000

Cloud Monitoring custom metric:

# Python example for Cloud Run
from google.cloud import monitoring_v3
import time

client = monitoring_v3.MetricServiceClient()
project_name = f"projects/{PROJECT_ID}"

# Create custom metric descriptor
descriptor = monitoring_v3.MetricDescriptor()
descriptor.type = "custom.googleapis.com/http_p95_latency_ms"
descriptor.metric_kind = monitoring_v3.MetricDescriptor.MetricKind.GAUGE
descriptor.value_type = monitoring_v3.MetricDescriptor.ValueType.DOUBLE
descriptor.description = "p95 latency in milliseconds"

client.create_metric_descriptor(
    name=project_name, metric_descriptor=descriptor
)

# Write metric
series = monitoring_v3.TimeSeries()
series.metric.type = "custom.googleapis.com/http_p95_latency_ms"
series.resource.type = "cloud_run_revision"
series.resource.labels["project_id"] = PROJECT_ID
series.resource.labels["location"] = "europe-west4"
series.resource.labels["service_name"] = "api"
series.resource.labels["revision_name"] = "api-001"
series.resource.labels["configuration_name"] = "api"

point = monitoring_v3.Point()
point.value.double_value = p95_latency_ms
point.interval.end_time.seconds = int(time.time())
series.points = [point]

client.create_time_series(name=project_name, time_series=[series])

Step 3: Configure HPA with a Demand Metric

Kubernetes HPA with Prometheus adapter:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 20
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300  # 5 min cooldown
      policies:
      - type: Percent
        value: 50  # Scale down max 50% at a time
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0  # Scale up immediately
      policies:
      - type: Percent
        value: 100  # Can double replicas
        periodSeconds: 30
      - type: Pods
        value: 4  # Or add 4 pods
        periodSeconds: 30
      selectPolicy: Max  # Use the more aggressive policy
  metrics:
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "50"  # Target requests/second per pod
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80  # Second safety metric; HPA uses the larger recommendation

Troubleshooting HPA:

# Check HPA status
kubectl get hpa api-hpa

# Describe HPA to see scaling decisions
kubectl describe hpa api-hpa

# Check if metrics are available
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests_per_second"

Step 4: Add Queue-Based Scaling (Optional)

For services with queues (Pub/Sub, RabbitMQ, Kafka), add queue lag metrics:

KEDA ScaledObject for Pub/Sub:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: worker-queue
spec:
  scaleTargetRef:
    kind: Deployment
    name: worker
  minReplicaCount: 1
  maxReplicaCount: 50
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus:9090
      metricName: pubsub_subscription_num_undelivered_messages
      threshold: "1000"  # Scale when > 1000 messages
      query: |
        sum(pubsub_subscription_num_undelivered_messages{subscription="worker-queue"})

Cloud Run / Cloud Functions Alternative

Managed serverless platforms do not use an HPA. Cloud Run scales primarily from incoming demand, using concurrency and CPU behavior. Treat p95 as the validation and alerting signal, then tune the platform’s built-in controls:

gcloud run services update api \
  --region=europe-west4 \
  --min-instances=0 \
  --max-instances=50 \
  --concurrency=40 \
  --cpu-boost

Export p95 to Cloud Monitoring, alert on sustained SLO breaches, and compare latency and cost before and after each concurrency or instance-limit change. A Monitoring alert does not scale Cloud Run by itself. Custom-metric scaling requires a separate controller that changes service configuration, with rate limits and rollback controls.

Monitoring and Alerting

Key Metrics Dashboard

Grafana dashboard queries:

# p95 latency over time
histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
) * 1000

# Instance count
count(kube_pod_info{pod=~"api-.*"})

# Cost per 1000 requests (if you have cost metrics)
sum(rate(cloud_cost_euros[1h])) / sum(rate(http_requests_total[1h])) * 1000

# Queue lag (if applicable)
sum(pubsub_subscription_num_undelivered_messages)

# Error rate
sum(rate(http_requests_total{status=~"5.."}[5m])) / 
sum(rate(http_requests_total[5m]))

Alerting Rules

groups:
- name: autoscaling
  rules:
  - alert: HighP95Latency
    expr: |
      histogram_quantile(0.95,
        sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
      ) * 1000 > 250
    for: 5m
    annotations:
      summary: "p95 latency above SLO threshold"
      description: "p95 latency is {{ $value }}ms, target is 250ms"
  
  - alert: ScalingThrashing
    expr: |
      changes(kube_horizontalpodautoscaler_status_current_replicas[10m]) > 4
    for: 10m
    annotations:
      summary: "HPA is thrashing (scaling up/down rapidly)"
      description: "Replica count changing too frequently"
  
  - alert: QueueLagHigh
    expr: |
      sum(pubsub_subscription_num_undelivered_messages) > 10000
    for: 5m
    annotations:
      summary: "Queue lag is high"
      description: "{{ $value }} messages undelivered"

Common Pitfalls and Solutions

Pitfall 1: Metric Delay

Problem: p95 latency can lag by 30-60 seconds, so it is a poor primary scaling trigger.

Solution:

• Scale on request rate or queue lag
• Use shorter p95 windows for alerting where the sample size is sufficient
• Add scheduled or predictive capacity for known traffic patterns

Pitfall 2: Noisy Metrics

Problem: p95 spikes from a small number of slow requests can make a healthy scaling policy look unstable.

Solution:

• Use longer aggregation windows (5 minutes)
• Add minimum duration before scaling (stabilization window)
• Filter out known slow endpoints (health checks, admin APIs)

Pitfall 3: Cold Start Latency

Problem: New instances have high latency during startup, temporarily worsening p95.

Solution:

• Set minimum instances to avoid cold starts
• Use startup CPU boost (Cloud Run)
• Exclude startup period from latency calculations

Pitfall 4: Downstream Dependency Issues

Problem: High latency from database/API, not your service.

Solution:

• Monitor downstream dependencies separately
• Scale based on queue depth, not just latency
• Implement circuit breakers to fail fast

Cost Optimization Tips

1. Right-size instances: Use smaller instances with higher concurrency
2. Set appropriate min/max: Don’t over-provision, but avoid cold starts
3. Use spot/preemptible instances: For non-critical workloads
4. Implement cost guardrails: Budget alerts, quota limits
5. Monitor cost-to-serve: Track €/1000 requests over time

Outcomes to measure:

• Cost per 1,000 successful requests
• Latency-related incident count
• p95 latency during steady and burst traffic

Next Steps

1. Week 1: Instrument p95 latency, create custom metrics
2. Week 2: Configure HPA with request-rate or queue-based scaling
3. Week 3: Add queue-based scaling (if applicable)
4. Week 4: Tune thresholds, add alerts, measure impact

Cloud Run/Gateway note: If you’re not on Kubernetes, tune concurrency and min/max instances, and use p95 alerts to validate the built-in scaling policy.