Monitoring + Alerting Baseline for a Real Production App

Cache (Redis):

• Latency: command duration p95.
• Traffic: ops/sec.
• Errors: eviction count.
• Saturation: memory used %, connection count.

Dependencies (external APIs you call):

• Latency + error rate per upstream + per outbound endpoint.

2. Alert classification:

Pages (wake someone up):

• API 5xx rate > 1% sustained 5 min
• API p95 latency > 1s sustained 5 min
• Postgres unreachable > 30 sec
• Postgres connection count > 90% of max sustained 2 min
• Postgres disk > 85% (this becomes critical at 95%)
• Service deployment failed in last 5 min

Slack only (don't wake):

• 4xx rate spike > 10% above baseline
• Single endpoint p95 latency > 2s (one bad endpoint is rarely SEV1)
• Redis eviction count > X
• Slow-query log entry
• Failed background job

Logs only (you check when triaging):

• 5xx for individual endpoints
• Auth failures
• Rate-limit hits

3. Pageable alerts (detail):

Alert: API_5XX_HIGH

• Trigger: (5xx count / total requests) > 1% over 5 min, evaluated every 1 min.
• Severity: SEV2 if affecting a single endpoint. SEV1 if cross-cutting.
• Responder action: check Datadog APM service map for the failing service. Check Sentry for new error fingerprint. Check ECS task health.
• Auto-resolution: if rate drops back under 0.5% for 10 min, page closes.

Alert: API_P95_LATENCY_HIGH

• Trigger: p95 > 1s over 5 min.
• Severity: SEV2.
• Responder action: check DB slow-query log first. Check Redis latency. Check ECS task saturation.
• Auto-resolution: p95 back under 600ms for 10 min.

Alert: DB_UNREACHABLE

• Trigger: zero successful queries for 30 seconds.
• Severity: SEV1.
• Responder action: AWS Console → RDS → check status. Check VPC + security groups. If RDS is down, AWS support ticket + customer comms.
• Auto-resolution: once queries resume for 60 sec.

Alert: DB_CONNECTIONS_HIGH

• Trigger: current connections > 90% of max sustained 2 min.
• Severity: SEV2.
• Responder action: check for connection leak (a process not closing them) via pg_stat_activity. Restart the chattiest service.
• Auto-resolution: drops under 75% for 5 min.

Alert: DB_DISK_HIGH

• Trigger: disk used > 85%.
• Severity: SEV2.
• Responder action: identify table bloat / log table growth. Trigger auto-grow or manual resize.
• Auto-resolution: drops under 80%.

Alert: DEPLOY_FAILED

• Trigger: CD pipeline failed in last 5 min.
• Severity: SEV2.
• Responder action: check pipeline logs. Rollback if needed.

4. Alert hygiene rules:

• Every alert must have a runbook link in its description. No runbook → alert disabled.
• Every alert must have an SLO link. If you can't tie it to user impact, it shouldn't page.
• Any alert that fires > 3x in a week and isn't an outage = noise. Re-tune or kill.
• No alerts on individual metric spikes without duration. 'P95 > 1s for 30 seconds' = false positive. 'P95 > 1s for 5 minutes' = real signal.
• Mute deploy windows. The hour after a deploy = soft thresholds. Hard thresholds resume an hour later.

5. Dashboards (exactly 3):

Dashboard 1 — 'Production Health' (what an on-call checks every morning):

• 5xx rate (24h)
• p95 latency (24h)
• Active error count (Sentry)
• DB connection count + slow queries
• Recent deploys + their commit
• Answers: 'is anything on fire right now?'

Dashboard 2 — 'API Performance' (deep-dive when alerts fire):

• Latency by endpoint (top 10)
• 5xx by endpoint
• Throughput by endpoint
• Top 5 slow DB queries
• Redis hit rate + latency
• Answers: 'where is the slowness coming from?'

Dashboard 3 — 'Capacity Planning' (monthly review):

• 30-day traffic trend
• Peak hour by day of week
• DB connection peaks vs max
• ECS task scale events
• Compute spend trend
• Answers: 'when do we run out of headroom?'

6. Reducing alert noise:

If an alert keeps firing on an external dependency you don't control (say, Stripe is flaky):

• Wrap your retry logic, and only alert if RETRIES EXCEED A THRESHOLD.
• Move the metric from 'every Stripe call failure' to 'sustained failure rate > 5% for 5 min'.
• Page only on customer-impact, not on infrastructure flap.

If an alert is a chronic 'flap' (firing + resolving every 10 min):

• It's not telling you something true. Either widen the duration or merge it with related signals.

7. Cost & effort:

• Datadog Pro: ~$15-23/host/month + log volume. At your traffic, expect $1.5K-3K/month.
• Sentry Team plan: $26/mo + per-event overage. ~$200-500/mo at your traffic.
• Initial setup: 8-12 engineering hours to wire all signals + alerts.
• Ongoing tuning: 1-2 hours/week for the first month, then 1 hour/month.

Better baseline 6 weeks in: on-call complains less. Average pages per week drops from 'every other day' to 'once a week.' False positives < 10% of total.