Alerting Strategies

Introduction

Alerts notify you when something needs attention. Good alerting catches issues early without causing alert fatigue.

What to Alert On

Symptoms: High error rate, high latency, service down.

NOT Causes: High CPU alone doesn't mean there's a problem.

Prometheus Alerting

yaml
groups:
- name: django
  rules:
  - alert: HighErrorRate
    expr: rate(http_responses_total{status="5xx"}[5m]) > 0.1
    for: 5m
    annotations:
      summary: High error rate detected

Sentry Alerts

python
import sentry_sdk
sentry_sdk.init(
    dsn=os.environ['SENTRY_DSN'],
    traces_sample_rate=0.1,
)
# Sentry auto-alerts on new errors

Best Practices

1. Alert on symptoms: User-facing impact.
2. Avoid noise: Only alert on actionable issues.
3. Write runbooks: Document responses.
4. Test alerts: Verify they fire correctly.

Summary

Alert on symptoms that affect users, not just causes. Use Prometheus AlertManager or Sentry. Avoid alert fatigue by keeping alerts actionable.