Prometheus, Grafana, Dynatrace, alerting
Practise these as flashcards. One or two lines each; the ones you miss come back first.
Start freeThe four golden signals: latency, traffic, errors and saturation. Plus host health (CPU, memory, disk), a synthetic check from outside, load balancer pool health, certificate expiry and backup success.
It pulls. On an interval it scrapes each target's HTTP /metrics endpoint, found by static config or service discovery. node_exporter exposes host metrics. Because it pulls, it knows when a target is down (up == 0).
Metrics are numbers over time, cheap and good for alerting. Logs are detailed event records for investigation. Traces follow one request across services to show where time was spent.
Scripted probes that act like a user from outside (load the URL, log in, check the response). They catch problems server-side metrics miss: DNS, certs, LB, WAF, network path. Run them from at least two locations to avoid false alarms.
Pull (Prometheus): the server scrapes targets, simple firewalling, detects dead targets. Push (Datadog agent, StatsD): agents send data out, which suits short-lived jobs and NAT. Prometheus uses Pushgateway for batch jobs.
Infrastructure monitoring watches hosts and services: CPU, disk, process up. APM instruments the application itself: transaction times, slow database calls, code-level errors. Dynatrace and Datadog do both.
Alert on symptoms rather than causes, add for: durations to stop flapping, split severities (page vs ticket), group and dedupe in Alertmanager, and review alerts monthly, deleting ones nobody acts on.
Ship them with Filebeat, Fluent Bit or rsyslog to a central store like ELK, Splunk or Loki. Use structured JSON logs where possible, set retention by log type, and alert on patterns like auth failures or OOM kills.
SLI is what you measure, like the share of requests under 300 ms. SLO is the target, like 99.5%. The error budget is the 0.5% you're allowed to miss; when it's used up, you slow down changes.
You were watching causes (CPU, process up) not symptoms. The break was outside what you monitored: DNS, cert, LB pool, WAF, a dependency, or the network path. Fix: synthetic checks from the user's side, error-rate and latency alerts at the LB, and dependency checks.
Kusto Query Language for Log Analytics / Azure Monitor. Example: Perf | where CounterName == "% Free Space" | summarize min(CounterValue) by Computer | where min_CounterValue < 10 to find VMs low on disk. Pattern: table | where | summarize | project.
It's actionable, tied to user impact or imminent failure, routed to the team that can fix it, has a severity and a runbook, uses a duration to avoid flapping, and gets reviewed. If nobody does anything when it fires, delete or downgrade it.