How do you set alert thresholds people will not ignore?

A threshold alone will not stop alert noise. If an alert has no action attached to it, people will ignore it regardless of the number in the condition. A good rule reports a situation that already affects the service or soon will, reaches someone with the authority to fix it, and carries enough context for the first decision.

I have watched teams try to cure hundreds of overnight messages in one move: raise CPU from 80 to 90 percent, extend the window from five minutes to ten, or disable the loudest channel. The noise came back a month later, while incidents were discovered later. The reason is simple: the threshold, signal persistence, routing, grouping, and suppression solve different problems. Configure them separately.

An alert has to earn the right to wake someone

An alert has a right to exist only when its recipient can name a specific action and an acceptable response time. If an engineer merely opens a graph and watches after receiving it, the signal belongs on a dashboard, not in a page.

In the Monitoring Distributed Systems chapter, Google SRE draws a useful line between a symptom and a cause. A symptom answers "what is broken for the user," while a cause helps explain "why." A symptom is usually stronger grounds for an urgent page: a rising share of failed requests already shows a service failure, while CPU utilization at 92 percent may be normal work or a cause that has not hurt anyone yet. You still need the resource metric, but it often belongs in diagnostics or a business-hours ticket.

Fill in five fields before creating a rule:

1. Observed harm: which service function is broken.
2. Recipient: which team owns the fix.
3. Action: what the engineer will check or change first.
4. Deadline: how long the situation can wait before the harm grows.
5. Resolution condition: when the incident has actually ended.

If one field remains empty, do not enable delivery. Fix ownership, telemetry, or the instructions first. "Let the on-call engineer take a look" is particularly dangerous: it turns an engineer's attention into a filter that should have been implemented in the query.

A useful test is deliberately strict: if no one acted on the last ten firings and nothing happened, the rule either went to the wrong people or reported the wrong state. Do not delete it immediately. First find out whether the condition was false, the action happened automatically, the priority was too high, or the message duplicated another signal.

In the notification, state the observed symptom, scope, start time, owner, and first safe action. A value such as 87.3 without a unit, window, or expected boundary forces the on-call engineer to reverse engineer the query. A link to a runbook does not repair a weak message either. The first line should let the reader decide whether to acknowledge an incident and where to look.

Give every rule an owner and a review date. The owner is not responsible for watching it manually around the clock. They are responsible for the meaning of the expression, its route, and its instructions. When a service moves to another team, the rule must move with it. Ownerless alerts nearly always become noise because no one has the authority to change or remove them.

State history reveals where noise starts

Threshold tuning starts with a review of state transitions over several weeks, not with picking a new number. You need the start and end time, metric value, label set, recipient, acknowledgement, action taken, and related incident. Without that history, the discussion quickly collapses into personal impressions.

Split noisy firings by origin. Four distinct classes appear often:

• a brief spike that ends faster than a human could respond;
• flapping near the boundary as the condition alternates between true and false;
• many instances of one outage caused by excess labels;
• a valid signal with the wrong urgency or recipient.

These are not one problem. A brief spike needs a pending period, flapping needs separate entry and exit boundaries, an instance flood needs grouping, and a wrong recipient needs a different routing policy. Raising the threshold in all four cases is dangerous. It will cut the message count but lose sensitivity where the value itself was right.

Do not count notifications alone. The share of firings that led to human action says more about usefulness. Track confirmed incidents that lacked a timely signal, repeated notifications for one event, manually disabled rules, and time to acknowledgement separately. A zero response rate sometimes means fatigue rather than a lack of harm.

Do not mix up a false alert and an insignificant alert. False means the expression described the observed state incorrectly. Insignificant means the state was real but did not warrant that channel and response time. Fix the query or data in the first case. Change severity and routing in the second.

Another class often hides behind the word "repeat": a new notification after a brief resolution. To a person it is the same incident, but the system sees a new state sequence. Compare messages by time window, service, and accepted incident, not only by technical identifier. This exposes a problem in the recovery condition rather than in the repeat delivery frequency.

Check how firings are distributed across hours and changes. Noise concentrated around backups, batch processing, or deployments points to a predictable operating mode. Do not mute monitoring for the whole scheduled period if the work can still exceed safe limits. Express permitted behavior as a separate condition and retain a signal for excessive duration or user harm.

Choose a threshold from impact and time margin

A threshold should mark the boundary after which the team has limited time before noticeable harm. The average from a quiet week says almost nothing about that boundary. Look for the relationship between the metric, depletion of the safety margin, and actual service failure.

For disk capacity, a simple "more than 80 percent used" condition travels badly between volumes. A 100 GB disk has 20 GB left, which may disappear in minutes. A 20 TB disk at the same percentage has 4 TB left. A signal based on predicted time to full answers the operational question better. A sudden increase in write rate still needs a separate safeguard because a linear forecast depends on recent behavior.

For latency and errors, choose a window that matches the user flow. An average hides the tail of a distribution, while a narrow percentile at low traffic jumps because of a few requests. It is useful to require both poor quality and enough observations. Otherwise, one failure among two requests at night produces a "50 percent error rate," even though an urgent response may change nothing.

A static threshold remains a good choice when a hard limit exists: certificate expiry, address capacity, queue length before a known failure, or an equipment temperature from its specification. A "smart" dynamic threshold must not override a physical or contractual boundary. It can detect deviation from seasonal behavior, but it can also learn from bad periods and accept gradual degradation as the new normal.

Write the basis for every number in plain language: "At this rate, space will run out in less than four hours, and a safe expansion takes two." If no one can reconstruct the basis, the number becomes magic and outlives changes in traffic, architecture, and recovery time.

For a service with an SLO, error budget burn rate is more useful than a single error threshold for every situation. A short window detects a fast fire, while a long window confirms that consumption is sustained. Requiring both windows to breach cuts reactions to random spikes without losing early detection of a severe outage. The signal only makes sense when the SLO is defined and successful events are counted correctly.

A capacity threshold should account for delivery and infrastructure change time. If expansion requires approval, hardware delivery, and migration, create a ticket earlier than for a resource that can be added safely by automation. In a physical server environment, measure the margin in more than a disk or memory percentage. Include the time the team actually needs to restore spare capacity.

Pending time and hysteresis remove different repeats

A pending period filters brief breaches, while hysteresis keeps the state from flapping around one boundary. They are not interchangeable. A long wait around an unstable threshold only postpones the next message, while wide hysteresis without a wait still admits every sharp spike.

In Prometheus, the for field requires an expression to remain true continuously before moving from pending to firing. The keep_firing_for field keeps the active state for a time after the condition disappears. The official Prometheus documentation explicitly recommends the second field to reduce flapping and false resolutions caused by missing data.

A working rule can look like this:

groups:
- name: api-slo
  rules:
  - alert: ApiHighErrorRate
    expr: |
      (
        sum(rate(http_requests_total{status=~"5.."}[5m]))
        /
        sum(rate(http_requests_total[5m]))
      ) > 0.05
      and
      sum(rate(http_requests_total[5m])) > 1
    for: 10m
    keep_firing_for: 5m
    labels:
      severity: page
      service: api
      team: platform
    annotations:
      summary: "API returns more than 5% server errors"
      action: "Check recent deployment and upstream availability"

Here the quality threshold is separate from minimum traffic, ten minutes protects against a short burst, and five minutes of retention prevents the message from resolving and reopening after one good measurement. Do not copy these numbers blindly. The for period should be shorter than the available response time but longer than a normal safe spike.

True hysteresis needs different entry and exit conditions, such as opening a state above 85 percent full and considering recovery stable only below 80. Not every rule system can express this in one condition. Sometimes it is easier to record the state in a separate metric or use two thresholds in the handler. The resolution should indicate restored margin, not a chance touch of the boundary.

Match for to the evaluation frequency. With a one-minute interval, a ten-minute wait produces about ten consecutive confirmations. With a five-minute interval, it produces only two. Check what happens when the rule server restarts and when a series disappears temporarily. State continuity may reset even though the user problem remains.

Do not put a long for on a signal that already means irreversible harm has occurred. Loss of one replica may tolerate confirmation while redundancy works. Confirmed data corruption, an expired certificate, or loss of the only entry point needs different logic. In those cases, prevent false pages by improving the measurement and adding independent confirmation, not by waiting.

Group by incident, not by source

One notification should describe one operational situation even when hundreds of instances detect it. Alertmanager deduplicates identical label sets and can group related alerts into one delivery. A poor label choice, however, makes every event unique.

Labels such as request_id, the full error text, a temporary container name, or an exact metric value almost always create uncontrolled cardinality. Keep stable properties that assign an owner and distinguish independent outages in the identity: service, cluster, region, team, and severity class. Put instance details in annotations or open them in a dashboard.

The route configuration shows the principle:

route:
  receiver: operations
  group_by: [cluster, service, alertname]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
  - matchers:
    - severity="page"
    receiver: oncall
    repeat_interval: 30m

group_wait gives related events a short time to collect into the first delivery. group_interval controls how often changes inside an existing group are sent. repeat_interval controls reminders while the state continues. Do not set the last value equally for every class. A half-hour reminder can make sense during a continuing critical outage but turns a business-hours ticket into spam.

Do not group only by team. Independent failures in different services will merge into one message, and partial resolution will be hard to read. Do not include instance by default. A separate page per instance is justified only when each instance requires its own action and the team can perform those actions in parallel.

An excessive group_wait delays the first page for the sake of a tidy summary. On a critical route, it should remain a small fraction of the response time. If it is too short, the first spark is sent alone and a flood follows seconds later. Tune it to the usual propagation time across dependent checks.

Deduplication depends on stable labels. If someone accidentally turns an annotation containing the metric value into a label, every evaluation gets a new fingerprint and bypasses deduplication. Define the label contract in a rule template and test changes automatically. That costs less than investigating why one condition created hundreds of "unique" events.

Suppression should reflect a dependency

When a known primary outage explains downstream symptoms, deliver the primary signal and suppress the consequences. Alertmanager calls this inhibition: a target alert is not delivered while a source alert with matching labels from the equal list is active.

If a cluster is unreachable, messages about every service and node rarely add another action. An inhibition rule can retain the ClusterUnreachable page while hiding InstanceDown in the same cluster:

inhibit_rules:
- source_matchers:
  - alertname="ClusterUnreachable"
  - severity="page"
  target_matchers:
  - alertname="InstanceDown"
  equal: [cluster]

Matching on cluster protects neighboring clusters from accidental suppression. The official Alertmanager configuration warns about a subtle point: a missing label and an empty label are considered equal. Fields in equal must therefore be present and validated in both rules. Otherwise, a source without the label may silence targets you never meant to connect.

Inhibition differs from a temporary silence. An operator creates a silence for a maintenance period, and it matches on labels. Inhibition permanently encodes a dependency between signals. Do not use a long silence instead of repairing a noisy rule. When it expires, the flood will return and the cause will remain unknown.

Automatic inhibition is safe only when the primary signal is dependable. If ClusterUnreachable relies on the same broken collection system, downstream messages may be the only evidence left. Test the rule on an archived incident and on loss of the data source, then confirm that inhibited events remain visible in the interface for diagnosis.

A planned maintenance window solves a third problem. It temporarily changes expected behavior for a known object but does not prove a dependency between signals. Keep the window narrow by service, cluster, and time, and record its author and reason. Disabling a whole channel overnight hides neighboring failures and leaves an unclear state after the work.

Do not suppress a user-impact symptom merely because a suspected cause is active. If the database is unavailable and an external test confirms failed payments, the on-call engineer benefits from seeing the scope of user impact in the main incident. Suppress extra pages, but keep symptoms in the incident record or group as evidence of how the outage develops.

Severity defines a delivery contract

A severity label should set the response time, channel, and authority of the recipient, not express the rule author's emotions. The words critical, warning, and info are useless until the team agrees on what happens to each class.

A practical scheme starts with consequences:

• page: harm is happening now or will begin before the next working window, so immediate action is required;
• ticket: there is enough time to assign an owner and fix the issue during working hours;
• info: retain the event for search and analysis without sending it to a person;
• security: use a separate route under the established procedure if access or data may be affected.

Do not raise severity because "otherwise no one will look." That admits the ticket queue is broken. The urgent channel soon inherits the same problem, while real outages disappear among attempts to attract attention.

Keep scope and urgency separate. The loss of one node in a redundant cluster can have a large technical scope but low user urgency. An error in a small authentication component can stop every login and require an immediate page. Build the route around time to harm and the system's ability to recover, not around server size or a loud component name.

Every class needs a limit on repeats. For page, require acknowledgement and escalation when the owner does not respond. For ticket, one opening and updates on material changes are enough. info should not leak into email unnoticed. An overflowing folder also trains people not to read system messages.

The same rule may have a different severity in different environments, but explain the difference. A node failure in a test environment usually creates a ticket, while failure of the only node in a production service pages someone. Do not encode the environment in the rule name. Use a stable label and explicit routing policy, or copies will gradually diverge in logic.

Check whether the recipient can perform the action. A support desk can record user impact but may not have cluster access. A platform engineer can see the infrastructure but may not decide to suspend a financial operation. During a complex outage, one route appoints an incident lead and other teams join through the procedure, instead of receiving identical independent pages.

Missing data does not mean a healthy state

A query with no result, a disappeared series, and an evaluation error indicate different failures. Turning any of them into zero is convenient but dangerous. The monitoring system may report health at the exact moment it has stopped seeing the service.

Grafana distinguishes No Data, Error, and a missing individual series. No Data occurs when a query succeeds but returns no points. Error means the query failed or timed out. When one series disappears, some data remains, so the overall rule may keep evaluating and say nothing about a particular region or instance.

Set the policy according to the meaning of the metric. A lack of points between runs is normal for a periodic batch job. For the heartbeat of a payment gateway, absence is itself a symptom. In Prometheus, absent_over_time(metric[5m]) detects the complete absence of a series in the window, but it does not always preserve the label set needed to name the missing instance. In a dynamic environment, a hard-coded instance list becomes stale quickly.

Do not send every DatasourceError to the application owner. A failure in a shared data source may produce a copy for every rule. Group those events by data source identifier and route them to the monitoring team, while checking user service health through an independent external signal. "Keep last state" is useful for a brief interruption, but prolonged silence from the source needs its own alert.

Monitoring also needs delivery testing. The official Prometheus practice recommends an end-to-end path test, such as a periodic signal traveling from the source through Prometheus and Alertmanager to the recipient, instead of separate messages about each link. That test answers the main question: can the channel call a person right now?

Check metric freshness separately from its value. A last-known temperature of 40 degrees does not prove the equipment is cool now if the sensor has been silent for two hours. Add the time of the last successful measurement and an acceptable data age. For critical sources, collect freshness independently so that one agent failure cannot hide both the metric and evidence of its absence.

Distinguish a query error from an observed service error in the name and route. ApiHighErrorRate and ApiAlertQueryFailed require different actions and often different owners. If you merge them under one name, the on-call engineer sees a familiar heading with the opposite meaning: the application may be fine while only the calculation is broken.

Change rules through an observed experiment

A threshold change is complete only after historical testing and a limited observation period. Compare the old and new expressions over the same intervals: how many real incidents each caught, how often each prompted action, how quickly it fired, and how many separate deliveries it created.

Do not change the threshold, window, for, grouping, and route at the same time. If the noise disappears, you will not know why. If a necessary signal vanishes, restoring sensitivity will also be difficult. Make one meaningful change, record the expected effect, and set a review date.

For a new rule, first enable evaluation without urgent delivery. Let it write state to a log or test receiver. Run a known bad interval, a normal traffic peak, a deployment, and missing data through it. Then enable the production route and set a review date. This is not an indefinite "shadow mode." Without a date, a rule can spend years protecting nothing.

Once a month, review rules that fired often, never fired, or remained disabled. Zero events do not prove quality. A condition may protect against a rare risk, or it may have been incompatible with metric names for months. The owner should confirm the query, route, and action after major architectural changes.

Store the reason next to each rule change: an incident identifier, expected noise reduction, missed-event risk, and rollback method. A quarter later, no one will remember why the number is 10m, while the record will show which safe spike it was meant to filter. If the traffic profile changes, you can recalculate the decision instead of debating it again from memory.

The post-launch assessment should answer two questions. Did deliveries per incident fall, and did detection time for meaningful outages remain intact? Noise reduction paired with more missed events is not an improvement. Good tuning cuts needless interruptions without buying quiet at the cost of invisible harm.

In GSE infrastructure projects, we connect the monitoring scheme to the actual data center components, areas of responsibility, and support model rather than leaving it as a set of factory thresholds. This matters especially in mixed environments, where one incident crosses hardware, platform, and application services.

Keep only rules in the urgent channel that can complete this sentence: "If this fires, the on-call engineer does this before the stated deadline." Do not lose the remaining signals. Give them an honest destination: a ticket, dashboard, log, or condition that automation fixes without a person.