Skip to content

Failures, issues and alerts

A monitor failure is not a separate notification stream. It becomes an ordinary issue, created through the same pipeline an SDK exception goes through, and everything downstream of issues applies to it unchanged.

What gets created

FailureIssue titleTags on the event
errorMonitor "slug" reported a failuremonitor_slug, plus environment and release if the SDK sent them
missedMonitor "slug" missed its expected check-inmonitor_slug
timeoutMonitor "slug" exceeded its max runtimemonitor_slug

Every monitor issue is level error.

The three are separate issues, grouped on separate fingerprints. A monitor that fails, recovers, then later stops running entirely produces one error issue and one missed issue, not one issue that changes character. Grouping is per slug per failure kind, so all of a monitor's missed runs collapse into a single issue that accrues events.

Only error carries environment and release, because those come off the check-in payload. A missed run has no payload; nothing arrived. That matters when you write tag filters: an environment = production filter will never match a missed-check-in issue.

Lifecycle

first failure ──▶ issue opens (unresolved)

      ├── failure repeats ──▶ same issue, event count climbs

      └── job succeeds ──▶ issue resolves automatically

                 └── fails again ──▶ same issue reopens as a regression

Recovery is automatic. One successful check-in on a monitor that was in a bad state resolves the open error, missed and timeout issues for that slug together.

That auto-resolve has a consequence for how you write alert rules, covered next.

Alert rules

Alert rules in the panel. A rule is conditions, then filters, then actions. Monitor issues are matched by ordinary issue alert rules; nothing monitor-specific is needed.

Conditions

ConditionFires when
A new issue is first seenThe issue is created, once ever for that fingerprint
A resolved issue regressesA resolved issue fails again
Issue is seen more than N times in an intervalThe event count crosses a threshold in a window

"First seen" alone will notify you exactly once

Monitor issues have stable fingerprints and resolve themselves on recovery. The first time nightly-reports is missed, the issue is created and "first seen" fires. The job runs again, the issue resolves. Next month it breaks again, and the same issue regresses rather than being created, so a rule that only listens for "first seen" stays silent.

Always pair the two. Conditions are OR'd, so adding both means "tell me when it breaks, and when it breaks again".

For a job that is genuinely flaky, add a frequency condition instead of paging on every event: "seen more than 3 times in 60 minutes" turns three consecutive missed hourly runs into one notification.

Filters

Every configured filter must pass. The useful one here is a tag filter on monitor_slug:

KeyValue
monitor_slugnightly-reports

That is how you route one critical job to a pager and let everything else go to a Slack channel. Two rules, one with the tag filter and a phone-shaped action, one without.

Filters are evaluated against the triggering event, not the issue's accumulated history. An environment tag filter therefore excludes missed and timeout issues entirely, since those events carry no environment.

Actions

Email, Slack webhook, Discord webhook, or a generic webhook with an optional signing secret. A rule can have several.

Cooldown

cooldown_minutes is the minimum gap between repeat notifications for the same issue and rule. For an every-minute job that has stopped, this is the difference between one message and sixty. The default comes from the platform config and can be raised per rule.

A rule that works

The setup most teams want, for a project with a handful of scheduled jobs:

  1. Rule type Issue alert, scoped to the project.
  2. Conditions: A new issue is first seen and A resolved issue regresses.
  3. Filters: tag monitor_slug = the job you care about, or no filter to catch every issue in the project including crashes.
  4. Actions: Slack webhook.
  5. Cooldown: 60 minutes.

Then break it on purpose once, as in the quickstart. An untested alert rule is not a monitoring system, it is an intention.

The agent

New and regressed issues are queued for an agent scan, monitor issues included.

Set expectations low here. The agent works from a stack trace and a repository, and a missed check-in has neither. It knows a job stopped reporting; it has no way to see that your crontab entry was dropped by a config-management change on a host it cannot read. For error check-ins the picture is better, because the exception your job also reported is a real issue with a real trace, and that is the one worth handing over.

Read a monitor issue as a page, not as a work item for the agent.

Uptime checks, briefly

Uptime checks are the sibling feature: Monitoring → Uptime checks, full CRUD, poll a URL on an interval and open an issue after N consecutive failures. Same issue pipeline, same alert rules.

They answer a different question. A monitor asks whether the job ran. An uptime check asks whether the endpoint responds. A job that runs on time and does the wrong thing is invisible to both.

Parsemend, by MAVA Design