Incidents

The incident ticket is the entry point of alerting. Unlike a process based on monitoring dashboards that require constant surveillance by operators, ticket-based alerting notifies responders proactively, and only when a problem is detected. It therefore reduces the resources needed to monitor a system, and improves detection times.

What does an Incident look like?

On Astry, any problem is qualified as an Incident. It takes the form of a ticket, which includes:

• a description of the problem
• a status (Open, Acknowledged, or Resolved)
• a priority (from 1 - critical - to 5 - non-critical)
• comments to track the incident
• events on the incident
• and metadata (tags, identifiers, creation date, ...)

Workflow when an incident is raised

When an incident occurs, a ticket is raised (via email, API, Prometheus, ... : see Integrations) and appears with the Open status. The following steps are then typically followed when handling the incident:

1. You acknowledge the incident to let your team members know that it has been seen and that someone has started investigating.
2. You analyze the problem, tracking progress on the incident ticket.
3. You bring the impacted service back up if possible (without necessarily fixing the root cause).
4. You resolve the incident on Astry.
5. You may then write an incident report and track long-term resolution tasks (fixing the root cause if it hasn't been addressed yet).

Incident states

An incident can have three different states:

1. Open: this is the default state when the incident is created and has not yet been seen or acknowledged. If an incident stays in the Open state, the on-call team member will keep being alerted (at regular intervals, according to their notification strategy).
2. Acknowledged: once the incident is acknowledged, it moves to the Acknowledged state. In this state, the on-call team member is no longer alerted. The incident can still move back to the Open state, either manually or automatically via escalation policies. In that case, the on-call member will be alerted again.
3. Resolved: once the incident is resolved, it cannot change state again without manual action. Naturally, the on-call member is no longer notified.

Note: Astry supports incident deduplication. This means that if two incidents of the same type are raised while the first one is still in the Open or Acknowledged state, the second incident will not be created, since Astry considers it to actually be a second alert for the same incident (this avoids opening multiple incident tickets for the same problem and being spammed). In that case, the second alert will be shown on the incident's page. Once the incident moves to the Resolved state, if the alert occurs again, a new incident will be created. Astry relies on a correlationId attribute to detect deduplication (see the documentation for each integration in the Astry tool to set this value).

Escalation policies

Astry lets you define flexible escalation policies. Based on a set of criteria you define (tags, priority, ...), incidents assigned to a team that remain in the Open state for more than an hour (or more than 3 days, etc.) will be automatically assigned to the team of your choice. This guarantees that an incident will always be detected and handled. Of course, you can also manually escalate any incident by assigning it to any team.

The image below describes an example of an incident being escalated from a team N1 to a team N2, either automatically or manually.

Your organization and processes may require different escalation policies. For example, if you want a managerial escalation (the incident is escalated from the on-call employee to their manager if no action is taken after a few hours, then to the next manager up, and so on) rather than a technical escalation as in the example above, this is possible through configuration. For example, you could define an escalation policy like this one: