Select a result to preview

Incident Management

Document: 39-incident-management.md
Version: 1.0.0

Purpose

This document defines the incident management standards used across all Rundown projects.

Incident management provides a structured process for detecting, responding to, resolving, and learning from production incidents. A consistent incident management process minimizes downtime, improves communication, and strengthens the reliability of future releases.

Incident Philosophy

Incidents are opportunities to improve the system.

The objectives of incident management are to:

Incident management should focus on learning and system improvement rather than assigning blame.

Incident Definition

An incident is any unplanned event that negatively affects:

Not every bug is an incident.

An incident is defined by its operational impact.

Incident Lifecycle

Every incident follows the same lifecycle.

Detection

↓

Assessment

↓

Containment

↓

Resolution

↓

Verification

↓

Post-Incident Review

↓

Improvement Actions

Every stage should be documented.

Detection

Incidents may be identified through:

Automated monitoring should identify incidents whenever practical.

Assessment

Initial assessment should determine:

Assessment guides the appropriate response.

Severity Levels

Critical (P1)

Examples:

Requires immediate response.

High (P2)

Examples:

Requires prompt investigation.

Medium (P3)

Examples:

Address according to operational priority.

Low (P4)

Examples:

Address through planned maintenance.

Containment

The immediate objective is to prevent further impact.

Containment may include:

Containment should prioritize service stability.

Resolution

Resolution should:

Temporary workarounds should be replaced by permanent solutions where practical.

Communication

Communication during incidents should be:

Clients should receive updates appropriate to the incident's impact and severity.

Avoid speculation until facts have been verified.

Post-Incident Review

Every significant incident should include a structured review.

The review should answer:

The objective is organizational learning.

Improvement Actions

Every post-incident review should generate actionable improvements.

Examples include:

An incident is considered fully closed only after agreed improvement actions have been recorded.

Documentation

Every incident record should include:

Incident documentation supports future operational learning.

Engineering Decision

Blameless Incident Management

Status: Accepted

Decision

Rundown follows a structured, blameless incident management process focused on rapid recovery, root cause analysis, and continuous improvement.

Reasoning

A blameless approach:

Incident Checklist

When responding to an incident, verify:

Summary

Incident management provides the operational framework for responding to production issues in a disciplined and repeatable manner.

By prioritizing rapid recovery, transparent communication, and continuous learning, Rundown transforms operational incidents into opportunities to improve systems, processes, and engineering practices.