Skip to content

adr-incident-manager-governance-workflows.md

Latest commit

 

History

History
305 lines (219 loc) · 14.7 KB

adr-incident-manager-governance-workflows.md

File metadata and controls

305 lines (219 loc) · 14.7 KB
Raw

Integrate Incident Manager in the Governance Workflows Framework

ADR-#: 1 Authors: Pablo Takara Reviewers: Teddy Crépineau, Ram Narayan Balaji Date: February 27, 2026 Status: Proposed

Migrate incident lifecycle into a governance workflow using a new Task Lifecycle Node. The node uses OpenMetadata tasks as the source of truth (not Flowable UserTask), receives a template with configurable statuses, and exposes each status transition to the main workflow graph via process variables. Users wire hooks on any transition using standard edges. Non-terminal statuses loop back; terminal statuses auto-close the task.

Context

The Incident Manager handles the lifecycle of data quality incidents in OpenMetadata. When a test case fails, an incident is created; it progresses through New → Ack → Assigned → Resolved as humans triage it.

Today, this lifecycle is a switch statement in TestCaseResolutionStatusRepository.storeInternal(). It handles state transitions, task creation, assignment, and resolution. The state machine is simple, correct, and performant, but it has no extension points. Adding a behavior like "on Assigned, notify via Slack" or "on New, auto-assign to table owner" requires modifying repository code, testing, and redeploying.

Meanwhile, OpenMetadata ships a governance workflows framework built on Flowable BPM. It is fully configurable via REST API and UI. Users configure workflows as abstract trigger → nodes → edges graphs (they never see BPMN XML). The backend compiles these to Flowable process definitions automatically via NodeFactory and MainWorkflow.

The two systems live side by side but do not interact.

Additionally, the task refactor promotes tasks to first-class entities with standard ChangeEvents. This enables Flowable to be notified of every status transition — not just resolution — unlocking configurable hooks on any transition from day one.

Specific Gaps

  1. No auto-close when tests pass. TestCaseResultRepository.setTestCaseResultIncidentId() sets incidentId = null when a test succeeds but never resolves the incident or closes its task.
  2. No auto-assign on incident creation. Every incident starts in New and requires manual acknowledgement.
  3. No extensibility. Organizations cannot define configurable rules like "on any status change, execute action X" without code changes.
  4. Fixed lifecycle. The New → Ack → Assigned → Resolved states are hardcoded. Organizations with different triage processes have no way to customize.
  5. No incident TTL. No mechanism to auto-close stale incidents.

Enterprise scale context

  • 5M assets, 10-30% with data quality tests = 500K-1.5M test cases
  • At 2-5% failure rate = 10K-75K concurrent open incidents (typical)
  • getOrCreateIncident() enforces one unresolved incident per test case

Use Cases

UC-1 — Auto-close incident when test passes The system automatically resolves the open incident (reason: AutoResolved) and closes its task. No human intervention required.

UC-2 — Auto-assign incident on creation When a new incident is created, the system automatically assigns it to a configured user or team.

UC-3 — Auto-close stale incidents (TTL) An incident open longer than a configurable deadline is automatically resolved (reason: Expired).

UC-4 — User-defined hooks on any status transition Users wire follow-up steps (notifications, Jira tickets, etc.) on any status change via workflow edges — no code changes.

Decision

Task Lifecycle Node

A new governance workflow node that does NOT use Flowable's BPMN UserTask. It creates an OpenMetadata task, waits for status changes via IntermediateCatchEvent, and exposes each status to the parent workflow for routing.

Internal BPMN structure:

┌─ SubProcess ──────────────────────────────────────────────────────┐
│                                                                    │
│  [Start] → [Setup] → [Gateway: created?]                          │
│               │          no → [End: skip]                          │
│               │          yes ↓                                     │
│               │        [IntermediateCatchEvent: wait]               │
│               │          ↓ message with {status}                   │
│               │        [Gateway: terminal?]                        │
│               │          yes → [CloseTask] → [SetResult] → [End]  │
│               │          no  → [SetResult] → [End]                 │
│               │                                                    │
│               │  Setup (idempotent):                               │
│               │    • Check for existing open incident              │
│               │      → if exists with active process: skip         │
│               │      → if orphaned process: terminate it           │
│               │    • Create incident record (New)                  │
│               │    • Create OM task                                │
│               │    • Auto-assign (from template config)            │
│               │    • Set process variable omTaskId = task UUID     │
│                                                                    │
│  + [TTL Boundary Timer: configurable, interrupting]                │
│      → [AutoResolve via repository] → [End]                       │
└────────────────────────────────────────────────────────────────────┘

Node config:

{
  "type": "taskLifecycleNode",
  "config": {
    "template": "incident",
    "statuses": ["New", "Ack", "Assigned", "Resolved"],
    "terminal": ["Resolved"],
    "responsibles": { "source": "tableOwner" },
    "ttl": "P30D"
  }
}

The node:

  1. Setup — Creates the OM task (idempotent on re-entry). Sets omTaskId process variable.
  2. Wait — IntermediateCatchEvent with messageExpression="${omTaskId}". Subscribes to a message named after the task UUID (~2 Flowable DB rows).
  3. On message — Evaluates whether the received status is terminal.
  4. Terminal — Closes the OM task (idempotent), sets {nodeName}_result at parent scope, subprocess exits.
  5. Non-terminal — Sets {nodeName}_result at parent scope, subprocess exits. Parent-level edges route back to the node.

Status exposed via graph edges (with cycles)

Status is set as a Flowable process variable when the subprocess exits. Parent-level edges condition on this variable. Non-terminal edges loop back to the node.

         ┌────── "ack" ───────────────────────────┐
         │  ┌─── "assigned" → [NotifySlack] ──────┤
         ▼  ▼                                      │
[Start] → [ManageIncident] ── "resolved" → [End]

Workflow definition example:

{
  "name": "incident-lifecycle",
  "trigger": {
    "type": "eventBasedEntity",
    "config": {
      "entityTypes": ["TestCase"],
      "events": ["Updated"],
      "filter": { "TestCase": { "==": [{"var": "testCaseStatus"}, "Failed"] } }
    }
  },
  "nodes": [
    { "type": "startEvent", "name": "start" },
    { "type": "taskLifecycleNode", "name": "incident", "config": {
        "template": "incident",
        "statuses": ["New", "Ack", "Assigned", "Resolved"],
        "terminal": ["Resolved"],
        "responsibles": { "source": "tableOwner" },
        "ttl": "P30D"
    }},
    { "type": "automatedTask", "subType": "sinkTask", "name": "notifySlack" },
    { "type": "endEvent", "name": "end" }
  ],
  "edges": [
    { "from": "start", "to": "incident" },
    { "from": "incident", "to": "incident", "condition": { "status": "Ack" } },
    { "from": "incident", "to": "notifySlack", "condition": { "status": "Assigned" } },
    { "from": "notifySlack", "to": "incident" },
    { "from": "incident", "to": "end", "condition": { "status": "Resolved" } }
  ]
}

Message delivery via task ChangeEvents

With the task refactor, tasks emit ChangeEvents on status changes. These drive message delivery to Flowable:

  1. Task status changes (via REST API / storeInternal)
  2. ChangeEvent emitted
  3. Listener correlates message to waiting IntermediateCatchEvent

The OM task is already updated before the message fires. If correlation fails, the task state is correct — Flowable catches up on the next status change.

Mechanism TBD: Listener on task ChangeEvents (clean separation) vs direct hook in task status update code (fewer hops).

What the workflow controls vs the repository

Action Who handles it
Task creation Node setup phase (idempotent)
Status changes (Ack, Assigned, etc.) Repository — synchronous, unchanged
Resolution Repository — synchronous, unchanged
Task closure Both — node closes on terminal, repository may also close. Idempotent.
Flowable notification Task ChangeEvent → message to IntermediateCatchEvent
Follow-up hooks Workflow edges — user-configurable
TTL auto-resolve Boundary timer on node
Auto-close on test pass Separate short-lived workflow

Why this approach

  1. Hooks on any transition. Status exposed to parent graph → users wire follow-up steps via edges.
  2. Configurable lifecycle. Template defines statuses and terminal set. No hardcoded lifecycle.
  3. OM task is source of truth. No BPMN UserTask. ~2 DB rows per task vs ~5-10.
  4. Repository stays in the critical path. All transitions are synchronous. Flowable is notified after the fact. If Flowable is down, transitions still succeed.
  5. Unified abstraction. Same node type for incidents, approvals, certifications — different templates.

Consequences

Positive

  • Hooks on any status transition without code changes.
  • Configurable lifecycle from day one via template config.
  • Lightweight — ~2 Flowable DB rows per task (IntermediateCatchEvent).
  • Safe — repository owns all transitions synchronously; Flowable is follow-up only.
  • Default workflow replicates current behavior and ships enabled.
  • Unified abstraction — incidents, approvals, certifications share one node type.

Negative

  • MainWorkflow compiler must support cycles. Today it assumes a DAG. Biggest technical risk.
  • More Flowable interactions. Every status change sends a message (vs resolution only). ~225K correlations over lifetime of 75K incidents with ~3 transitions each.
  • Task refactor dependency. Fallback: direct reportOutcome() from storeInternal() if not ready.

Neutral

  • REST API surface unchanged.
  • TestCaseResolutionStatus schema changes minimally (add AutoResolved, Expired reasons).
  • Resolution business logic in the repository is unchanged.

Alternatives Considered

Bookends only (no intermediate state hooks)

Handle only creation + resolution in the workflow. Intermediate states stay entirely in storeInternal().

Not chosen: Users cannot wire hooks on Ack/Assigned. The task refactor makes full lifecycle hooks possible now — deferring them means two migrations.

Internal loop (cycle hidden inside SubProcess)

The message loop lives inside the node. Status exposed only on terminal exit. Outer graph stays a DAG.

Not chosen: Users cannot wire hooks on non-terminal transitions. The point is exposing every status change to the parent graph.

Resolution through Flowable (not fire-and-forget)

Route resolution through the Flowable process.

Not chosen: Puts Flowable in the critical path. If Flowable is slow/down, resolution is blocked.

Extend state machine with Java hooks

Rejected: Parallel automation system, requires code changes for every new behavior.

CMMN (Case Management)

Rejected: Zero existing infrastructure, overkill.

Design Choices

IntermediateCatchEvent with messageExpression

messageExpression="${omTaskId}" gives unique-per-instance subscriptions. EventSubscriptionQuery.eventName(taskId) is an indexed lookup. No MessageCorrelationBuilder (doesn't exist in Flowable 7.2.0).

Idempotent setup on loop re-entry

When non-terminal edges loop back, Setup detects the existing task and reuses it. Safe for any number of loops.

Terminal auto-close — both sides

storeInternal(Resolved) closes the task. The node's CloseTask also closes on terminal status. Both are idempotent. This handles TTL (node-initiated) and human resolution (repository-initiated) uniformly.

Business key = test case FQN

Enables idempotent creation, fire-and-forget termination, auto-close correlation.

Governance-bot loop prevention

WorkflowEventConsumer skips events from governance-bot. The workflow runs as governance-bot, so its own events don't re-trigger workflows.

Open Questions

  • Message delivery mechanism: Listener on task ChangeEvents vs direct hook in task status update.
  • TestCaseResult.incidentId linking: If creation moves to async workflow, test result may store before incident exists. Recommendation: keep getOrCreateIncident() synchronous.
  • Cycle validation: Should the compiler enforce that every non-terminal edge path routes back to a task node?

Risks

Risk Impact Mitigation
Cycle support in MainWorkflow Blocks the design Spike early. Workaround: invisible gateway node.
Task refactor not ready No ChangeEvents for message delivery Fall back to direct reportOutcome() from storeInternal()
Race condition Message lost during follow-up execution EventSubscriptionQuery returns null → skipped. Java-side buffer later.
ACT_RU growth ~2 rows per open incident 75K incidents = 150K rows. Measure in hardening phase.
Process orphaning Never-resolved incidents linger TTL handles deadlines. Batch sweep for the rest.

Follow-up Work

  1. Batch sweep for orphaned processes.
  2. Migrate UserApprovalTask (glossary) to same node type with template: "approval".
  3. SLA timer escalation — optional boundary timer using same infrastructure as TTL.

References

  • TestCaseResolutionStatusRepository.storeInternal() — Current state machine
  • WorkflowHandler.java — Flowable ProcessEngine, message delivery
  • MainWorkflow.java — BPMN compiler (needs cycle support)
  • UserApprovalTask.java — Current UserTask pattern (being replaced)
  • NodeFactory.java — Node type registration
  • WorkflowEventConsumer.java — Event routing, governance-bot loop prevention