← IncidentFlow

IncidentFlow architecture

Seven .NET services that turn repeated production errors into incidents, then correlate, notify and record them. Every part below was checked against the code.

Overview

Select any part of the diagram to read about it
IncidentFlow service topologyA YARP API gateway fronts an Ingestion service that accepts five signal types and returns 202 Accepted. Signals are published over RabbitMQ using MassTransit. A Detection worker counts repeats in a five-minute Redis sliding window and publishes an anomaly when a threshold is crossed. An Incident service consumes that anomaly and owns the incident lifecycle. Its lifecycle events fan out to three consumers: a Correlation worker that groups incidents by deployment or fingerprint, a Notification worker, and a Timeline projection.API Gateway YARPIngestion 5 signal typesRabbitMQ MassTransitDetection Redis windowIncident lifecycleCorrelation deploy · fingerprintNotification MassTransitTimeline projection

API Gateway

A YARP reverse proxy fronts the services. It does nothing clever, and that is the point. Routing and cross-cutting concerns live here, so no service behind it needs to know how the outside world reaches it.

Ingestion

Ingestion accepts five signal types (errors, logs, metrics, deployments and health checks), stores each one and returns 202 Accepted rather than waiting for a verdict. Every request can carry an idempotency key, because the systems that raise alerts are exactly the ones most likely to retry.

RabbitMQ

Signals are published to RabbitMQ through MassTransit. Ingestion returns as soon as a signal is stored and published, so an alert storm queues up instead of blocking the way in. Absorbing the spike is the job; passing it along would be the failure.

Detection

Detection is where a pile of signals becomes a judgement. Each error is added to a Redis sorted set keyed by service, environment and fingerprint and scored by timestamp, and anything older than five minutes is trimmed. When five repeats survive inside that window, the detector publishes an anomaly and deletes the key, so one sustained fault produces one anomaly rather than one per signal. Redis is deliberately transient here: losing it costs a window, not data.

Incident lifecycle

The Incident service consumes anomalies and owns the lifecycle from there: created, acknowledged, resolved, closed. It is the only service that decides what an incident is, and it publishes every state change as its own event.

The fan-out

Those lifecycle events fan out to three independent consumers. Correlation groups related incidents by deployment or fingerprint, because “did this start when we shipped?” is worth answering automatically. Notification handles delivery. Timeline keeps a record of what happened when. None of them can block the others, and none of them can stop an incident being created.

What is not built yet

There is no sign-in yet: identity and access wait until the core workflow settles. The operator dashboard is planned and does not exist, so everything above is API and worker surface only. The gateway still references IncidentFlow.Core, the monolith from before the services were split out, and retiring it is outstanding work.

Flows

A repeated error becomes an incident

Ingestion answers straight away and everything after that happens in the background. Event arrows are delivered through RabbitMQ. Timeline also records the original signal and the anomaly, which are left out here to keep the picture readable.

A repeated error becomes an incident1. Client to Gateway: POST /api/v1/signals. 2. Gateway to Ingestion: forward. 3. Ingestion to Gateway: 202 Accepted (reply). 4. Gateway to Client: 202 Accepted (reply). 5. Ingestion to Detection: ErrorSignalReceived. 6. Detection to Incident: AnomalyDetected (5th repeat in 5 min). 7. Incident to Correlation: IncidentCreated. 8. Incident to Notification: IncidentCreated. 9. Incident to Timeline: IncidentCreated. 10. Correlation to Timeline: CorrelationFound.ClientGatewayIngestionDetectionIncidentCorrelationNotificationTimeline1. POST /api/v1/signals2. forward3. 202 Accepted4. 202 Accepted5. ErrorSignalReceived6. AnomalyDetected (5th repeat in 5 min)7. IncidentCreated8. IncidentCreated9. IncidentCreated10. CorrelationFound
A repeated error becomes an incident