IncidentFlow architecture
Seven .NET services that turn repeated production errors into incidents, then correlate, notify and record them. Every part below was checked against the code.
Overview
API Gateway
A YARP reverse proxy fronts the services. It does nothing clever, and that is the point. Routing and cross-cutting concerns live here, so no service behind it needs to know how the outside world reaches it.
Ingestion
Ingestion accepts five signal types (errors, logs, metrics, deployments and health checks), stores each one and returns 202 Accepted rather than waiting for a verdict. Every request can carry an idempotency key, because the systems that raise alerts are exactly the ones most likely to retry.
RabbitMQ
Signals are published to RabbitMQ through MassTransit. Ingestion returns as soon as a signal is stored and published, so an alert storm queues up instead of blocking the way in. Absorbing the spike is the job; passing it along would be the failure.
Detection
Detection is where a pile of signals becomes a judgement. Each error is added to a Redis sorted set keyed by service, environment and fingerprint and scored by timestamp, and anything older than five minutes is trimmed. When five repeats survive inside that window, the detector publishes an anomaly and deletes the key, so one sustained fault produces one anomaly rather than one per signal. Redis is deliberately transient here: losing it costs a window, not data.
Incident lifecycle
The Incident service consumes anomalies and owns the lifecycle from there: created, acknowledged, resolved, closed. It is the only service that decides what an incident is, and it publishes every state change as its own event.
The fan-out
Those lifecycle events fan out to three independent consumers. Correlation groups related incidents by deployment or fingerprint, because “did this start when we shipped?” is worth answering automatically. Notification handles delivery. Timeline keeps a record of what happened when. None of them can block the others, and none of them can stop an incident being created.
What is not built yet
There is no sign-in yet: identity and access wait until the core workflow settles. The operator dashboard is planned and does not exist, so everything above is API and worker surface only. The gateway still references IncidentFlow.Core, the monolith from before the services were split out, and retiring it is outstanding work.
Flows
A repeated error becomes an incident
Ingestion answers straight away and everything after that happens in the background. Event arrows are delivered through RabbitMQ. Timeline also records the original signal and the anomaly, which are left out here to keep the picture readable.
