Firewatch architecture

Status and guiding constraint

Firewatch is a modular on-call and incident-alerting product. Schedules, escalation execution, alert ingestion, incidents, runbooks, safe message templates, and email, Twilio, Slack DM, and signed-webhook delivery are implemented across independently deployable processes.

Firewatch has one product implementation with multiple independently deployable processes. The public repository is the source of truth for the domain, application behavior, HTTP API, worker, web application, and portable deployment artifacts. Self-hosted and managed installations use the same product behavior.

flowchart TB accTitle: Firewatch runtime architecture accDescr: The React web app calls the API, which persists work to PostgreSQL for the worker to deliver through provider abstractions. Web["React / Vite Web"] -->|"HTTP /api/v1"| Api["ASP.NET Core API"] Mobile["Future native clients"] -.->|"same public API"| Api Api --> Db["PostgreSQL<br/>state + outbox"] Db --> Worker[".NET Worker"] Worker --> Delivery["Delivery-provider<br/>abstractions"] Delivery --> Channels["Email · SMS · voice<br/>Slack · signed webhooks"] Worker --> Queue["Queue-provider<br/>abstraction"] Queue --> PgQueue["PostgreSQL queue<br/>self-hosted default"] Queue -.-> Sqs["AWS SQS<br/>optional provider"] Api --> Slack["Slack OAuth<br/>member mapping"] Slack -.-> Channels

Firewatch keeps API, worker, and provider boundaries consistent across deployment models.

The arrows describe the durable asynchronous path, not separate product implementations. The API and worker share libraries and a database schema but run, scale, restart, and deploy independently.

Repository boundaries

Project	Responsibility	May depend on
Firewatch.Domain	Pure business types and rules	Nothing outside the base class library
Firewatch.Contracts	Versioned HTTP and message DTOs	Base class library only
Firewatch.Application	Use cases, ports, and orchestration	Domain and contracts
Firewatch.Infrastructure	EF Core context, PostgreSQL persistence, outbox, repositories, DI	Application, contracts, queue abstractions, EF Core/Npgsql
Firewatch.Queueing.Abstractions	Small publishing/consuming contracts and message identity conventions	Base class library and stable contracts as needed
Firewatch.Queueing.Postgres	Portable PostgreSQL queue/outbox processing	Queue abstractions and PostgreSQL infrastructure
Firewatch.Queueing.Sqs	SQS publishing, long polling, acknowledgement, and visibility leases	Queue abstractions and AWS SDK for SQS
Firewatch.Storage.S3	Optional runbook-image object storage	Application asset-store port and AWS SDK for S3
Firewatch.Email.Ses	Optional transactional email through the SESv2 API	Application email port, infrastructure options, AWS SDK
Firewatch.Api	REST endpoints, HTTP concerns, health, OpenAPI, composition root	Application and concrete runtime adapters
Firewatch.Worker	Hosted consumers, retries, outbox work, composition root	Application and concrete runtime adapters

The API never exposes EF Core entities. Versioned types from Firewatch.Contracts form the wire boundary. Endpoint registration is kept out of Program.cs so the composition root stays readable.

This is a modular monolith: modules share a release and relational data store, while dependencies point inward toward domain and application code. It avoids network boundaries inside the product until there is evidence that one is needed.

Deployable processes

API

Firewatch.Api owns synchronous HTTP handling under /api/v1, OpenAPI, exception translation, request/correlation IDs, structured JSON logs, and liveness and readiness endpoints. It is stateless apart from PostgreSQL and can be scaled horizontally after migrations have been run.

Worker

Firewatch.Worker owns asynchronous execution. It polls or consumes through the selected queue provider, dispatches durable outbox work, records completion, and honors cancellation during shutdown. API replicas do not run worker loops, and worker replicas do not serve the public API.

Web

apps/web is a standalone React/Vite application. It communicates only through the public HTTP API by using packages/api-client; it does not reference .NET projects or persistence types. The production web container serves static files and proxies same-origin /api, /health, and /openapi requests to the API upstream selected at deployment time.

Authentication and tenant authorization

ASP.NET Core Identity owns local user credentials and account-security behavior. A user has a first name, last name, and normalized unique email address. Users do not own tenant data directly: an organization membership connects a user to an organization and records whether that member is an organization administrator. Organization-scoped use cases resolve the active tenant from an authenticated membership and enforce that boundary in application and persistence queries; they do not trust an arbitrary organization identifier supplied by a client.

The browser signs in with an HTTP-only, Secure authentication cookie. Requests that change state also require a CSRF token, including requests made through the same-origin web proxy. Cookie contents are protected by the ASP.NET Core Data Protection key ring stored in the shared PostgreSQL database, so horizontally scaled API replicas and replacement containers share the same keys. Production operators remain responsible for protecting that key ring at rest and rotating its deployment-managed protecting key.

Organization API keys are tenant-scoped machine credentials rather than user sessions. An organization administrator can create and revoke them. The full secret is returned only at creation, while Firewatch stores a non-reversible hash for constant-time verification. Authorization uses the key’s organization and active/expiry state and never treats an API key as a global administrator credential. Automated organization provisioning uses a separate workload-token policy and does not reuse tenant API keys.

Services and incidents

A monitored service is an organization-scoped thing that can degrade or fail. Its normalized name is unique within the organization. A service always begins with at least one owner: the member who created it as a primary owner. Owners carry a persisted primary or secondary label; multiple owners may use either label, API responses order primary owners first, and both labels grant the same service-management permission. Service owners and organization administrators can edit the service, add or remove active members as owners, and archive it. The domain rejects removal of the final owner, and organization-member removal is blocked when it would orphan an active service. Archival is non-destructive: archived services retain their owners, incidents, alert events, and runbook associations but cannot receive new incidents or be edited. A service may select one escalation policy for newly created incidents. Alert sources and runbooks reference this same service identifier rather than introducing separate copies of the service concept.

An incident belongs to exactly one service and organization. It has a global, human-readable INC-000001 reference, severity (SEV1, SEV2, or SEV3), status (triggered, acknowledged, or resolved), and a source kind. Manual declaration accepts the actual incident start time rather than assuming the API request time. Resolution likewise records its selected time and an optional reason. Manual declarations and normalized alert ingestion create the same incident aggregate; alert-created incidents retain their source events and correlation identity.

Every incident owns a chronological timeline. Creation, acknowledgement, resolution (including its optional reason), reopening, responder notes, severity changes, escalation stages, notification deliveries, failed attempts, and skipped notifications append new immutable entries containing the actor, status transition, optional severity transition, message, and timestamp. Reopening begins a new response cycle without removing the prior cycle’s events. The domain exposes no mutation or removal operation for entries, and the EF context rejects modified or deleted timeline rows. Incident listing performs tenant-scoped server-side search, filtering, and pagination; dashboard counts and recent incidents are computed from the same persisted data. Tenant identity is repeated on timeline rows and enforced through composite foreign keys so an incident, service, and timeline cannot cross organization boundaries.

Time zones, schedules, and escalations

An organization stores a default IANA timezone for new configuration, while every user stores their own display timezone. Neither value silently overrides a schedule: each on-call schedule explicitly owns the IANA timezone that defines its wall-clock rotation start and handoff cadence. This keeps a 09:00 handoff at 09:00 through daylight-saving transitions. Notification content converts dates to the recipient’s personal timezone.

A schedule contains an ordered rotation of organization members. A rotation can use one fixed segment length or a repeating pattern such as 4,3, allowing one schedule to alternate Sun–Wed and Thu–Sat coverage. Schedules also store the weekdays that provide coverage; excluded weekdays create explicit gaps and do not advance the pattern. Each participant can have optional effective-from and effective-until instants, so planned hires, leave, and removals are reflected in future calendars without a precisely timed manual edit. Membership changes are treated as schedule transitions and the active roster is recalculated at the effective instant.

Non-overlapping overrides use exact start and end instants and temporarily supersede both the rotation and weekday exclusions. Organization member removal is blocked while that member remains referenced by a rotation.

An escalation policy contains up to ten ordered steps. Step one is immediate; each later step stores a delay relative to the preceding step. A step targets an on-call schedule, the service’s primary owners, or all service owners. When an incident is created for a service with a policy, the API snapshots every step into a delayed outbox message in the same transaction. The worker resolves the current schedule responder or service owners when each step becomes due, so rotation changes and active overrides are honored at delivery time. Acknowledgement or resolution causes remaining stages to be audited as skipped.

Registration is deployment-configurable. Self-hosted Compose selects Bootstrap, so exactly the first administrator can create an organization and choose a passkey (recommended) or password without needing an outbound mail server. Registration then closes automatically and additional members join by administrator invitation. Open deliberately keeps public registration available and may require email ownership verification according to deployment configuration. Provisioned removes public registration while allowing an authenticated provisioning service to create an organization and its initial administrator. It can support a self-service checkout flow without exposing an unauthenticated organization-creation endpoint. Disabled is the fail-closed product default. The same web build reads the registration capability from the API rather than embedding a deployment-edition switch.

ASP.NET Core Identity 10 owns passkey attestation, assertion, replay counters, and credential storage. Public passkeys are discoverable and require local user verification, enabling username-less sign-in. The explicit RP ID is inferred from the canonical public URL unless an operator deliberately overrides it. An authenticated user can keep up to ten named passkeys. Credential removal is serialized per user, and a passkey-only account cannot remove its final passkey.

Local Identity credentials are one authentication method, not a permanent identity-provider boundary. External login records are modeled by stable provider and provider-subject identifiers. The public API exposes a trusted, explicitly configured extension loader plus a provider-neutral federated sign-in service. That service owns account linking, organization-boundary enforcement, optional JIT membership creation, invitation completion, and the normal Identity cookie. Protocol adapters validate their upstream response and submit only the resulting identity to this service.

Trusted authentication extensions can validate an external identity, veto local credential login, add product routes, and register a namespaced PostgreSQL context whose migrations participate in the one-off migration mode. A missing, invalid, or duplicate configured extension fails API startup. Self-hosted deployments load no extension by default.

Durable work and delivery semantics

The PostgreSQL outbox closes the failure window between changing application state and requesting asynchronous work:

1. An application use case stores its state change and a uniquely identified outbox record in one database transaction.
2. A worker claims pending work without allowing another worker to own the same lease concurrently.
3. The selected provider publishes or handles the message.
4. Completion is recorded durably. Notification deliveries store every attempt, recipient, channel, outcome, and error. Failures use exponential backoff and become dead letters after the configured maximum.

Delivery is at least once. A stable message identifier travels with every message; it is not regenerated during retry. Consumers must use that identifier as an idempotency key and make their state change plus processed-message marker atomic wherever possible. A handler must assume it can run again after a timeout, lost acknowledgement, process crash, or queue visibility expiry.

The abstractions intentionally cover only publishing and consuming Firewatch messages. They are not a general-purpose event bus. PostgreSQL is the default and requires no additional broker beyond the database Firewatch already needs. Its FOR UPDATE SKIP LOCKED claim path supports multiple worker replicas. When SQS is selected, committed outbox rows are relayed to SQS; workers long-poll and use a durable processed-message record plus a PostgreSQL advisory lock for idempotent redelivery. Email delivery uses IEmailSender; phone verification and delivery use IPhoneSender. SMTP, SES, and optional Twilio adapters do not change incident behavior. Slack delivery uses the immutable user ID captured by verified-email mapping. Each delivered Slack alert stores its bot-message reference. Signed Block Kit actions mutate the incident in the API transaction, and the resulting outbox event makes the worker update the original message and append a lifecycle thread reply. Outbound webhooks use a stable delivery ID plus timestamped HMAC signature. Organization template records are shared through PostgreSQL, avoiding divergent template files across worker replicas.

Architectural smoke flow

The asynchronous processing smoke flow is authenticated system-test infrastructure:

1. POST /api/v1/system/test-jobs creates a test-job row and its outbox message in one PostgreSQL transaction.
2. The API returns 202 Accepted, including the stable job identifier and a status location.
3. The separately running worker claims and processes the outbox item, logs its message identifier, and marks the test job complete.
4. A duplicate delivery observes the processed identifier and does not perform duplicate completion.
5. GET /api/v1/system/test-jobs/{id} reports the state to the web application or a developer.

This flow validates the architecture; it is not a model for alerts, incidents, or escalation policies.

Health and observability

• GET /health/live reports that the API process can serve requests. It does not fail merely because PostgreSQL is unavailable.
• GET /health/ready verifies required dependencies, including PostgreSQL, and is the deployment readiness gate.
• API and worker logs are structured JSON and include message or request context plus correlation/request identifiers where available. Deployments keep the process streams distinct and may add service-name enrichment in their log collection layer.
• OpenTelemetry registration is an extension point for traces and metrics. An exporter or full observability backend is deliberately not bundled.

Liveness should restart a stuck process. Readiness should remove an unhealthy API from traffic. Neither endpoint should leak credentials or detailed exception data.

One product, two deployment models

For self-hosting, Compose runs PostgreSQL, API, worker, and web, with the PostgreSQL queue provider. A future public Helm chart may package the same containers.

Managed installations configure the same provider interfaces and product contracts while operating infrastructure, secrets, backups, and delivery providers for customers. Private infrastructure does not introduce a second copy of Firewatch’s domain behavior.

Future iOS client

A native iOS application will be another client of the same versioned REST API; the web app is not a backend-for-frontend. The checked OpenAPI document can be fed to a maintained Swift OpenAPI generator (for example Apple’s Swift OpenAPI Generator) when iOS work begins. Authentication and API version rules live at the public API boundary rather than in a web-only endpoint; future compatibility changes must preserve that boundary.