Deployment And Operations

Alfred runs as a small set of long-lived Railway services in Singapore. The API contains the durable workflow runtime; separate processes are used only when their availability or scaling boundary is meaningfully different.

Production Topology

.railway/railway.ts is the infrastructure source for the active project. Clients use https://alfred.headout.com for the API and https://alfred-docs.headout.com for the documentation site.

Each application service has one replica in Railway's Singapore region asia-southeast1-eqsg3a. Postgres and Redis use Singapore volumes with online resize and usage alerts.

The API deployment:

1. builds the Nitro server bundle;
2. runs pnpm db:migrate as a pre-deploy command;
3. starts node apps/api/.output/server/index.mjs;
4. configures and starts Postgres World in the API process.

workflow-admin is internal. The API proxies its UI through the authenticated admin surface using Railway's private domain. It does not execute a second workflow runtime.

Provider adapters under packages/workers are libraries called by Workflow SDK steps. Durable orchestration and recovery run through Workflow SDK.

Readiness

GET /healthz proves that the API process can answer a request. It does not inspect dependencies.

GET /readyz checks:

• a Postgres query;
• the expected Drizzle migration count;
• a Redis PING.

A failed check returns 503 with a named reason. Railway should send traffic only when these control-plane dependencies are available.

Provider credentials and reachability are intentionally not global readiness checks. An API that can serve execution history should remain available when Zendesk or AWS has an incident. A successful /readyz response must therefore be combined with bounded, read-only validation of the providers required by the workflow being operated. Certificate and onboarding workflows also perform workflow-specific readiness before their first mutation.

Command Reconciliation

The API periodically discovers runnable concurrency groups, including the global aws-infrastructure lane and target-specific Lambda version lanes, and compares active commands with Workflow SDK run status.

This prevents an API restart or stale projection from releasing shared infrastructure work prematurely. Structured workflow_command_reconciled logs include the command, execution, run ID, SDK status, lane owner, decision, and operator action.

Observability

The API exposes Prometheus text at /metrics for event streaming and webhook ingress. The dispatcher exposes its own metrics port with:

• outbox_events_pending;
• outbox_dispatch_total;
• outbox_dispatch_duration_seconds;
• outbox_subscriber_deliveries_total;
• outbox_subscriber_failures_total.

Structured logs carry stable event names and execution or command IDs. Search Railway logs by those IDs rather than by domain alone, because one root certificate execution can coordinate multiple domain chunks.

The workflow admin UI and Workflow SDK CLI inspect durable runs, steps, sleeps, hooks, and errors. Alfred's execution endpoint remains the public projection; the SDK view is the operator source for runtime internals and retained receipts.

Failure Diagnosis

Use this order to avoid changing provider state before the control plane is understood:

1. read the Alfred execution, stages, and events;
2. inspect the bound Workflow SDK run and its last completed or failed step;
3. check api Railway logs for orchestration, reconciliation, and provider errors;
4. check dispatcher logs and metrics only when execution succeeded but event delivery is missing;
5. check workflow-admin logs when the inspection surface itself is unavailable;
6. validate the claimed resource state with read-only AWS CLI, GitHub, ArgoCD, GoDaddy, or Zendesk queries;
7. use Alfred's resume, equivalent re-submission, finalization recovery, or rollback contract.

AWS CLI is read-only during normal production validation. If provider state requires a manual mutation, stop and report the exact resource and intended change. Perform it only as a separate, explicit operator action.

Common failure boundaries:

Local AWS Emulation

docker compose up -d starts:

• Postgres 16 on localhost:5432;
• Redis 7 on localhost:6379;
• Floci on localhost:4566.

Floci is Alfred's AWS-compatible local emulator. Tests configure the supported AWS SDK v3 clients with the Floci endpoints and synthetic credentials. Do not introduce LocalStack or point local integration tests at production AWS.

Focused adapter tests prove ACM, Route53, CloudFront, and ELB request/response behavior. Full E2E tests start through Alfred's public API, inspect persisted Workflow SDK receipts, and validate both forward mutations and service-driven rollback against Floci.

Floci does not emulate GitHub, ArgoCD, GoDaddy, or Zendesk. Those boundaries use test doubles locally; production validation for them remains read-only unless the workflow itself performs the mutation.

Deployment Checks

Before a production workflow run:

• verify the latest intended commit is deployed to api, dispatcher, and docs where relevant;
• confirm /healthz and /readyz;
• confirm Postgres migrations and Postgres World bootstrap are current;
• verify required provider secrets on the api service without exposing their values;
• inspect recent service-level errors in Railway logs;
• confirm no unexpected active owner is holding the AWS lane.

Then perform bounded read-only checks for the providers used by the selected workflow. For example, /readyz cannot establish that the API principal can read ACM or CloudFront, that GitHub and ArgoCD are reachable, or that GoDaddy, Zendesk, and Slack credentials are valid.

Deployment checks establish that Alfred can run. They do not authorize a production mutation; the workflow request remains the explicit production action.