Recover Workflows

Recovery starts from Alfred's persisted execution and stage history. Do not reproduce provider mutations manually before checking the execution record.

Choose a Recovery Path

All five runnable workflows support retry from an eligible failed or canceled execution through Slack. The generic execution API retry is narrower, and supports failed or canceled update-lambda-version executions only. Compensation is narrower still: certificate/onboarding rollback and Lambda version restoration require validated receipts from the source run.

Recover in Slack

1. Open the execution's canonical workflow thread in the configured operations channel.
2. On the root card, choose Retry workflow or, when available, Retry finalization or Retry Maps whitelist.
3. Review the workflow, execution ID, current status, selected action, and persisted inputs.
4. Choose Confirm retry. To make no change, choose Keep current state.

The persisted inputs are read-only. Recovery uses the request that Alfred previously validated and stored; it does not let an operator change domains, ticket IDs, topology flags, or other workflow inputs in the retry dialog.

Alfred checks the execution's workflows:<name> permission and recomputes recovery eligibility when the dialog is submitted. If the execution changed after the card or dialog opened, Alfred queues nothing, reports the current status privately, and asks the operator to refresh before acting again.

When recovery is queued, Alfred posts a public Retry requested by @requester milestone in the existing canonical thread. It preserves the root card and all earlier milestones, then appends progress from the new Workflow SDK run to that thread. While waiting for the new run, Alfred does not reread the previous terminal run.

For onboarding, a successful stage-1-registration is durable external state. A later certificate failure does not remove it. Retry workflow skips valid persisted registration and resumes from the first incomplete certificate stage.

Retry a Lambda Version Update Through the API

Use the execution control for a failed or canceled update-lambda-version execution when you already have the execution ID:

curl --fail-with-body --silent --show-error \
  -X POST "$ALFRED_API_URL/v1/executions/$EXECUTION_ID/resume" \
  -H "Authorization: Bearer $ALFRED_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{}' | jq

Alfred validates the persisted request with the current workflow schema. A new retry returns status: retry_queued; a duplicate pending retry returns status: retry_already_queued. Both are asynchronous, so continue monitoring the execution. The endpoint returns 409 execution_not_resumable for failed or canceled onboard-domain, add-domain, add-certificate, and add-subdomain executions.

Continue by Re-submitting the Logical Request

For add-domain, add-certificate, and add-subdomain, submit the same workflow and normalized body through the normal workflow route. A new delivery key is allowed:

curl --fail-with-body --silent --show-error \
  -X POST "$ALFRED_API_URL/v1/workflows/add-domain" \
  -H "Authorization: Bearer $ALFRED_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: recovery-$(date +%s)" \
  -d "$ORIGINAL_REQUEST_BODY" | jq

For a failed or canceled execution, Alfred finds the logical request identity and keeps the original executionId, principal, Slack binding, and canonical thread. Completed stage outputs are reused. This route-based continuation does not continue a failed or canceled onboarding execution: domain onboarding keeps that execution as the domain blocker until an operator retries it in Slack or rolls it back.

Do not use continuation for an execution whose status is rolled_back. Rollback permanently closes that execution. Replaying its original Idempotency-Key returns the same terminal audit record and starts no work. Submit a fresh request with a new delivery key instead. Alfred creates a new execution ID and links it to the rolled-back predecessor. In Slack, open /alfred and complete the workflow launcher again.

Do not change fields such as isKirby, ticket IDs, or recentCertOverrideDomain when the intent is continuation. Those fields are part of the logical request identity.

Retry Finalization

Finalization retry is available only when all of these conditions are true:

• the workflow is add-certificate or onboard-domain;
• the execution failed at stage-6-finalization;
• the execution has a source Workflow SDK run ID;
• the required certificate stages have successful persisted output;
• onboarding also has successful registration and platform-routing output.

The Slack confirmation is read-only and states that Alfred will reuse the successful persisted stage outputs. To use the API instead:

curl --fail-with-body --silent --show-error \
  -X POST \
  "$ALFRED_API_URL/v1/executions/$EXECUTION_ID/retry-finalization" \
  -H "Authorization: Bearer $ALFRED_API_KEY" | jq

The endpoint reconstructs finalization input from successful durable outputs and queues a finalization-only command. It returns 409 execution_not_finalization_recoverable when the persisted state does not meet the implemented recovery guard.

Retry Google Maps Whitelisting

The Maps-only retry is available when onboarding is still successful but the latest stage-7-google-maps-whitelisting output is manual-required. Resolve the reported capacity, authentication, or provider issue first, then use Retry Maps whitelist in Slack or call:

curl --fail-with-body --silent --show-error \
  -X POST \
  "$ALFRED_API_URL/v1/executions/$EXECUTION_ID/retry-maps-whitelist" \
  -H "Authorization: Bearer $ALFRED_API_KEY" | jq

Alfred queues the command in the independent google-maps-api-keys lane. It preserves the original successful execution and source-run lineage, reruns only stages 7 and 8, and deduplicates an already pending request. Verified convergence completes Zendesk; another manual outcome leaves the ticket open. The endpoint returns 409 execution_not_maps_whitelist_recoverable when the execution is failed, belongs to another workflow, lacks source-run lineage, or the latest Maps stage has already converged.

Roll Back

Queue receipt-driven compensation for a supported add-certificate, onboard-domain, or update-lambda-version run:

curl --fail-with-body --silent --show-error \
  -X POST "$ALFRED_API_URL/v1/executions/$EXECUTION_ID/rollback" \
  -H "Authorization: Bearer $ALFRED_API_KEY" | jq

Rollback uses the source Workflow SDK run and saved mutation receipts. It compensates in reverse order and removes only changes owned by that execution. It queues in the same global AWS infrastructure lane as forward work.

Onboarding rollback may compensate certificate, CloudFront, load-balancer, Route53, platform-routing, and Zendesk mutations recorded by the execution. It never deregisters or refunds the primary domain. The successful registration remains external state that a fresh onboarding request can detect.

For update-lambda-version, rollback starts restoration only when the terminal execution contains at least one valid update receipt. It restores each exact before ARN and refuses to overwrite association drift.

Monitor until the execution becomes rolled_back or failed. A successful rollback is permanently terminal: retry controls are unavailable, and retry requests are rejected. Start a fresh workflow request if the work must run again. A failed rollback requires inspection of stage errors and may require operator intervention.

Avoid Unsafe Recovery

• Do not trigger a step-specific private endpoint.
• Do not create a second request with altered parameters to continue the first.
• Do not assume a failed HTTP poll means the provider mutation failed.
• Do not mutate AWS manually unless an operator has reviewed Alfred's receipts and explicitly chosen that action.
• Do not delete provider resources before rollback has detached their dependencies.
• Do not attempt to deregister an onboarding domain during rollback.