Provider Workflows And Rollback

Alfred's implemented workflows coordinate systems that converge at different speeds and expose different rollback capabilities. The forward path and its compensation path are designed together.

Add Domain

add-domain has one stable registration stage:

1. verify that the domain is available through the configured GoDaddy account;
2. find an existing Route53 hosted zone or create one;
3. when a zone was created, update the registrar nameservers;
4. record the projected stage and terminal events.

An active-run hook deduplicates concurrent starts for the same logical request. Provider errors record a failed registration stage before the durable run fails.

This workflow establishes ownership and delegation. It does not perform the certificate and routing work described below.

Onboard Domain

onboard-domain composes registration and certificate onboarding under one execution and one aws-infrastructure lane owner. Registration completes before certificate planning begins. An already registered domain with an existing hosted zone is recorded as adopted and continues; a registration failure stops before certificate mutations.

The certificate stages use the same safety and receipt contracts as the standalone add-certificate workflow. The normalized execution outcome keeps domain, certificate, distribution, cleanup, Google Maps, Zendesk, and structured assistance states explicit even when a later stage fails.

After certificate finalization, Alfred reconciles browser referrers on the configured Kirby and Zapdos Google Maps API keys. Each root domain requires exactly two entries: https://example.com/* and https://*.example.com/*.

Alfred reads and preflights every configured key before changing any key. A key supports at most 1,200 referrers, which is 600 root domains under this two-entry rule. If any key lacks capacity, Alfred changes no key and records requester assistance. When fewer than 50 root-domain slots remain, the outcome includes a capacity warning. Alfred does not shard referrers across keys.

Updates preserve the key's other restrictions and API targets, use the current etag, wait for the provider operation, and reread the key to verify both referrers. One etag conflict triggers one fresh read and retry. A verified partial update retains its receipt so rollback can remove only referrers added by that execution.

Google Maps manual assistance does not turn completed domain infrastructure into a failed execution. Alfred leaves the Zendesk ticket open, adds an internal pending comment, and mentions only the immutable Slack requester. After the issue is resolved, the Maps-only retry reruns whitelisting and onboarding completion without rerunning infrastructure stages.

Add Certificate

The current workflow has six stable stages:

Before final DNS mutation, Alfred inspects every planned record name. When an existing Route53 A or AAAA alias already targets the intended CloudFront distribution, Alfred preserves that alias and coexisting records such as TXT instead of replacing them with a CNAME. A non-equivalent same-name record fails finalization before any final-DNS batch is submitted. Only records Alfred changes receive rollback receipts.

Final-DNS Route53 requests share one paced request stream across hosted-zone lookup, reconciliation, receipt snapshots, and record changes. Alfred starts requests at least 300 milliseconds apart to stay below Route53's account-level API quota while retaining the complete preflight-before-mutation boundary.

Operator pause checkpoints exist before CloudFront, ELB, and finalization. Readiness checks run before the first provider mutation and include the services required by the selected Kirby mode.

Certificate Selection

Without an override, planning intersects the certificates attached to the configured Kirby, NGINX, and Mystique listener topology. A reusable candidate must be:

• ACM ISSUED, Amazon-issued, and unexpired;
• attached to the required CloudFront and ELB topology;
• associated with the current Mystique generation;
• structurally valid, with every wildcard paired with its apex;
• at or below 28 existing SAN entries.

The 28-entry ceiling leaves room for the requested apex and wildcard without exceeding Alfred's 30-SAN recovery boundary. Eligible candidates are ranked deterministically.

If discovery succeeds but no candidate is eligible, Alfred plans an isolated new certificate group. A discovery error fails and retries; it never silently changes the decision into certificate creation.

Alfred also rejects reuse when a candidate certificate's existing MicroBrands or NGINX aliases span multiple CloudFront distributions. Automatic selection checks eligible candidates in deterministic rank order and uses the first one whose aliases have unified ownership. Only after every candidate is unsuitable does Alfred prove that each newly requested domain is unowned and plan an isolated certificate group. It does not move or rewrite any existing alias. An already-owned requested domain fails closed. An explicit recentCertOverrideDomain never falls back silently.

recentCertOverrideDomain is an operator recovery input, not a bypass. Alfred resolves it to a certificate and applies the same topology and safety checks. An invalid override fails and reports the highest-ranked eligible alternative when one exists, or instructs the operator to remove the override so a new group can be created.

Each run handles at most 15 apex domains. Larger sets become deterministic, awaited child executions so no certificate crosses the 30-SAN boundary and the root execution remains the single AWS-lane owner.

Kirby Routing

Both modes add entries to the NGINX hosts file and converge the NGINX ArgoCD application.

When isKirby is true, Alfred also:

• adds the domains to the Kirby domain list in the Mystique values file;
• refreshes and syncs the Mystique ArgoCD application;
• requires both NGINX and Mystique to be healthy, successfully operated, and running a revision that contains Alfred's expected commit.

When isKirby is false, Alfred changes only the NGINX hosts file and waits only for the NGINX application.

Git changes use a library-backed repository service and stable operation IDs. After a changed commit, Alfred waits 20 seconds before the first ArgoCD action. Convergence polling starts at 30 seconds, grows exponentially up to 30 minutes, and is bounded to 30 polls. Unrelated ArgoCD drift does not satisfy the check: the observed and operation revisions must contain the expected commit.

Add Subdomain

add-subdomain reads the root hosted zone and apex A-alias, then plans one matching alias for each requested relative hostname. Missing records and already-correct aliases require no operator decision. A conflicting CNAME or a different A record pauses the execution with the current record snapshot and a hash of the complete proposed plan.

Approval is scoped to that plan hash. Alfred rereads Route53 before mutation; changed DNS invalidates the approval and produces a fresh plan. An accepted plan becomes one Route53 change batch, and the workflow waits durably for INSYNC. This workflow does not change certificates, CloudFront distributions, or platform routing.

Update Lambda Version

update-lambda-version discovers a bounded CloudFront fleet for one approved target and environment, resolves a numbered Lambda version in us-east-1, and freezes the plan before mutation. It updates only existing eligible Lambda@Edge associations and preserves every unrelated distribution field.

CloudFront writes run in bounded batches. Cancellation stops later submissions but continues observing updates already accepted by CloudFront. Each accepted association change records its before and after ARN. Retry skips completed or already-current distributions; rollback restores the exact before ARN and fails closed if current provider state matches neither side of the receipt.

The target-and-environment concurrency lane rejects a different requested version while work for that fleet is active. Other lanes can run concurrently.

Configuration Preservation

An adopted CloudFront distribution is read with GetDistributionConfig. Alfred constructs the write payload from that complete configuration and changes only:

• the requested aliases, appended without removing existing aliases;
• the viewer certificate ARN and its compatibility field when present.

A guard compares the current and desired configuration and rejects any other change. Origins are normalized only to fields required by the CloudFront update API, including CustomHeaders, connection settings, origin access control, origin path, and Origin Shield. Cache behaviors, policies, logging, restrictions, HTTP versions, functions, and unrelated settings are carried forward.

ELB operations use the listener-certificate APIs. They attach or detach the target certificate and do not rewrite listener actions, protocols, ports, policies, or the default certificate.

Mutation Receipts

Every successful mutation can return a versioned receipt with:

• the Alfred execution and stable operation key;
• the resource identity;
• enough prior and desired state to prove ownership;
• the timestamp and receipt schema version.

Receipt kinds cover ACM requests, Route53 record changes, CloudFront changes, ELB links, GitOps routing, Google API key referrer changes, old-certificate deletion, and Zendesk attempts. Operation keys deduplicate retries. Conflicting receipts with the same key fail validation.

Receipts live in completed Workflow SDK step output. Stage output may summarize them for operators, but rollback never reconstructs provider state from stage projections.

Compensation Order

Rollback loads every completed source-run step page, hydrates its output, verifies that the run belongs to the requested execution, validates receipt versions, deduplicates operation keys, and reverses the plan.

Compensation is receipt-specific:

Infrastructure compensation failures make rollback fail. A Zendesk correction failure is reported but does not claim that AWS recovery failed.

Old Certificate Cleanup

Finalization resolves every configured listener that still uses the old certificate, detaches it, and waits 15 seconds before requesting ACM deletion. Deletion receives up to two Workflow SDK retries with exponential delays.

If ACM still reports the certificate in use, or deletion still errors after the retry budget, Alfred continues finalization with a manual-required result. The execution response identifies the certificate, remaining attachments when known, the error when available, and the manual deletion action. It must not silently report cleanup as complete.

If the old certificate was already deleted, rollback cannot recreate it. The receipt is reported as irreversible; other reversible mutations are still compensated.

Recovery Preconditions

Automated rollback requires the retained source Workflow SDK run. It fails closed when the run is missing, belongs to another execution, contains unreadable encrypted output, returns truncated receipt pages, or has invalid or conflicting receipts.

AWS CLI and provider consoles are validation tools during normal production runs. Manual mutation is a separate operator decision and should be recorded; Alfred's service endpoints remain the preferred forward and rollback path.

Supported compensation is intentionally narrower than retry support: