Aurora Rollout Runbook
Overview
Aurora is the deployment pipeline that ships the Meridian web app to production. This runbook covers the standard weekly rollout, the canary stage that precedes it, and the rollback procedure for when a release goes wrong. It applies to every service in the checkout group except the payments ledger, which ships on its own schedule.
Releases leave the main branch every Wednesday at 14:00 UTC. A release candidate is cut automatically, tagged RC, and deployed to the staging cluster within fifteen minutes. Nothing reaches production without passing the full staging suite twice in a row.
Approvals
Every production rollout needs two approvals: the release captain on duty and one engineer from the owning team. The release captain checks the error budget before signing off; if more than 25 percent of the monthly error budget has been burned, the rollout is deferred to the next window.
Emergency fixes follow a shorter path. A hotfix can ship with a single approval from the release captain, but it must be linked to an open incident and the change must touch fewer than 200 lines.
Canary stage
The canary stage routes 5 percent of production traffic to the new build for 45 minutes. The canary cohort is selected by session hash, so a returning user stays on the same build for the whole window.
Three metrics gate the canary automatically. Promotion is blocked if p95 checkout latency rises above 450 milliseconds, if the error rate exceeds 0.5 percent, or if crash-free sessions drop below 99.8 percent. The gates are evaluated every five minutes on the Helios dashboard.
If all three gates hold for the full window, Aurora promotes the build to the remaining fleet in four waves of 25 percent, with a ten minute soak between waves.
Rollback
Rollback is one command: aurora rollback --to-last-good. It repoints the fleet at the previous build and completes in under four minutes. Run it from the deploy bastion, never from a laptop.
After any rollback, deployments freeze for 24 hours unless the release captain lifts the freeze in writing. The postmortem is due within three working days and follows the blameless template in the wiki.
On-call
The on-call rotation is weekly and hands over every Tuesday at 10:00 UTC. Each shift has a primary and a shadow; the shadow becomes the primary the following week.
Pages route through Atlas. Acknowledge within five minutes; if the primary does not acknowledge, the page escalates to the shadow and then to the engineering manager. Our target MTTR for a checkout-impacting incident is 30 minutes.
Metrics review
The release captain reviews the weekly SLO report every Monday. The checkout group currently commits to two SLOs: 99.9 percent availability measured at the load balancer, and p95 checkout latency under 450 milliseconds over a rolling 28 day window.