Zero-Downtime Deployment: Engineering Standard

Companion to cicd-zdd-whitepaper.md. This sets out the detailed engineering guardrails behind the ZDD discipline summarised in the paper (section 5.2). Authored by engineering.

Overview

Zero Downtime Deployment (ZDD) is our target delivery model: deploy software during business hours without taking systems offline, and without coupling every change to the end-of-sprint major release window.

The goal is to decouple deployment (shipping code to production) from release (turning a change on for users), so teams can ship minor fixes and larger features more often without operational bottlenecks or live-user impact.

Today vs Target

	Today	Target (ZDD)
Major releases	End of each 3-week sprint; production taken offline out-of-hours (currently 5am)	Deploy during business hours with no planned downtime
Urgent fixes	Stealth releases, rolling load-balancer drain (service stays up, but steps are largely manual)	Same outcome via fully automated canary or blue-green pipelines
Quality gate	End-of-sprint staging regression and QA sign-off	Continuous automated testing through the cycle, with less reliance on a single late manual gate
Architecture	Mostly fix forward, we version APIs, but we do not consistently design for old and new code running together	Strict dual-version guardrails (expand/contract, non-breaking contracts, dual-version-safe consumers) so mixed fleets are safe by design

Stealth releases already achieve zero downtime for a single system via rolling updates. That works today largely because we accept fix-forward risk during the roll, not because every change is dual-version safe. ZDD extends stealth's uptime outcome to always-on, automated delivery, including what we today treat as major releases, and requires the stricter architectural guardrails below. API versioning stays; the shift is applying it consistently, plus the same discipline for databases, serialized data, and consumers.

Balancing Speed and Stability

Moving faster must not mean breaking things. This transition requires rigorous automated testing coverage and strict architectural guardrails. The goal is to eliminate high-risk deployment days by moving cost into upfront automation rather than late manual regression.

Architectural Guardrails

To deploy while traffic is live, the system must support two versions of the software running at once. ZDD-eligible services must follow the guardrails below. Legacy systems that cannot meet them may still require coordinated maintenance windows (see Exceptions).

Database Schema Changes (Expand/Contract Pattern)

Destructive database changes (such as dropping columns, renaming tables, or changing data types) are prohibited in a single release. All schema modifications must follow a multi-phase Expand and Contract approach.

See also the shorter backwards-compatibility guidance under Development, Database Changes. The pattern below is the full ZDD sequence.

• Expand: deploy DB changes that support both old and new code
• Deploy new code: writes to both/new, reads from old/new as appropriate
• Contract: data migration complete; only then drop old columns/tables

Renaming a Column Example

• Sprint N (Expand): add the new column (NewColumn) as nullable. Update the code to write to both OldColumn and NewColumn, but continue reading from OldColumn.
• Backfill: run a background data migration script to copy historical data from OldColumn to NewColumn.
• Sprint N+1 (Transition): update the code to read and write entirely from NewColumn.
• Sprint N+2 (Contract): drop OldColumn from the database.

Backwards Compatibility and API Versioning

HTTP API versioning follows Web API standards: a new major API version is required for breaking wire-contract changes. Keep the old route until telemetry shows zero traffic, then retire it.

Also apply:

• Serialized data: if state or messages are serialized (Redis, message queues, and similar), new code must deserialize objects created by the previous version.
• Forward compatibility: where practical, old code should tolerate new optional fields and unknown enum values so mixed-version fleets degrade safely.
• Shared API client packages (NuGet libraries that ship models plus the HTTP/WCF tooling so consuming apps do not hand-roll their own implementation) are a separate contract from the HTTP wire format. Treat them with the process below.

Shared API Client Packages

Many systems expose an official client package that defines request/response models and the communication stack. For those consumers, the package is the contract. A change can be compatible on the wire (JSON still deserializes) but still source-breaking in C# for every app that references the package.

Classify the change

Layer	Question	Example: decimal → decimal?
HTTP / JSON wire	Does the published API break existing callers?	Often compatible (number still works; null is additive). Making a required input optional is non-breaking per Web API standards.
Client package	Do consuming apps need code changes to compile or behave correctly?	Usually yes, assignments, interfaces, and non-nullable assumptions break.
Runtime behaviour	Can a value that was always present now be absent?	Breaking if callers do not null-check.

Do not assume "JSON still works" means a minor client package bump is safe.

Prefer expand / contract over in-place mutation

Default path for client and model changes:

• Expand: add the new shape without removing the old one (new property, new DTO, new method overload, and/or new HTTP API major with corrected models). Server should accept/return both shapes where needed.
• Migrate consumers: owning team raises work for each known consumer; apps move to the new shape.
• Contract: remove or hard-obsolete the old member only after adoption shows it is unused (package versions upgraded, obsolete call sites cleared, and HTTP telemetry quiet if the change was wire-visible).
• Avoid changing a property's type in place on a minor package release (for example renaming nothing but changing decimal to decimal? on the same member).

[Obsolete] rules

• Mark superseded members with [Obsolete("...", error: false)] in a minor package version.
• The message must name the replacement and a target removal (package major and/or date), for example: "Use AmountOrNull. Will be removed in 3.0."
• Keep the obsolete member callable for a deprecation window (at least one PI unless all consumers agree a shorter window).
• Removal, or [Obsolete(..., error: true)], only in a major NuGet version, and pair with an HTTP major when the wire contract also breaks.

Client package versioning

Change	NuGet bump
Additive: new optional property, new method, new endpoint wrapper	Minor
Obsolete with warning; old members still work	Minor
Type change on an existing member, rename, remove, or behaviour that can null where it previously could not	Major
Bug fix with no contract change	Patch

Document breaking changes, obsoletions, and migration steps in the PR and package release notes.

Producer obligations before a breaking client

Before shipping a major (or any intentionally breaking) client package:

• Prefer an RFC when the client is widely shared.
• Inventory known consumers (package references / pipeline mapping).
• Publish a short migration guide (before/after).
• State the deprecation window.
• Order the rollout for dual-version safety: service expand → client minor with both shapes → consumer upgrades → contract (major remove).

How consuming teams operate

• Use the official client: do not hand-roll HTTP/WCF against services that publish a client package.
• Stay on a supported major: current major, or previous major for at most one PI unless agreed otherwise.
• On package upgrade: treat obsolete warnings as migration debt for call sites you own; prefer adopting the replacement in the same PR that bumps the package, or raise a follow-up task with an explicit deadline.
• Multi-repo work: when a PBI needs a client contract change, raise a child task per consuming app (same pattern as other multi-repo work under Development).
• No silent majors: do not take a major client bump mid-feature without a planned migration.

Example: changing decimal to decimal?

• Server: accept/return null where needed; keep the old field populated when possible during the transition.
• Client minor: add a nullable member (or new request/response type); mark the old decimal property [Obsolete] with migration text.
• Consumers: switch to the nullable shape and handle null.
• Client major (later): remove the old property, or change its type in place only after the obsolete window.
• If consumer count is small and all owners agree, a single coordinated major that changes the type in place is acceptable: still document it as breaking and schedule migration tasks. Do not bury that change in a minor release.

Application Statelessness

For ZDD-eligible services:

• Servers should be stateless: no sticky in-memory session affinity that breaks when traffic moves between nodes.
• Session state must be externalised (for example Redis or another shared store). If the load balancer routes a user from an old server to a new server mid-session, the user must see no disruption.
• Legacy apps that still hold local session state are not ZDD-ready until that state is externalised.

Deploy Order and Dual-Version Safety

• Schema before (or with) dependent code: expand the schema before deploying application code that requires the new shape. Contract only after every node and consumer is on code that no longer needs the old shape.
• Config and feature flags: deploy behaviour dark (flag off), enable later. Do not couple a breaking behaviour change to the same deploy as a schema contract drop.
• Jobs and consumers: background workers and queue consumers must be dual-version safe, not only web/API servers.
• Health and readiness: new nodes must pass health checks before receiving live traffic.

Testing Strategy and Coverage Guardrails

To deploy to production at any time of day with confidence, reliance on a single manual QA sign-off at the end of a sprint must reduce. Testing should be multi-layered and largely automated before code reaches production.

Automated Test Coverage Mandates

• Unit and integration testing: code changes will not clear the build pipeline if they cause overall branch coverage to drop (same rule as today's PR checks). Integration tests must validate interaction boundaries between services.
• API contract testing: automated contract tests should run against a published schema (and, where adopted, consumer-driven contracts), not by probing the live production API as the source of truth. Production telemetry remains the signal for when old routes can be retired.

Testing Pipeline Hierarchy

QA environments remain the place for functional testing of individual services. Staging remains the integration gate for systems released together. Under ZDD, automated suites run continuously rather than waiting for a single end-of-sprint staging sweep:

Test Phase	Trigger	Objective	Responsibility
Commit / PR	Code check-in	Fast feedback: unit tests, linting, and security scans	Engineering
QA environment	Deploy to QA	Functional testing of the service under change	QA
Staging deployment	Merge / release candidate	Automated integration and regression against a replica environment	QA Automation / Devs
Pre-flight / Canary	Production rollout	Smoke tests against the canary (or green) node before full traffic	Pipeline / platform automation

Shift in QA Focus

QA ownership stays with the QA team: writing and maintaining automation suites, designing edge-case scenarios, and exploratory testing. The shift is away from end-of-sprint manual regression as the primary production gate, and towards earlier, continuous automated coverage before merge and during staging.

Target Release Process

ZDD evolves today's major and stealth workflows towards continuous delivery. During the transition, release branches and release marshals may still be required for coordinated multi-system releases; the end state reduces reliance on end-of-sprint release branching for every change.

• Feature branch: short-lived branch; incomplete work merges only behind feature flags.
• Merge to main/master: continuous integration against the main line.
• Build and deploy to QA: functional testing as today where needed.
• Deploy to staging: automated integration and regression run throughout the cycle, not only at sprint end.
• Production trigger: automated canary or blue-green rollout once quality gates pass.
• Incomplete features that must land on main stay wrapped in feature flags.

Choosing a Deployment Pattern

• Canary: prefer when you want gradual traffic shift and early signal from a small slice of users (for example 5%, then scale to 100% if telemetry is clean). Best default for most application rollouts.
• Blue-green: prefer when you need an instant cutover to a fully prepared environment (and equally instant rollback by flipping back), and infrastructure can host a duplicate stack for the release.
• In both patterns, new nodes must pass health checks before serving traffic. Manual load-balancer drain (today's stealth process) remains the fallback until automation is in place.

Operational Guardrails (Target)

These gates are the target operating model. Until the pipeline and tooling enforce them automatically, teams should still design and review work as if they apply.

• Architecture review gate: any RFC (or ADR-bound design) involving database schema, HTTP API, or shared API client contract changes must explicitly detail the Expand/Contract, versioning, and (for clients) obsolete/migration strategy before coding begins.
• Feature flag practice: Phenix already has a manual feature-flag implementation. ZDD requires consistent use of those toggles (and, over time, stronger lifecycle/process: ownership, enablement, and cleanup) so code can deploy dark and separate deployment from release.
• Zero-drop test gate (target): the CI/CD pipeline should reject a deployment if the automated regression suite fails. Flaky tests must be quarantined immediately rather than ignored.
• Automated rollback triggers (target): the deployment pipeline should react to New Relic (or equivalent) alerts such as HTTP 5xx spikes or error-log frequency. If a new deployment triggers an alert within a short window of rollout (for example 15 minutes), the pipeline should roll back to the previous stable version without waiting on a human.
• Definition of Done: include: "Code is verified to be backwards compatible with the current live production environment" (for shared clients: additive or obsolete-with-migration path followed, or a scheduled major with consumer tasks raised).

Exceptions: Coordinated Maintenance Windows

ZDD is the default target standard. Some changes remain technically unfeasible or prohibitively high-risk without downtime.

Rule: schedule a coordinated maintenance window (downtime release) when a change breaks a tightly coupled shared contract, or otherwise cannot be incrementally versioned, dual-written, or gracefully handled by consumers.

For shared REST/JSON API client packages, prefer the expand / obsolete / migrate / contract process above. Downtime is for contracts that cannot be dual-versioned (classic example: WCF enum changes below).

Other examples that may require downtime:

• Irreversible data migrations that cannot be dual-written or backfilled safely online
• Infrastructure cutovers that cannot run two versions side by side

Core Example: PhenixServicesClient (WCF) Enum Modifications

Our BackOffice ecosystem relies heavily on the PhenixServicesClient via WCF. Because WCF enforces strict binary and structural serialization contracts, certain modifications break backwards compatibility instantly:

• The risk: if a team adds a new value to a shared enum in the WCF contract, any of the 100+ consuming applications that have not yet been recompiled and redeployed with the updated client stub will throw serialization exceptions when they encounter that new value.
• The reality: orchestrating a zero-downtime rolling deployment across 100+ distinct consuming applications at once is operationally unsafe. Systems with an outdated contract will throw critical exceptions.

Protocol for Tight Contract Changes

When a breaking change like a PhenixServicesClient enum update is unavoidable, follow this protocol:

• Architecture review sign-off: prove the contract cannot be updated with an additive pattern (for example a new service method or endpoint instead of modifying the existing one). Prefer capturing this in an RFC.
• Scheduled coordinated downtime: if approved, bypass the standard ZDD path and schedule an out-of-hours maintenance window.
• Big bang deployment plan: draft a runbook for taking the core service offline, updating the contract, and deploying recompiled clients across consuming apps before bringing the core system back online.

Reducing Exceptions Over Time

To reduce scheduled downtime windows, teams on core shared services (such as BackOffice) should design away from rigid serialization where possible:

• Defensive enum deserialization: map unknown or new enum values to a safe default (for example Unknown) rather than throwing.
• Looser coupling: phase out legacy WCF endpoints in favour of RESTful APIs (JSON) or gRPC/Protobuf, which support additive, backwards-compatible changes more safely.

Transition Plan (Draft)

Relative phasing below is a draft roadmap and will change as tooling lands. Prefer tracking concrete work in Azure DevOps rather than treating these sprint offsets as fixed dates.

Phase 1: Tools and training

• Strengthen feature-flag practice and lifecycle on top of the existing Phenix toggles (ownership, enablement, cleanup)
• Workshops on the Expand/Contract database pattern for developers and DBAs

Phase 2: Automation and pipeline upgrades

• Automate load-balancer / traffic switching in CI/CD so humans are not required to trigger each drain step
• Wire staging automated regression and target rollback behaviour (for example New Relic-driven)

Phase 3: First ZDD major release

• Execute a major sprint release during business hours using the guardrails above
• Success criteria: no planned downtime, canary or blue-green path used, rollback exercised in staging beforehand, error rates within agreed budget during and after rollout