Image promotion: the prod isolation pattern I use for enterprise AI

On every serious AI platform engagement I end up re-implementing the same rule. Production container registries only accept promoted images that have already been verified in a staging environment, and CI never pushes builds to them directly. This rule has a lot to do with whether you sleep through your on-call rotation.

The usual default

This is the pipeline I see in many real-world AI teams:

git push → GitHub Actions / Azure Pipelines → docker build → push to prod registry → prod pulls :latest

The prod registry is fed directly from CI. Every green build becomes a potential prod image. The prod deploy is then gated by a manual approval or a merge to main, but the image that’s about to run in production is the exact same artifact that CI produced moments ago, from the same build process, Dockerfile and layer cache.

That is fine for a marketing site, but on a GenAI platform that handles customer data and calls $0.03-per-token models it will eventually cause an incident.

What goes wrong

I’ve seen these four failure modes repeatedly.

1. Supply chain compromise with full blast radius

If a dependency in your image is compromised (actions/checkout typosquat, malicious npm package pulled into your Python container via pip transitive dep, etc.), your next CI build pushes that compromised image directly into prod. The blast radius is whatever your prod service can reach, which usually means customer data, API keys in envs and downstream services.

If prod could only pull from a promoted registry, the compromise sits in staging waiting for your E2E tests or security scans to notice, which gives you time to react.

2. “Mostly green” CI that ships a bad image anyway

A green CI run is only a signal. Tests can pass on a broken image because of environment-specific behavior, missing env vars that don’t show up until runtime, or ML model weights that load locally but fail on the prod GPU pool.

With direct-build-to-prod, “tests pass, ship it” means nobody confirms the image runs until prod users hit it. Image promotion puts the image through staging first, where problems surface an hour before they touch a customer.

3. The “what’s in prod?” audit hole

In direct-build models, identifying exactly what’s running in prod at a given moment is harder than it should be. You have a tag, and the tag points to a digest, but that digest came from a CI run that pulled a :latest base image, installed packages from a requirements.txt that resolved at a specific time, and ran against a cache that may or may not have been warm.

The reproducibility of that exact image, three months later, is not guaranteed. Promoted images pin digests explicitly. You know what’s running because you promoted it on a specific day, from a specific source.

4. Rollbacks that rebuild a different image

Say you discover a prod issue and want to roll back to yesterday’s working image. In a direct-build model, you re-run the pipeline for yesterday’s commit and get a different digest. A transitive dep may have had a patch release overnight, your base image may have moved, or your cache may have been purged.

The “rollback” ends up being a re-deploy of a subtly different artifact. For most bugs that’s fine, but during a security incident it’s horrifying.

Promoted images are retrievable by digest, forever (within your registry’s retention policy). Rollback is a 30-second operation to re-promote an older image to the prod registry.

The promotion pattern

This is what I ship on enterprise platforms:

git push → CI builds → pushes to STAGING registry (e.g. acrstaging.azurecr.io)
                              │
                              ▼
                     Staging environment pulls and runs
                              │
                              ▼
                       E2E + security scans
                              │
                              ▼
                      [QA sign-off gate]
                              │
                              ▼
            Promotion job copies STAGING → PROD registry
                              │
                              ▼
                     Prod environment pulls

Concretely, on Azure with ACR, this looks like:

• Two separate ACRs: acrstaging (permissive push, anyone in CI can write) and acrprod (locked down, only a promotion service principal can push to it)
• A promotion pipeline stage that uses az acr import or docker pull + tag + push with a scoped identity
• The promotion is gated on: E2E test suite green, security scan clean, QA lead approval (for bigger changes)
• Prod compute has IAM that can only pull from acrprod

On AWS with ECR the pattern is identical, with two separate ECR repos, a Lambda or CodeBuild that does the promotion, and scoped IAM that gates what prod can pull.

The harder parts

Identity scoping. The whole point collapses if your CI service principal can write to prod, so make sure it can’t. You need a separate identity that only the promotion stage has access to, with short-lived credentials (workload identity federation is the right primitive here).

Cache coherence. If staging runs image sha256:abc... and the promotion copies it to prod, you want prod running the same digest, so copy it with a registry-level import.

Retention. Promoted prod images should be retained longer than staging ones. You want a 30-day rollback window minimum, usually 90.

Base image pinning. The promotion guarantee is meaningless if your Dockerfile says FROM python:3.12. Pin the digest of your base image, as in FROM python:3.12.2-slim@sha256:....

When to skip it

I don’t use this pattern on personal projects, marketing sites, or toys. The machinery overhead isn’t worth the protection you’re buying against low-stakes incidents.

I always use it when:

• The image handles customer data
• The image makes paid external API calls (LLMs, payment processors)
• The image has production credentials in its environment
• You are regulated (finance, health, anything with compliance obligations)

For enterprise GenAI, all four are usually true, so every serious platform I’ve shipped has image promotion.

---

If you’re architecting a production AI platform and aren’t sure your CI/CD is trustworthy, let’s talk. I’ll walk through which parts are safe and what it would cost to close the gaps.