Cloud Run ingress lockdown to LB-only behind Cloud Armor (with delivery-worker path verification)
The production conversion gateway was reachable directly on its Cloud Run run.app URL, bypassing the Cloud Armor WAF/rate-limiter in front of the Load Balancer. Flipping ingress to internal-and-cloud-load-balancing risked breaking delivery because Cloud Tasks calls the worker at the run.app and pending tasks are baked with that URL. Without a staging LB, I validated on real prod: an unauthenticated probe to /internal/tasks/deliver returned a 401 app-level (not a Cloud Armor 403 or LB 404), proving the route works through the LB, and Cloud Run docs confirmed same-project Cloud Tasks count as internal, so the 48 in-flight tasks survive and DELIVERY_WORKER_URL stays on run.app. The flip was two decoupled steps with instant --ingress all rollback; afterward run.app is 404 from the internet, ingest and worker keep working, and a revision-pinning footgun was reset to latest.
Closed a public back door on live conversion infrastructure without a staging LB and without losing a single in-flight delivery.
Role: Diagnostician and change designer; gcloud executed by paired Claude Code
The Problem
The production gateway's Cloud Run service was reachable directly via its run.app URL, bypassing the Cloud Armor WAF/rate-limiter in front of the Load Balancer, a defense-in-depth gap on the conversion API.
Setting Cloud Run ingress to internal-and-cloud-load-balancing risked breaking delivery, because Cloud Tasks invokes the worker at DELIVERY_WORKER_URL (the run.app), and already-enqueued tasks are 'baked' with that URL.
Staging had no Load Balancer to validate against, and closing ingress could silently drop in-flight delivery tasks scheduled up to 24h in the future.
Approach & Architecture
Verified the LB routes /internal/ to the worker (a 401 app-level, not a 403 Cloud Armor or 404 LB, on an unauthenticated probe), then confirmed from Cloud Run docs that same-project Cloud Tasks reaching run.app counts as internal under internal-and-cloud-load-balancing, so pending baked tasks survive the flip and DELIVERY_WORKER_URL could stay on run.app. Flipped ingress in two decoupled prod steps with instant --ingress all rollback.
External HTTPS LB (serverless NEG scg-backend-prod) + Cloud Armor scg-armor-prod → Cloud Run gateway. Ingress set to internal-and-cloud-load-balancing so external traffic must traverse the LB; Cloud Tasks (internal) still reaches the worker.
Key Decisions & Trade-offs
Hardest Part
Proving delivery would survive the flip without a staging LB: an unauthenticated probe to [internal-host]/internal/tasks/deliver returning a 401 app-level (not 403/404) confirmed the LB routes /internal/* to the worker and Cloud Armor allows it.
Technical Detail
Google Cloud Run, external HTTPS Load Balancer + serverless NEG, Cloud Armor (CRS 4.22), Cloudflare-proxied api origin
Unauthenticated /internal probe (401 app-level), real conversion sent post-flip and confirmed delivered, run.app confirmed 404 from the internet, [internal-host] ingest confirmed 202; --ingress all rollback ready
Code
# scg-armor-prod: add SQLi+XSS 4.22 in preview
# evaluatePreconfiguredWaf('sqli-v422-stable', {'sensitivity':1})
# evaluatePreconfiguredWaf('xss-v422-stable', {'sensitivity':1})gcloud run services update scg-gateway-prod --ingress all
Change designed and risk-analyzed in Claude Cowork (identified the baked-task hazard, wrote the two-step decoupled plan with fallback), executed by paired Claude Code (CC) which verified against Cloud Run docs rather than assuming, ran probes, and reset a revision-pinning footgun.
Measured Results
Production ingress is now internal-and-cloud-load-balancing: the direct run.app returns 404 from the internet while [internal-host] ingest and the internal delivery worker keep working, with no conversions lost. A separate revision-pinning footgun was fixed so future deploys auto-serve.
| Metric | Value | Before | Source |
|---|---|---|---|
| Pending Cloud Tasks baked to run.app preserved through the flip Draining ~24h; confirmed not lost | 48 | n/a | CC report |
Every figure above was recorded during the work itself. Where no number was measured, none is claimed.