Patagon → Gateway conversion ingestion automation: scheduled daily bridge + multi-agent historical reprocess
The client had no live Patagon→gateway integration, so leads had to be shipped reliably by machine. I built a weekday scheduled task (cron 0 12 * * 1-5, Monday=96h window) that spawns subagents to page Patagon's MCP, pull deep attribution per lead (attribution.click.clickData., attribution.source.referralUrl), dedupe, and write per-country datasets, returning only counts to protect the orchestrator's context. A Node sender HMAC-signs each event and posts to the gateway in foreground chunks (START/LIMIT at 900ms) because the sandbox reaps background jobs and caps calls at ~45s; stable event_id dedup makes retries safe. For history I created a one-time task orchestrating one extraction subagent per country (deep for MX/AR, list-only for high-volume CL) plus a chunked-send subagent, ingesting 754 events with zero errors, and re-fired Meta-only by deleting just the Meta dedup docs.
A machine now ships three countries' WhatsApp leads into Meta and Google every weekday, orchestrated across scheduled tasks, MCP subagents and a paired coding agent.
Role: Sole designer and operator of the automation and multi-agent orchestration
The Problem
The client has no live Patagon→gateway integration, so recent WhatsApp leads (and 7 to 15 days of history) had to be reliably shipped to Meta and Google without a human running scripts by hand each day.
Patagon list_leads payloads are huge and attribution lives in deep per-lead objects (requiring get_lead per lead); the workspace bash sandbox reaps background processes and caps calls at ~45s; the gateway returns HTTP 429 at 500ms for hundreds of Chile leads. Naïve single-call or background sends failed.
Balancing session-context limits (huge tool outputs), sandbox constraints (no surviving background jobs, 45s cap), rate limits, dedup idempotency, and two different platform windows (Google 15 days, Meta 7 days) in one send.
Approach & Architecture
Built a scheduled weekday bridge (client-patagon-daily, cron 0 12 * * 1-5, Monday=96h window else 48h) that spawns a subagent to extract deep attribution via patagon_get_lead, writes per-country JSON datasets, then sends via a Node backfill script in foreground chunks at 900ms. For history, created a one-time reprocess task and orchestrated one subagent per country (MX/AR deep get_lead, CL list-only by volume) plus a chunked-send subagent, using a stable/RESEND event_id for dedup idempotency. Meta-only re-fire achieved by deleting only Meta dedup docs so Google auto-skips as duplicate.
Scheduled task (its own agent session) → extraction subagent(s) using Patagon MCP → per-country backfill-data/*.json → foreground chunked sender (START/LIMIT, DELAY_MS) → HMAC-signed POST to /v1/collect/{client}/backend → gateway → Meta/Google. Idempotency via gateway dedup keyed on event_id.
Key Decisions & Trade-offs
Hardest Part
Delivering hundredsthousands of events reliably within a sandbox that reaps background jobs and caps calls at ~45s, without blowing the main agent's context, solved by delegating both extraction and the chunked send to subagents that return only summaries.
Technical Detail
17-key per-lead JSON dataset; dedup key client_id+normalized_event+event_id+destination_platform
Claude scheduled tasks (cron + one-time fireAt); workspace bash sandbox; Cloud Run gateway; Firestore dedup
Idempotency validated by observing retries return HTTP 200 'duplicate'; per-country ingest tallies (queued/duplicate/errors) checked each run
Code
set -a; . ./client-keys.env; set +a SC_API_KEY="$CLIENT_CL_API_KEY" SC_SIGNING_SECRET="$CLIENT_CL_SIGNING_SECRET" DELAY_MS=900 RESEND=rp0805 START=0 LIMIT=20 CLIENT_ID=client_cl node backfill-client.mjs # repeat START=20,40,... until 'plan: 0 eventos'
const ts = Math.floor(Date.now()/1000).toString();
const sig = crypto.createHmac('sha256', SIGNING_SECRET).update(ts+'.'+rawBody).digest('hex');
await fetch(${BASE}/v1/collect/${CLIENT_ID}/backend, {method:'POST', headers:{'x-api-key':API_KEY,'x-sc-timestamp':ts,'x-sc-signature':sig}, body:rawBody});The whole ingestion pipeline is AI-native: a Claude scheduled task runs unattended on a weekday cron; each run spawns general-purpose subagents that page Patagon's MCP (list_leads → get_lead per lead), deduplicate, write per-country datasets and return only counts to keep the parent context clean; then a send subagent runs the Node backfill in foreground chunks. A one-time scheduled task performed the 15-day/7-day historical backfill the same way. Repo/config work was delegated to a paired Claude Code (CC).
Unattended weekday delivery of the client WhatsApp leads (MX/CL/AR) into Meta + Google with correct attribution, plus a repeatable backfill pattern
Subagent delegation kept the orchestrator's context clean across thousands of Patagon reads and hundreds of chunked sends that would otherwise exhaust a single session
Measured Results
The client WhatsApp leads now flow automatically each weekday from Patagon into Meta and Google with deep attribution, and 7 to 15 days of history were backfilled idempotently. The pipeline survives the sandbox's constraints (no background jobs, 45s cap, 429s) via subagent-delegated extraction and foreground chunked sends.
| Metric | Value | Before | Source |
|---|---|---|---|
| Historical reprocess events ingested (15d/7d) 0 errors, 0 429 on the successful pass | 754 (CL 441, MX 196, AR 117) | n/a | Gateway ingest responses |
| Reprocess extraction, leads per country (15d) MX 48 gclid/2 fbclid; CL 327 ctwa/7 gclid; AR 5 gclid/77 pageUrl | MX 184, CL 429, AR 112 | n/a | Subagent extraction counts |
| Daily run stale leads skipped (>7 days) | 145 (96 CL + 48 MX + 1 AR) | n/a | backfill-client.mjs SKIP_STALE output |
| Received_today per client after observability fix | CL 1267, MX 391, AR 146 | n/a | Gateway /v1/admin/metrics per-client (CC) |
Every figure above was recorded during the work itself. Where no number was measured, none is claimed.