OpenAI Ads (OAI-AdsBot) Crawler Access Client Guide
A nine-page client engineering guide scoped strictly to OpenAI/ChatGPT Ads crawler access for the client, leaving the client's existing pixel and Signup conversion event untouched. It covers both the marketing site and the app subdomain, explaining that each has its own robots.txt and security stack, and walks the web/security team through robots.txt, WAF/CDN, bot mitigation, authentication, CAPTCHA/JS challenges, geo rules and rate limiting until OAI-AdsBot receives an HTTP 200. The document was authored as branded HTML/CSS and rendered to PDF via headless Chromium, then QA'd programmatically with pdfplumber and pypdfium2. Critically, every technical claim was verified against OpenAI's official crawler documentation: an inaccurate Cloudflare allowlisting claim and an unsupported '24 to 72 hours' figure inherited from the source template were removed, and the official OAI-AdsBot user-agent and published IP-range files (adsbot.json, searchbot.json) were added for reliable allowlisting.
Caught and corrected two inaccuracies inherited from the source template by verifying every claim against OpenAI's official crawler documentation.
Role: Sole author and document engineer
The Problem
ChatGPT Ads reviews the landing page behind each ad before and while it serves. If OpenAI's ad-review crawler cannot reach the page, ads can stay stuck in review, which blocks the campaign from serving or optimizing regardless of correct downstream tracking. The client needed a clear engineering handoff to unblock this.
OpenAI's OAI-AdsBot must be able to fetch the submitted landing page URLs and receive a successful HTTP response. Access can be blocked at any of several layers (robots.txt, WAF, CDN, bot mitigation, authentication, CAPTCHA/JS challenges, geo rules, rate limiting), across two separate domains (marketing site and app subdomain) that each have their own robots.txt and security stack.
The blocking layer is not knowable in advance, so every layer must be verified in turn on each domain; a subdomain does not inherit the parent's crawler allowances. The source template the client PDF was based on contained an unverifiable/incorrect claim (Cloudflare 'verified and allowlisted') and an unsupported timing figure ('24 to 72 hours'), which had to be caught and corrected against primary documentation.
Approach & Architecture
Produced a 9-page branded Summit Chase PDF scoped only to OpenAI Ads crawler access for client. Built as HTML/CSS rendered to PDF via headless Chromium (Playwright) to match the brand's gradient header, navy tables, callout boxes and confidential footer. Iterated from a full guide down to a crawler-only, two-domain document, then stripped all invented content, then verified every technical claim against OpenAI's official crawler documentation and corrected two inaccuracies inherited from the source template.
Single self-contained HTML file with inline CSS (CSS variables for brand tokens, full-bleed gradient hero on page 1, per-section page breaks) rendered to Letter-size PDF by Playwright Chromium with displayHeaderFooter for a running confidential footer and page numbers. Verification loop used pdfplumber (text extraction / margin measurement) and pypdfium2 (page rasterization) to confirm layout and that corrected/removed strings were actually gone.
Key Decisions & Trade-offs
Hardest Part
Catching that the source template's Cloudflare claim ('OAI-AdsBot is officially verified and allowlisted by Cloudflare') was not supported by OpenAI's official documentation, and that the '24 to 72 hours' figure was not official (OpenAI states ~24h only for robots.txt changes reflecting in search, not ad review). Resolved by reading the official developers.openai.com/api/docs/bots page directly and the Help Center advertiser-guidance article text, then correcting both.
Technical Detail
Document build pipeline: HTML/CSS → Playwright Chromium PDF (Letter) with running footer/page numbers; QA via pdfplumber and pypdfium2
pdfplumber text extraction to confirm removed strings absent (Cloudflare allowlist claim, 24 to 72h) and required strings present (adsbot.json, OAI-AdsBot/1.0, 'manual bypass', 'not used to train'); margin measurement (content top ~49.8pt, left 48pt); pypdfium2 page rasterization for visual spot checks
Code
User-agent: OAI-SearchBot Allow: / User-agent: OAI-AdsBot Allow: /
footer = '''<div style="width:100%;font-family:Helvetica,Arial,sans-serif;font-size:8px;color:#6b6880;padding:0 64px;display:flex;justify-content:space-between;"><span><b>SUMMIT CHASE DIGITAL</b> Smarter paid media starts here.</span><span>CONFIDENTIAL, FOR the client <span class="pageNumber"></span></span></div>'''
with sync_playwright() as p: b=p.chromium.launch(); pg=b.new_page() pg.goto('file:///home/claude/client_v4.html') pg.pdf(path='client_-OpenAI_Ads_Crawler_Access_Guide.pdf',format='Letter',print_background=True,display_header_footer=True,header_template='<div></div>',footer_template=footer,margin={'top':'0','bottom':'48px','left':'0','right':'0'}) b.close()OAI-AdsBot user-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +[url] Published IP ranges: OAI-AdsBot [url] OAI-SearchBot [url]
The entire deliverable was authored and produced inside an AI coding/execution environment: reading the source template, authoring branded HTML/CSS, rendering to PDF via a Playwright script, running a programmatic QA loop (pdfplumber/pypdfium2), then web-searching and web-fetching OpenAI's official documentation to fact-check and correct the content across four iterations.
Measured Results
Delivered a brand-accurate, single-purpose client engineering guide that tells the client's web/security team exactly how to make OAI-AdsBot reach ad landing pages on two domains without touching existing tracking. Two factual errors inherited from the source template were caught and corrected against OpenAI's primary documentation, and layout defects (missing top margins) were fixed and verified programmatically.
| Metric | Value | Before | Source |
|---|---|---|---|
| Final document length | 9 pages | n/a | pdfplumber page count |
| Content-page top margin after fix Margin defect identified and corrected during iteration | ~49.8 pt | ~0 pt (content butted against top edge before fix) | pdfplumber word position measurement |
| Number of iterations v1 full guide, v2 crawler-only + two domains, v3 removed invented content, v4 verified vs official docs | 4 (v1v4) | n/a | conversation history |
Every figure above was recorded during the work itself. Where no number was measured, none is claimed.