GateLab

Independent crawler identity control

Know who is crawling. Choose what they may reach.

GateLab sits before your origin, distinguishes Google Search from Google Ads, Meta Ads, Stripebot, and other known crawler families, verifies the source where the provider supports it, and applies one explicit policy per website.

Deploy crawler control See crawler policies
crawler-identity-2026-08Verified
GOOGLEAds crawler
  1. 01Receive requestBefore origin
  2. 02Recognize claimRegistry
  3. 03Verify sourceIndependent
  4. 04Apply site ruleExact match
  5. 05Enforce + recordAuditable

Built-in registry

Named crawlers, honest status.

“Verified” means GateLab found evidence the requester cannot place in a header. When no public verifier exists, the identity remains explicitly “claimed.”

Google Search

Verified by Googlebot FCrDNS

Search

Google Ads

Verified by Google special-crawler FCrDNS

Ads

Bing + Apple

Verified by published provider domains

Search

Stripebot

Verified through crawl.stripe.com

Payment

Meta crawlers

Verified through rotating AS32934 prefixes or strict FCrDNS

Ads / preview / AI
Meta rotation is coveredGateLab refreshes currently announced AS32934 prefixes every six hours. Strict fwdproxy-*.fbsv.net FCrDNS is retained as a second confirmation path; failure of both remains claimed.

Per-site policy

Fine control at the exact path.

Verified Google Search and Meta Ads crawlers can receive narrow public-page access while their unverified lookalike claims are monitored or denied.

01

Allow

Forward the request with a signed identity decision.

02

Monitor

Forward it and retain a privacy-minimized audit event.

03

Rate limit

Control crawl pressure without blocking the entire provider.

04

Deny

Stop the request before the private origin.

05

Custom notice

Return a truthful, no-store crawler-access error page.

Product boundaries

The value Cloudflare does not own.

GateLab owns your crawler registry, provider verification, site-specific policy, signed origin decision, and identity history. Optional edge evidence remains optional.

01

Claim is not identity

A user-agent token identifies what the caller says it is. GateLab promotes that claim only when provider-controlled evidence verifies the source.

02

Rules belong to the website

Each deployment can allow, monitor, rate-limit, deny, or return a truthful notice by crawler, verification status, path, and method.

03

Independent by design

Crawler recognition, verification, enforcement, and reporting run inside GateLab. A CDN signal can be added, but Cloudflare is not required.