Independent crawler identity

Recognize the claim. Verify the source. Apply your rule.

GateLab sits before the origin and owns the complete crawler flow. It does not require Cloudflare: the registry, provider verification, per-site policy, enforcement, signed decision, and audit trail run inside GateLab.

01Receive

DNS sends the public hostname to GateLab.

02Recognize

Match a specific crawler claim in the registry.

03Verify

Confirm provider-controlled origin evidence.

04Control

Apply the site rule and record the result.

Control-plane readiness

Deployment probes

Use the application probe for orchestration and the authenticated Gateway probe for a specific site. A healthy process is not evidence that crawler attribution is correct; verify registry and policy outcomes separately.

GET /api/v1/gateway/health authenticated site policy and key stateGET /api/health/ready application, database, and required schema

Install artifacts

Gateway first, adapter second.

Adapters receive only public verification material. Gateway private keys and owner API credentials never enter WordPress, JavaScript, or the browser.

WordPress + WooCommerceintegrations/gatelab-wordpress-woocommerce-0.2.0.zip

Verifies Ed25519 decisions at login, registration, password reset, comments, cart, coupons, checkout, Store API, and other mutating REST hooks.

Node + Next.jspackages/gatelab-node

Includes the decision verifier, Node middleware, Next App Router guard, exact request binding, per-action failure modes, and runnable examples.

Inline Gatewaypackages/gatelab-gateway

Classifies once at the edge, applies the active route policy, owns challenges, strips forged inbound headers, and signs the decision.

Operational onboarding

Verify ownership, publish policy, deploy.

Perform these calls from a secure server or operator terminal with the owner key. Never call the control plane from frontend code.

01
Verify the hostname first
Register the public origin in the response-integrity monitored-site flow, publish its DNS TXT record, and complete ownership verification. Gateway site registration rejects an unverified hostname.
02
Generate a Gateway decision key
Keep the Ed25519 private key in Gateway secret storage. Retain only its key ID and public SPKI for site registration and adapters.
03
Register the Gateway site
Copy the returned site ID and bounded decision-key ring into WordPress settings or server-only Next environment variables.
04
Create and activate route policy
Map every context plus each HTTP method/path rule. Unknown traffic should have an explicit outcome, not an accidental default.
05
Place Gateway before the origin
Restrict direct origin access. Gateway must strip inbound X-GateLab headers before adding its signed decision.
06
Test and rotate
Exercise allow, challenge, rate-limit, deny, forged header, expired decision, key overlap, and outage cases.

Control plane / site

Register public verification material.

Prerequisite: the same hostname must already have completed DNS ownership verification in the monitored-site flow. The public SPKI can be copied to adapters; the corresponding private key remains inside the Gateway.

Owner-authenticated requestNo browser access

POST /api/v1/gateway/sites
Authorization: Bearer <OWNER_API_KEY>
Content-Type: application/json

{
  "name": "Main shop",
  "hostname": "shop.example",
  "decisionKeyId": "gateway_decision_2026_08",
  "decisionPublicKeySpki": "<BASE64URL_PUBLIC_SPKI>"
}

Control plane / crawler policy

Make every crawler decision explicit.

Crawler rules match identity, verified or claimed status, path prefix, and method before legacy traffic-class policy. Use GET /api/v1/gateway/crawlers to inspect the active registry and verification support.

Activate per-site crawler policyAuthenticated owner API

POST /api/v1/gateway/policies
Authorization: Bearer <OWNER_API_KEY>
Content-Type: application/json

{
  "siteId": "<SITE_ID>",
  "modelVersion": "crawler-identity-2026-08",
  "bundleTtlSeconds": 300,
  "requestRate": { "windowMs": 60000, "limit": 600 },
  "decisionTtlSeconds": 60,
  "verifiedAutomationProviders": [],
  "crawlerRules": [
    {
      "id": "verified-google-search",
      "priority": 500,
      "crawlerId": "google_search",
      "status": "verified",
      "pathPrefix": "/",
      "methods": ["GET", "HEAD"],
      "action": "allow"
    },
    {
      "id": "forged-google-search-claim",
      "priority": 600,
      "crawlerId": "google_search",
      "status": "claimed",
      "pathPrefix": "/",
      "methods": ["GET", "HEAD"],
      "action": "deny"
    },
    {
      "id": "meta-ads-campaign-notice",
      "priority": 550,
      "crawlerId": "meta_ads",
      "status": "claimed",
      "pathPrefix": "/campaigns",
      "methods": ["GET"],
      "action": "custom_response",
      "response": {
        "statusCode": 451,
        "title": "Crawler access disabled",
        "message": "Meta Ads crawling is disabled for this site."
      }
    }
  ],
  "defaultActionContext": "browse",
  "defaultActions": {
    "verified_automation": "monitor",
    "authenticated_session": "allow",
    "likely_automated": "challenge",
    "likely_human": "allow",
    "unknown": "challenge"
  },
  "contextActions": {
    "browse": { "verified_automation": "monitor", "authenticated_session": "allow", "likely_automated": "monitor", "likely_human": "allow", "unknown": "monitor" },
    "login": { "verified_automation": "deny", "authenticated_session": "allow", "likely_automated": "challenge", "likely_human": "allow", "unknown": "challenge" },
    "register": { "verified_automation": "deny", "authenticated_session": "allow", "likely_automated": "deny", "likely_human": "allow", "unknown": "challenge" },
    "password_reset": { "verified_automation": "deny", "authenticated_session": "allow", "likely_automated": "deny", "likely_human": "challenge", "unknown": "challenge" },
    "comment": { "verified_automation": "deny", "authenticated_session": "allow", "likely_automated": "rate_limit", "likely_human": "allow", "unknown": "challenge" },
    "cart": { "verified_automation": "deny", "authenticated_session": "allow", "likely_automated": "challenge", "likely_human": "allow", "unknown": "challenge" },
    "coupon": { "verified_automation": "deny", "authenticated_session": "allow", "likely_automated": "rate_limit", "likely_human": "allow", "unknown": "challenge" },
    "checkout": { "verified_automation": "deny", "authenticated_session": "allow", "likely_automated": "deny", "likely_human": "allow", "unknown": "challenge" },
    "api": { "verified_automation": "rate_limit", "authenticated_session": "allow", "likely_automated": "deny", "likely_human": "challenge", "unknown": "deny" }
  },
  "rules": [{
    "id": "checkout-post",
    "priority": 100,
    "pathPrefix": "/checkout",
    "methods": ["POST"],
    "actionContext": "checkout",
    "actions": {
      "verified_automation": "deny",
      "authenticated_session": "allow",
      "likely_automated": "challenge",
      "likely_human": "allow",
      "unknown": "challenge"
    },
    "minConfidence": 50,
    "minCoverage": 0.4
  }]
}

Identity rule: never allow a claimed crawler to bypass login, checkout, API authentication, or another protected action. “Claimed” means the user-agent matched; “verified” means provider-controlled evidence also matched.

WordPress / WooCommerce

Map real hooks to Gateway actions.

Install the ZIP, enter the exact hostname/site/key values, then choose a failure mode for each action. PHP Sodium is required.

/wp-login.phplogin / register / password reset

/wp-comments-post.phpcomment

Woo wc-ajax and /cartcart / coupon

/checkout and Store API checkoutcheckout

Other mutating /wp-json routesapi

Origin protection: firewall the origin to the Gateway where possible. On shared hosting, enable the plugin's strict-origin mode so public pages and GET APIs also require signed decisions. Keep only narrow health/static exclusions; protect webhooks with explicit machine authentication.

WooCommerce note: use stable add-to-cart URLs, wc-ajax, or Store API routes so the Gateway and origin hook resolve the same action. Do not buffer or replay payment request bodies.

Node / Next.js

Verify the exact forwarded request.

The verifier checks Ed25519 signature against a bounded public-key ring, plus site ID, action context, hostname, method, exact path and query, and a maximum 120-second TTL with 30 seconds of clock skew.

App Router route handlerServer-only public key configuration

const gate = createGatewayGate({
  siteId: process.env.GATELAB_SITE_ID,
  hostname: "shop.example",
  decisionKeys: JSON.parse(process.env.GATELAB_DECISION_KEYS_JSON),
  policies: {
    browse: { onFailure: "closed" },
    api: { onFailure: "closed" }
  }
});

const guard = createNextGatewayGuard({ gate });
const result = await guard(request, "checkout");

if (result.outcome !== "allow") {
  return Response.json(
    { error: result.outcome, decisionId: result.requestId },
    { status: result.outcome === "rate_limit" ? 429 :
      result.outcome === "challenge" ? 428 : 403 }
  );
}

428Gateway challenge must complete before retry

429Rate limited by active policy

403Denied or failed closed

Close the origin bypass: self-hosted deployments should firewall the origin to the Gateway. Hosted Next/Vercel deployments should verify decisions and fail closed across all pages and APIs, excluding only framework static assets and a minimal health probe. A provider origin URL must not remain an unguarded copy of the site.

Non-idempotent challenge

Solve, then explicitly retry.

For a login, checkout, or API mutation, the Gateway returns structured 428 JSON with a same-origin challenge URL and never stores or replays the request body. The first-party browser adapter opens that challenge, waits for same-origin completion, and explicitly retries once from browser memory with the new HttpOnly proof cookie. Ordinary GET/HEAD navigation may complete inline and redirect.

Payment safety: checkout bodies stay in the customer's browser and the payment flow. Neither GateLab nor the origin adapter stores them for later replay.

Key rotation

Keep transitions explicit.

Use POST /api/v1/gateway/credentials/rotate for Gateway credentials and POST /api/v1/gateway/decision-keys/rotate for decision keys. Keep both keys valid during overlap, install the next public key, switch Gateway signing, then retire the old key. Policy-bundle signing keys are separate.

Evaluation feedback

Measure outcomes by policy and context.

Submit only a known labeled outcome from manual review, account abuse, payment fraud, a crawler registry, or customer support. Metrics can then be filtered by model, policy, action context, and class. Completing a challenge is evidence of a fresh browser session; it is not a human label and must not be submitted as one.

Owner-authenticated feedback and readoutIdempotent labeled outcomes

POST /api/v1/gateway/feedback
Authorization: Bearer <OWNER_API_KEY>
Content-Type: application/json

{
  "siteId": "<SITE_ID>",
  "eventId": "<DECISION_EVENT_ID>",
  "label": "human",
  "source": "manual_review",
  "idempotencyKey": "support-case-1842"
}

GET /api/v1/gateway/metrics?siteId=<SITE_ID>&policyVersion=4
Authorization: Bearer <OWNER_API_KEY>

Operating rule

One resource, one policy.

Traffic evidence may protect an action. It never switches marketing, pricing, claims, or destination content by class or reviewer identity.

Start onboarding

POST /api/v1/gateway/sites Authorization: Bearer <OWNER_API_KEY> Content-Type: application/json { "name": "Main shop", "hostname": "shop.example", "decisionKeyId": "gateway_decision_2026_08", "decisionPublicKeySpki": "<BASE64URL_PUBLIC_SPKI>" }

POST /api/v1/gateway/policies Authorization: Bearer <OWNER_API_KEY> Content-Type: application/json { "siteId": "<SITE_ID>", "modelVersion": "crawler-identity-2026-08", "bundleTtlSeconds": 300, "requestRate": { "windowMs": 60000, "limit": 600 }, "decisionTtlSeconds": 60, "verifiedAutomationProviders": [], "crawlerRules": [ { "id": "verified-google-search", "priority": 500, "crawlerId": "google_search", "status": "verified", "pathPrefix": "/", "methods": ["GET", "HEAD"], "action": "allow" }, { "id": "forged-google-search-claim", "priority": 600, "crawlerId": "google_search", "status": "claimed", "pathPrefix": "/", "methods": ["GET", "HEAD"], "action": "deny" }, { "id": "meta-ads-campaign-notice", "priority": 550, "crawlerId": "meta_ads", "status": "claimed", "pathPrefix": "/campaigns", "methods": ["GET"], "action": "custom_response", "response": { "statusCode": 451, "title": "Crawler access disabled", "message": "Meta Ads crawling is disabled for this site." } } ], "defaultActionContext": "browse", "defaultActions": { "verified_automation": "monitor", "authenticated_session": "allow", "likely_automated": "challenge", "likely_human": "allow", "unknown": "challenge" }, "contextActions": { "browse": { "verified_automation": "monitor", "authenticated_session": "allow", "likely_automated": "monitor", "likely_human": "allow", "unknown": "monitor" }, "login": { "verified_automation": "deny", "authenticated_session": "allow", "likely_automated": "challenge", "likely_human": "allow", "unknown": "challenge" }, "register": { "verified_automation": "deny", "authenticated_session": "allow", "likely_automated": "deny", "likely_human": "allow", "unknown": "challenge" }, "password_reset": { "verified_automation": "deny", "authenticated_session": "allow", "likely_automated": "deny", "likely_human": "challenge", "unknown": "challenge" }, "comment": { "verified_automation": "deny", "authenticated_session": "allow", "likely_automated": "rate_limit", "likely_human": "allow", "unknown": "challenge" }, "cart": { "verified_automation": "deny", "authenticated_session": "allow", "likely_automated": "challenge", "likely_human": "allow", "unknown": "challenge" }, "coupon": { "verified_automation": "deny", "authenticated_session": "allow", "likely_automated": "rate_limit", "likely_human": "allow", "unknown": "challenge" }, "checkout": { "verified_automation": "deny", "authenticated_session": "allow", "likely_automated": "deny", "likely_human": "allow", "unknown": "challenge" }, "api": { "verified_automation": "rate_limit", "authenticated_session": "allow", "likely_automated": "deny", "likely_human": "challenge", "unknown": "deny" } }, "rules": [{ "id": "checkout-post", "priority": 100, "pathPrefix": "/checkout", "methods": ["POST"], "actionContext": "checkout", "actions": { "verified_automation": "deny", "authenticated_session": "allow", "likely_automated": "challenge", "likely_human": "allow", "unknown": "challenge" }, "minConfidence": 50, "minCoverage": 0.4 }] }

POST /api/v1/gateway/feedback Authorization: Bearer <OWNER_API_KEY> Content-Type: application/json { "siteId": "<SITE_ID>", "eventId": "<DECISION_EVENT_ID>", "label": "human", "source": "manual_review", "idempotencyKey": "support-case-1842" } GET /api/v1/gateway/metrics?siteId=<SITE_ID>&policyVersion=4 Authorization: Bearer <OWNER_API_KEY>

Recognize the claim. Verify the source. Apply your rule.

Deployment probes

Gateway first, adapter second.

Verify ownership, publish policy, deploy.

Verify the hostname first

Generate a Gateway decision key

Register the Gateway site

Create and activate route policy

Place Gateway before the origin

Test and rotate

Register public verification material.

Make every crawler decision explicit.

Map real hooks to Gateway actions.

Verify the exact forwarded request.

Solve, then explicitly retry.

Keep transitions explicit.

Measure outcomes by policy and context.

One resource, one policy.

Recognize the claim. Verify the source. Apply your rule.

Deployment probes

Gateway first, adapter second.

Verify ownership, publish policy, deploy.

Verify the hostname first

Generate a Gateway decision key

Register the Gateway site

Create and activate route policy

Place Gateway before the origin

Test and rotate

Register public verification material.

Make every crawler decision explicit.

Map real hooks to Gateway actions.

Verify the exact forwarded request.

Solve, then explicitly retry.

Keep transitions explicit.

Measure outcomes by policy and context.

One resource, one policy.