Search & AEO Skill
AI Crawler Allowlist
AI crawler User-Agent allowlist + robots.txt + ai.txt + Cloudflare WAF / nginx policy +…
$15
AI Bot UA Classifier
Real-time bot classification engine for AI crawler traffic.
A real-time AI bot classification engine that proves whether traffic claiming to be GPTBot, ClaudeBot or ChatGPT-User actually is. It combines four-layer fingerprinting (User-Agent, IP range, ASN cross-check, reverse-DNS) with behavioral anomaly detection and Bayesian reputation scoring, deployed both as a sub-50ms edge classifier and a batch log re-classifier. Because a wrong call means either blocking a legitimate AI bot (lost citations) or burning bandwidth on a spoofer, every threshold is deliberately allow-biased.
$15 one-time
Add to a kit → Prices include 20% VAT. · Forged on real agency work · one-time, no lock-in
- Type Skill
- Category Search & AEO
- Delivery Email · instant
- License One-time
Inside the run · no black box
See the actual work before you buy it.
A User-Agent header is a claim, not an identity. The classifier tests each one against three independent signals before any allow, throttle or block verdict lands:
- Parses each request's User-Agent against the known-bot registry (kept in sync with DarkVisitors) to extract the claimed identity: GPTBot, ClaudeBot, Bytespider and 20+ more.
- Cross-checks the source IP against the vendor's official IP range (openai.com/gptbot.json and equivalents), the ASN (OpenAI is AS396982) and a reverse-DNS PTR lookup, three independent identity signals beyond the UA claim.
- Aggregates behavior from the IP's last 100 requests: did it fetch robots.txt, did it probe /admin, is the request rate stable, what is the 2xx/3xx success ratio.
- Combines the prior (how credible the UA claim is) with the fingerprint likelihood into a Bayesian posterior, then assigns TIER 1 allow, TIER 2 rate-limit, TIER 3 flag or spoof block. The block threshold is deliberately conservative (confidence above 0.9) because blocking a real AI bot costs you citations.
- Writes a signal-by-signal audit trace for every verdict, then reruns the whole window as a nightly batch over access logs.
- Runs a monthly registry diff plus a chi-square cohort drift test, so a new bot or a behavior shift in an existing one raises an alarm instead of silently skewing your AEO numbers.
One power source. 6 lines out.
ai-bot-ua-classifier · core
core active · 6 lines
- ✓ verify that chatgpt-user
Verify that ChatGPT-User referral traffic is real and not a UA spoofer
- ✓ detect aggressive crawlers
Detect aggressive crawlers burning bandwidth with zero citation value
- ✓ close the measurement loop
Close the measurement loop on an existing crawler allowlist policy
- ✓ baseline ai bot traffic
Baseline AI bot traffic before a large multilingual site launch
- ✓ flag behavioral anomalies
Flag behavioral anomalies like admin-path probing and request bursts
- ✓ run monthly re-classific…
Run monthly re-classification to catch new bots and behavior drift
Yours to keep.
Drag time forward. Watch what stays.
Forever
That's what owning means.
The rented stack
ai writing tool: subscription
expired · access lostanalytics suite: subscription
expired · access lostdesign platform: subscription
expired · access lost(nothing left)
Your forge
-
Stop trusting User-Agent claims: confirm bot identity with 4 independent signals
license: perpetual -
Protect AEO citation revenue by never blocking a legitimate AI crawler
license: perpetual -
Cut wasted bandwidth spend by tiering and rate-limiting low-value bots
license: perpetual -
Get an auditable, replayable reason for every block, rate-limit or allow decision
license: perpetual
subscriptions expire · deeds don't
Everything in the box.
Pick a piece up. Watch it work.
Cloudflare Worker edge classifier with KV-cached IP ranges and behavior buckets
6 parts · one working system · ships instantly by email
From the field · a real case
This wasn’t written at a desk.
The problem
The fix
The result
This wasn't forged for everyone.
- Not for you if you'd rather rent a tool than own one.
- Not for you if you want someone else to run your stack.
- Not for you if you're happy guessing.
Teams running AEO/GEO programs who need to measure and trust their AI bot traffic instead of guessing from raw User-Agent strings.
then this was forged for you.Works with
Universal by design: these run in any AI. Delivered in the open Agent Skills + MCP format (native in Claude); ChatGPT, Gemini, Cursor and Copilot adapt the same files their own way.
- Claude Native format
- ChatGPT Adapts via open standards
- Gemini Adapts via open standards
- Cursor Adapts via open standards
- Copilot Adapts via open standards
Catch what's on your mind.
the air is clear. nothing between you and the forge.
catch a spark: the forge will answer
-
Where does this run, and does it need a particular CDN?
It is built to run at the edge so classification happens in the request path, in real time. You deploy it in front of your traffic rather than parsing server logs after the fact.
-
User-Agents are trivial to fake, so how does this actually prove a bot is who it claims?
A UA string alone is exactly what it does not trust, which is why it cross-checks IP range, ASN and reverse-DNS on top of the User-Agent. A real GPTBot has to pass all four layers, not just send the right name.
-
Does it block the spoofers, or only identify them?
It classifies and scores traffic with a reputation signal; it tells you what is real. Enforcing a block or allow decision is a separate layer, the job of an allowlist system.
-
How is it delivered?
By email right after purchase: ready to run, downloaded instantly, no setup wait.
-
One-time or subscription?
A one-time purchase; no subscription or hidden fees. VAT (20%) is included.
-
Can I get a refund?
As a digital product, it can’t be refunded once downloaded. That’s why we show exactly what’s inside and who it’s for, right here.