Agents
Every user-agent token the classifier knows. Curated entries were checked against the vendor's own documentation; the long tail is vendored from the ai.robots.txt project and refreshed by hand.
157
known agent definitions
78 curated, 79 from ai.robots.txt
16
published IP-range lists
vendor-published reference lists
Documented agents
| Agent | Operator | Category | Reported robots.txt policy | Verification |
|---|---|---|---|---|
| AI2Bot | Allen Institute for AI | Training crawler | reported respect | user agent only |
| Amazonbot | Amazon | Training crawler | reported respect | user agent only |
| Amzn-SearchBot | Amazon | AI search index | reported respect | user agent only |
| Amzn-User | Amazon | User-triggered fetch | reported respect | user agent only |
| ApifyBot | Apify | AI tooling / data provider | unknown | user agent only |
| Applebot | Apple | Search engine | reported respect | published IP ranges |
| Baiduspider | Baidu | Search engine | reported respect | user agent only |
| Bingbot | Microsoft | Search engine | reported respect | reverse DNS documented; not checked here |
| Brightbot | Bright Data | AI tooling / data provider | unknown | user agent only |
| Buy For Me | Amazon | Browsing agent | unknown | user agent only |
| Bytespider | ByteDance | Training crawler | reported noncompliance | user agent only |
| CCBot | Common Crawl | Training crawler | reported respect | published IP ranges |
| ChatGPT agent | OpenAI | Browsing agent | reported noncompliance | user agent only |
| ChatGPT-User | OpenAI | User-triggered fetch | mixed reports | published IP ranges |
| Claude Code | Anthropic | Coding agent | unknown | user agent only |
| Claude-SearchBot | Anthropic | AI search index | reported respect | published IP ranges |
| Claude-User | Anthropic | User-triggered fetch | reported respect | published IP ranges |
| ClaudeBot | Anthropic | Training crawler | reported respect | published IP ranges |
| cohere-ai | Cohere | Training crawler | unknown | user agent only |
| Crawl4AI | Crawl4AI | AI tooling / data provider | unknown | user agent only |
| Cursor | Anysphere | Coding agent | unknown | user agent only |
| DeepSeekBot | DeepSeek | Training crawler | unknown | user agent only |
| Devin | Cognition | Browsing agent | unknown | user agent only |
| Diffbot | Diffbot | AI tooling / data provider | mixed reports | user agent only |
| Diffbot-User | Diffbot | User-triggered fetch | unknown | user agent only |
| DuckAssistBot | DuckDuckGo | User-triggered fetch | reported respect | published IP ranges |
| DuckDuckBot | DuckDuckGo | Search engine | reported respect | user agent only |
| Exa | Exa | AI search index | unknown | user agent only |
| FacebookBot | Meta | Training crawler | reported respect | user agent only |
| Firecrawl | Firecrawl | AI tooling / data provider | unknown | user agent only |
| Gemini CLI | Coding agent | unknown | user agent only | |
| Gemini Deep Research | User-triggered fetch | reported noncompliance | user agent only | |
| Google-Agent | Browsing agent | reported noncompliance | published IP ranges | |
| Google-CloudVertexBot | AI tooling / data provider | reported respect | published IP ranges | |
| Google-GeminiNotebook | User-triggered fetch | reported noncompliance | published IP ranges | |
| GoogleAgent-Mariner / URLContext | Browsing agent | reported noncompliance | user agent only | |
| Googlebot | Search engine | reported respect | published IP ranges | |
| GoogleOther | AI tooling / data provider | reported respect | published IP ranges | |
| GPTBot | OpenAI | Training crawler | reported respect | published IP ranges |
| ImagesiftBot | Hive | AI tooling / data provider | unknown | user agent only |
| img2dataset | various | Training crawler | unknown | user agent only |
| Kimi-SearchBot | Moonshot AI | AI search index | unknown | user agent only |
| Kimi-User | Moonshot AI | User-triggered fetch | unknown | user agent only |
| KimiBot | Moonshot AI | Training crawler | unknown | user agent only |
| Lightpanda | Lightpanda | Browsing agent | unknown | user agent only |
| Link previewers | various | Other bot | unknown | user agent only |
| Manus | Butterfly Effect | Browsing agent | unknown | user agent only |
| Meta-ExternalAgent | Meta | Training crawler | reported respect | user agent only |
| Meta-ExternalFetcher | Meta | User-triggered fetch | mixed reports | user agent only |
| Meta-WebIndexer | Meta | AI search index | reported respect | user agent only |
| MistralAI-Index | Mistral AI | AI search index | reported respect | published IP ranges |
| MistralAI-Training | Mistral AI | Training crawler | reported respect | user agent only |
| MistralAI-User | Mistral AI | User-triggered fetch | reported respect | published IP ranges |
| Nova Act | Amazon | Browsing agent | unknown | user agent only |
| OAI-AdsBot | OpenAI | AI tooling / data provider | unknown | published IP ranges |
| OAI-SearchBot | OpenAI | AI search index | reported respect | published IP ranges |
| omgili / webz.io | Webz.io | Training crawler | unknown | user agent only |
| opencode | SST | Coding agent | unknown | user agent only |
| PanguBot | Huawei | Training crawler | unknown | user agent only |
| Perplexity-User | Perplexity | User-triggered fetch | reported noncompliance | published IP ranges |
| PerplexityBot | Perplexity | AI search index | reported respect | published IP ranges |
| PetalBot | Huawei | AI search index | reported respect | user agent only |
| SEO crawlers | various | Other bot | unknown | user agent only |
| Tavily | Tavily | AI tooling / data provider | unknown | user agent only |
| TikTokSpider | ByteDance | Training crawler | unknown | user agent only |
| Timpibot | Timpi | AI search index | unknown | user agent only |
| TongyiBot | Alibaba | Training crawler | unknown | user agent only |
| Trae | ByteDance | Coding agent | unknown | user agent only |
| TwinAgent | Twin | Browsing agent | unknown | user agent only |
| Uptime monitors & platform bots | various | Other bot | unknown | user agent only |
| Yahoo Slurp | Yahoo | Search engine | reported respect | user agent only |
| YandexAdditional | Yandex | Training crawler | unknown | user agent only |
| YandexBot | Yandex | Search engine | reported respect | user agent only |
| YiyanBot | Baidu | Training crawler | unknown | user agent only |
| YouBot | You.com | AI search index | reported respect | user agent only |
Control tokens
These names exist only for robots.txt rules. Blocking them is meaningful; seeing them in logs is not expected.
| Agent | Operator | Category | Reported robots.txt policy | Verification |
|---|---|---|---|---|
| anthropic-ai / Claude-Web (legacy) | Anthropic | Training crawler | unknown | control token only |
| Applebot-Extended | Apple | Training crawler | reported respect | control token only |
| Google-Extended | Training crawler | reported respect | control token only |
Coding agents on GitHub
Agents that open pull requests under their own identity. Daily counts come from the GitHub search API across all of GitHub; watched-repository counts come from our own collector.
| Agent | Vendor | Identified by | Watched-repo PRs (90d) | Last activity |
|---|---|---|---|---|
| GitHub Copilot coding agent | GitHub | bot account #198982749 | 871 | 11d ago |
| Claude Code | Anthropic | bot account #209825114 | 0 | – |
| OpenAI Codex (connector) | OpenAI | bot account #199175422 | 0 | – |
| Devin | Cognition | bot account #158243242 | 595 | 30h ago |
| Jules | bot account #161369871 | 0 | – | |
| Gemini Code Assist | bot account #176961590 | 0 | – | |
| Cursor | Anysphere | bot account #206951365 | 0 | – |
| CodeRabbit | CodeRabbit | bot account #136622811 | 0 | – |
| Amazon Q Developer | Amazon | bot account #208079219 | 0 | – |
| Factory Droid | Factory | bot account #138933559 | 0 | – |
| Greptile | Greptile | bot account #165735046 | 0 | – |
| Kiro | Amazon | bot account #245459735 | 0 | – |
| OpenHands | All Hands AI | bot account #175740463 | 0 | – |
| Codex (codex/ branches) | OpenAI | branch prefix codex/ | 0 | – |
| Claude (claude/ branches) | Anthropic | branch prefix claude/ | 0 | – |
| Cursor (cursor/ branches) | Anysphere | branch prefix cursor/ | 0 | – |
Long tail (79)
Vendored from ai-robots-txt/ai.robots.txt. Categories are inferred from the list's descriptions; a few entries are corrected by hand.
| Agent | Operator | Category | Reported robots.txt policy | Verification |
|---|---|---|---|---|
| AddSearchBot | Unknown operator | AI search index | unknown | user agent only |
| AgentTimes | [The Agent Times](https://theagenttimes.com/about) | Training crawler | unknown | user agent only |
| AI2Bot-DeepResearchEval | Ai2, a non-profit AI research institute | User-triggered fetch | unknown | user agent only |
| aiHitBot | [aiHit](https://www.aihitdata.com/about) | Training crawler | reported respect | user agent only |
| AIWebIndex | [Lyrenth](https://lyrenth.com) | AI search index | reported respect | user agent only |
| amazon-kendra | Amazon | AI search index | reported respect | user agent only |
| amazon-QBusiness | Unknown operator | User-triggered fetch | unknown | user agent only |
| Andibot | [Andi](https://andisearch.com/) | User-triggered fetch | unknown | user agent only |
| Anomura | [Direqt](https://direqt.ai) | AI search index | reported respect | user agent only |
| ApifyWebsiteContentCrawler | Unknown operator | AI tooling / data provider | unknown | user agent only |
| Aranet-SearchBot | Unknown operator | Browsing agent | unknown | user agent only |
| atlassian-bot | [Atlassian](https://www.atlassian.com) | User-triggered fetch | reported respect | user agent only |
| Awario | Awario | Training crawler | unknown | user agent only |
| AzureAI-SearchBot | Unknown operator | AI search index | unknown | user agent only |
| bedrockbot | [Amazon](https://amazon.com) | Training crawler | reported respect | user agent only |
| bigsur.ai | Big Sur AI that fetches website content to enable AI-powered web agents, sales assistants, and content marketing solutions for busi… | User-triggered fetch | unknown | user agent only |
| Bravebot | https://safe.search.brave.com/help/brave-search-crawler | AI tooling / data provider | reported respect | user agent only |
| BuddyBot | [BuddyBotLearning](https://www.buddybotlearning.com) | AI tooling / data provider | unknown | user agent only |
| Channel3Bot | Unknown operator | AI search index | unknown | user agent only |
| ChatGLM-Spider | Unknown operator | Training crawler | unknown | user agent only |
| Cloudflare-AutoRAG | [Cloudflare](https://developers.cloudflare.com/autorag) | AI search index | reported respect | user agent only |
| CloudVertexBot | Unknown operator | Training crawler | unknown | user agent only |
| Cotoyogi | [ROIS](https://ds.rois.ac.jp/en_center8/en_crawler/) | Training crawler | reported respect | user agent only |
| CragCrawler | CragSoftware, a Brazil-based software company specializing in data engineering and AI web scraping services | AI tooling / data provider | unknown | user agent only |
| Crawlspace | [Crawlspace](https://crawlspace.dev) | AI tooling / data provider | reported respect | user agent only |
| Datenbank Crawler | Datenbank | Training crawler | unknown | user agent only |
| Echobot Bot | Echobox | Training crawler | unknown | user agent only |
| EchoboxBot | [Echobox](https://echobox.com) | AI tooling / data provider | unknown | user agent only |
| Factset_spyderbot | [Factset](https://www.factset.com/ai) | Training crawler | unknown | user agent only |
| FriendlyCrawler | Unknown | Training crawler | reported respect | user agent only |
| GeistHaus-PageFetcher | GeistHaus, a company developing AI systems for therapy and psychological assessment | User-triggered fetch | unknown | user agent only |
| Google-Firebase | AI tooling / data provider | unknown | user agent only | |
| HenkBot | Unknown operator | AI tooling / data provider | unknown | user agent only |
| iAskBot | Unknown operator | Browsing agent | unknown | user agent only |
| iaskspider | Unknown operator | Browsing agent | unknown | user agent only |
| ICC-Crawler | [NICT](https://nict.go.jp) | Training crawler | reported respect | user agent only |
| imageSpider | Unknown operator | Training crawler | unknown | user agent only |
| ISSCyberRiskCrawler | [ISS-Corporate](https://iss-cyber.com) | Training crawler | reported noncompliance | user agent only |
| kagi-fetcher | Kagi that fetches web content to answer user queries through Kagi AI, their suite of AI-powered tools including Assistant, Res… | User-triggered fetch | unknown | user agent only |
| Kangaroo Bot | Unknown operator | Training crawler | unknown | user agent only |
| KlaviyoAIBot | [Klaviyo](https://www.klaviyo.com) | User-triggered fetch | reported respect | user agent only |
| KunatoCrawler | Unknown operator | Browsing agent | unknown | user agent only |
| laion-huggingface-processor | Unknown operator | Training crawler | unknown | user agent only |
| LAIONDownloader | [Large-scale Artificial Intelligence Open Network](https://laion.ai/) | AI search index | reported noncompliance | user agent only |
| LinerBot | Unknown operator | User-triggered fetch | unknown | user agent only |
| Linguee Bot | [Linguee](https://www.linguee.com) | AI tooling / data provider | reported noncompliance | user agent only |
| LinkupBot | Unknown operator | AI search index | unknown | user agent only |
| Mozilla-Tabstack | Mozilla that performs programmatic, AI-driven interactions with web content through Tabstack | AI tooling / data provider | reported respect | user agent only |
| MyCentralAIScraperBot | Unknown operator | Training crawler | unknown | user agent only |
| NagetBot | Naget Inc (founded by Chris Samarinas, headquarter in Amherst, Massachusetts) | Training crawler | unknown | user agent only |
| netEstate Imprint Crawler | netEstate | Training crawler | unknown | user agent only |
| newsai | Unknown operator | Training crawler | unknown | user agent only |
| NotebookLM | Unknown operator | User-triggered fetch | unknown | user agent only |
| OpenAI | [OpenAI](https://openai.com) | AI tooling / data provider | reported respect | user agent only |
| Panscient | [Panscient](https://panscient.com) | Training crawler | reported respect | user agent only |
| PhindBot | [phind](https://www.phind.com/) | User-triggered fetch | unknown | user agent only |
| Poggio-Citations | Poggio, a company that provides AI sales enablement tools for creating tailored narratives, business cases, and account plan… | User-triggered fetch | unknown | user agent only |
| Poseidon Research Crawler | [Poseidon Research](https://www.poseidonresearch.com) | AI search index | unknown | user agent only |
| QualifiedBot | [Qualified](https://www.qualified.com) | User-triggered fetch | unknown | user agent only |
| Querit-SearchBot | Querit that indexes web content for their search API service, which is designed to provide real-time search results for larg… | AI tooling / data provider | unknown | user agent only |
| QueritBot | Querit, a company providing a search API for large language model integration | AI tooling / data provider | unknown | user agent only |
| QuillBot | [Quillbot](https://quillbot.com) | AI tooling / data provider | unknown | user agent only |
| Reflectionbot | [Reflection](https://reflection.ai/) | Browsing agent | unknown | user agent only |
| SBIntuitionsBot | [SB Intuitions](https://www.sbintuitions.co.jp/en/) | AI tooling / data provider | reported respect | user agent only |
| SemrushBot-OCOB | [Semrush](https://www.semrush.com/) | AI tooling / data provider | reported respect | user agent only |
| SemrushBot-SWA | [Semrush](https://www.semrush.com/) | User-triggered fetch | reported respect | user agent only |
| Shap-User | Unknown operator | User-triggered fetch | unknown | user agent only |
| ShapBot | [Parallel](https://parallel.ai) | AI tooling / data provider | reported respect | user agent only |
| Sidetrade indexer bot | [Sidetrade](https://www.sidetrade.com) | Training crawler | unknown | user agent only |
| Terra Cotta | Unknown operator | AI tooling / data provider | unknown | user agent only |
| TerraCotta | [Ceramic AI](https://ceramic.ai/) | AI tooling / data provider | reported respect | user agent only |
| Thinkbot | [Thinkbot](https://www.thinkbot.agency) | AI tooling / data provider | reported noncompliance | user agent only |
| UseAI | Unknown operator | User-triggered fetch | unknown | user agent only |
| VelenPublicWebCrawler | [Velen Crawler](https://velen.io) | Training crawler | reported respect | user agent only |
| WARDBot | WEBSPARK | Training crawler | unknown | user agent only |
| Webzio-Extended | Unknown operator | Training crawler | unknown | user agent only |
| wpbot | [QuantumCloud](https://www.quantumcloud.com) | AI tooling / data provider | unknown | user agent only |
| WRTNBot | Unknown operator | Browsing agent | unknown | user agent only |
| ZanistaBot | Unknown operator | AI search index | unknown | user agent only |