New agents
Two public lists are diffed daily: the ai.robots.txt catalogue of AI crawler user agents, and the registry of agents that sign their requests with Web Bot Auth. Anything that appears for the first time is recorded here with the date we first saw it.
175
crawler tokens catalogued
ai.robots.txt
88
signing-registry entries
Web Bot Auth registry
9
crawler tokens first listed in the last 30 days
dated from the ai.robots.txt git history
Signing-registry entries
about Web Bot Auth ↗These registry entries advertise signing keys. Membership does not establish the identity of a visitor here. Signature-header presence is recorded without cryptographic verification.
Crawler tokens
ai.robots.txt ↗| Token | Operator | Function | First listed |
|---|---|---|---|
| DoubaoBot | ByteDance | AI crawler for ByteDance's Doubao AI assistant. | 19d ago |
| ERNIEBot | Baidu | Collects public web content for Baidu's ERNIE large language models. | 19d ago |
| QwenBot | Alibaba | Collects public web content for Alibaba's Qwen (Tongyi Qianwen) large language m | 19d ago |
| MistralAI-Index | Mistral AI | Indexes web content for Mistral AI's search, used to answer questions in Le Chat | 19d ago |
| OAI-AdsBot | [OpenAI](https://openai.com) | Validates and targets ads on ChatGPT. | 21d ago |
| MistralAI-Training | [Mistral AI](https://mistral.ai) | AI Data Scrapers | 21d ago |
| KimiBot | [Moonshot AI](https://www.moonshot.ai) | AI Data Scrapers | 21d ago |
| Kimi-SearchBot | [Moonshot AI](https://www.moonshot.ai) | AI Search Crawlers | 21d ago |
| Diffbot-User | [Diffbot](https://www.diffbot.com/) | AI Assistants | 28d ago |
| Reflectionbot | [Reflection](https://reflection.ai/) | Undocumented AI Agents | 35d ago |
| ExaSearchBot | [Exa](https://exa.ai) | AI Search Crawlers | 40d ago |
| Lightpanda | Anyone who downloads the Lightpanda client. Possibly being used by a [Grok-adjacent](https://github.com/lightpanda-io/browser/issues/3156#issuecomment-5217843616) organization's botnet. | AI Data Scrapers | 50d ago |
| TongyiBot | Alibaba that fetches web content for the Tongyi Qianwen assistant and related Qwen-generated answers | AI Assistants | 3mo ago |
| YiyanBot | Baidu that fetches web content for the yiyan | AI Assistants | 3mo ago |
| AIWebIndex | [Lyrenth](https://lyrenth.com) | AI Search Crawlers | 3mo ago |
| amazon-QBusiness | Unclear at this time. | AI Assistants | 3mo ago |
| Mozilla-Tabstack | Mozilla that performs programmatic, AI-driven interactions with web content through Tabstack | AI Data Providers | 3mo ago |
| Cursor | Unclear at this time. | AI Coding Agents | 3mo ago |
| Querit-SearchBot | Querit that indexes web content for their search API service, which is designed to provide real-time search results for larg… | AI Data Providers | 3mo ago |
| CragCrawler | CragSoftware, a Brazil-based software company specializing in data engineering and AI web scraping services | AI Data Providers | 3mo ago |
| GeistHaus-PageFetcher | GeistHaus, a company developing AI systems for therapy and psychological assessment | AI Assistants | 3mo ago |
| GoogleAgent-URLContext | Google that retrieves web content on behalf of Gemini API users | AI Assistants | 3mo ago |
| Kimi-User | Moonshot AI that fetches web content on behalf of users interacting with Kimi | AI Assistants | 3mo ago |
| QueritBot | Querit, a company providing a search API for large language model integration | AI Data Providers | 3mo ago |
| UseAI | Unclear at this time. | AI Assistants | 3mo ago |
| AgentTimes | [The Agent Times](https://theagenttimes.com/about) | Data Scraper from RSS Feeds. | 4mo ago |
| Shap-User | Unclear at this time. | AI Assistants | 4mo ago |
| Claude-Code | Unclear at this time. | AI Coding Agents | 5mo ago |
| Trae | Unclear at this time. | AI Coding Agents | 5mo ago |
| Terra Cotta | Unclear at this time. | AI Data Providers | 5mo ago |
| Code | Unclear at this time. | AI Coding Agents | 5mo ago |
| Google-Gemini-CLI | Unclear at this time. | AI Coding Agents | 5mo ago |
| HenkBot | Unclear at this time. | AI Data Providers | 5mo ago |
| opencode | Unclear at this time. | AI Coding Agents | 5mo ago |
| Brightbot | Unclear at this time. | AI Data Providers | 5mo ago |
| NagetBot | Naget Inc (founded by Chris Samarinas, headquarter in Amherst, Massachusetts) | AI data scraper | 5mo ago |
| newsai | Unclear at this time. | AI data scraper | 5mo ago |
| Google-Agent | Unclear at this time. | AI Agents | 6mo ago |
| Aranet-SearchBot | Unclear at this time. | Undocumented AI Agents | 6mo ago |
| ApifyBot | Unclear at this time. | AI Data Providers | 7mo ago |
| ApifyWebsiteContentCrawler | Unclear at this time. | AI Data Providers | 7mo ago |
| kagi-fetcher | Kagi that fetches web content to answer user queries through Kagi AI, their suite of AI-powered tools including Assistant, Res… | AI Assistants | 8mo ago |
| Amzn-SearchBot | Unclear at this time. | AI Search Crawlers | 8mo ago |
| AzureAI-SearchBot | Unclear at this time. | AI Search Crawlers | 8mo ago |
| Amzn-User | Amazon, used for fetching web content to answer user queries through Alexa and other Amazon AI services | AI Assistants | 8mo ago |
| TavilyBot | Unclear at this time. | AI Data Providers | 9mo ago |
| Channel3Bot | Unclear at this time. | AI Search Crawlers | 10mo ago |
| TwinAgent | Twin, a platform that creates automated workers to perform tasks by integrating with APIs and controlling web applications through browser automa… | AI Agents | 10mo ago |
| Crawl4AI | Unclear at this time. | Undocumented AI Agents | 10mo ago |
| Manus-User | Butterfly Effect, a company based in China | AI Agents | 10mo ago |
| ChatGLM-Spider | Unclear at this time. | AI Data Scrapers | 10mo ago |
| WRTNBot | Unclear at this time. | Undocumented AI Agents | 10mo ago |
| Spider | Unclear at this time. | AI Data Scrapers | 10mo ago |
| LCC | Unclear at this time. | AI Data Scrapers | 10mo ago |
| laion-huggingface-processor | Unclear at this time. | AI Data Scrapers | 10mo ago |
| KunatoCrawler | Unclear at this time. | Undocumented AI Agents | 10mo ago |
| imageSpider | Unclear at this time. | AI Data Scrapers | 10mo ago |
| iaskspider | Unclear at this time. | Undocumented AI Agents | 10mo ago |
| iAskBot | Unclear at this time. | Undocumented AI Agents | 10mo ago |
| ZanistaBot | Unclear at this time. | AI Search Crawlers | 10mo ago |
| Poggio-Citations | Poggio, a company that provides AI sales enablement tools for creating tailored narratives, business cases, and account plan… | AI Assistants | 10mo ago |
| LinkupBot | Unclear at this time. | AI Search Crawlers | 10mo ago |
| AI2Bot-DeepResearchEval | Ai2, a non-profit AI research institute | AI Assistants | 10mo ago |
| LAIONDownloader | [Large-scale Artificial Intelligence Open Network](https://laion.ai/) | AI tools and models for machine learning research. | 10mo ago |
| amazon-kendra | Amazon | Collects data for AI natural language search | 10mo ago |
| NotebookLM | Unclear at this time. | AI Assistants | 11mo ago |
| KlaviyoAIBot | [Klaviyo](https://www.klaviyo.com) | AI Assistants | 11mo ago |
| atlassian-bot | [Atlassian](https://www.atlassian.com) | AI search, assistants and agents | 11mo ago |
| AmazonBuyForMe | [Amazon](https://amazon.com) | AI Agents | 11mo ago |
| BuddyBot | [BuddyBotLearning](https://www.buddybotlearning.com) | AI Learning Companion | 11mo ago |
| Anomura | [Direqt](https://direqt.ai) | Collects data for AI search | 11mo ago |
| Bravebot | https://safe.search.brave.com/help/brave-search-crawler | AI Data Providers | 11mo ago |
| Cloudflare-AutoRAG | [Cloudflare](https://developers.cloudflare.com/autorag) | Collects data for AI search | 11mo ago |
| Google-NotebookLM | Unclear at this time. | AI Assistants | 12mo ago |
| DeepSeekBot | DeepSeek | Training language models and improving AI products | 12mo ago |
| TerraCotta | [Ceramic AI](https://ceramic.ai/) | AI Data Providers | 13mo ago |
| meta-webindexer | [Meta](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) | AI Assistants | 13mo ago |
| ShapBot | [Parallel](https://parallel.ai) | AI Data Providers | 13mo ago |
| Google-Firebase | Used as part of AI apps developed by users of Google's Firebase AI products. | 13mo ago | |
| OpenAI | [OpenAI](https://openai.com) | Unclear at this time. | 14mo ago |
| bigsur.ai | Big Sur AI that fetches website content to enable AI-powered web agents, sales assistants, and content marketing solutions for busi… | AI Assistants | 14mo ago |
| CloudVertexBot | Unclear at this time. | AI Data Scrapers | 14mo ago |
| LinerBot | Unclear at this time. | AI Assistants | 14mo ago |
| ChatGPT Agent | [OpenAI](https://openai.com) | AI Agents | 14mo ago |
| GoogleAgent-Mariner | AI Agents | 15mo ago | |
| Thinkbot | [Thinkbot](https://www.thinkbot.agency) | Insights on AI integration and automation. | 15mo ago |
| netEstate Imprint Crawler | netEstate | AI Data Scrapers | 15mo ago |
| Gemini-Deep-Research | Unclear at this time. | AI Assistants | 15mo ago |
| MistralAI-User | Mistral | AI Assistants | 15mo ago |
| Awario | Awario | AI Data Scrapers | 15mo ago |
| WARDBot | WEBSPARK | AI Data Scrapers | 15mo ago |
| Devin | Devin AI | AI Coding Agents | 15mo ago |
| Datenbank Crawler | Datenbank | AI Data Scrapers | 15mo ago |
| Echobot Bot | Echobox | AI Data Scrapers | 15mo ago |
| Poseidon Research Crawler | [Poseidon Research](https://www.poseidonresearch.com) | AI research crawler | 16mo ago |
| MyCentralAIScraperBot | Unclear at this time. | AI data scraper | 16mo ago |
| EchoboxBot | [Echobox](https://echobox.com) | Data collection to support AI-powered products. | 16mo ago |
| bedrockbot | [Amazon](https://amazon.com) | Data scraping for custom AI applications. | 16mo ago |
| SBIntuitionsBot | [SB Intuitions](https://www.sbintuitions.co.jp/en/) | Uses data gathered in AI development and information analysis. | 16mo ago |
| Panscient | [Panscient](https://panscient.com) | Data collection and analysis using machine learning and AI. | 16mo ago |
| QuillBot | [Quillbot](https://quillbot.com) | Company offers AI detection, writing tools and other services. | 16mo ago |
| quillbot.com | [Quillbot](https://quillbot.com) | Company offers AI detection, writing tools and other services. | 16mo ago |
| YandexAdditional | [Yandex](https://yandex.ru) | Scrapes/analyzes data for the YandexGPT LLM. | 16mo ago |
| YandexAdditionalBot | [Yandex](https://yandex.ru) | Scrapes/analyzes data for the YandexGPT LLM. | 16mo ago |
| PhindBot | [phind](https://www.phind.com/) | AI Assistants | 16mo ago |
| Andibot | [Andi](https://andisearch.com/) | Search engine using generative AI, AI Search Assistant | 16mo ago |
| wpbot | [QuantumCloud](https://www.quantumcloud.com) | Live chat support and lead generation. | 16mo ago |
| MistralAI-User/1.0 | Mistral AI | Takes action based on user prompts. | 17mo ago |
| QualifiedBot | [Qualified](https://www.qualified.com) | AI Assistants | 17mo ago |
| Google-CloudVertexBot | Build and manage AI models for businesses employing Vertex AI | 17mo ago | |
| FirecrawlAgent | Firecrawl that extracts web content and converts it into structured data for use in LLM and AI applications | AI Data Providers | 17mo ago |
| Cotoyogi | [ROIS](https://ds.rois.ac.jp/en_center8/en_crawler/) | AI LLM Scraper. | 17mo ago |
| Factset_spyderbot | [Factset](https://www.factset.com/ai) | AI model training. | 17mo ago |
| TikTokSpider | ByteDance | LLM training. | 17mo ago |
| NovaAct | Unclear at this time. | AI Agents | 18mo ago |
| Operator | Unclear at this time. | AI Agents | 18mo ago |
| Perplexity-User | [Perplexity](https://www.perplexity.ai/) | AI Assistants | 18mo ago |
| Claude-User | [Anthropic](https://www.anthropic.com) | AI Assistants | 18mo ago |
| Claude-SearchBot | [Anthropic](https://www.anthropic.com) | Claude-SearchBot navigates the web to improve search result quality for users. I | 18mo ago |
| Brightbot 1.0 | https://brightdata.com/brightbot | LLM/AI training. | 20mo ago |
The first run records the whole list at once; from then on only genuinely new entries appear with a fresh date.