# AI-bot tokens by purpose on the Tranco top 1000

Run 2026-09-07 from the server. 1000 domains, 554 with a parseable robots.txt. Tokens and purposes from crawler-registry.json (37 tokens). Definitions of blocked/restricted as in README.md.

## Per token

| token | operator | purpose | named | fully blocked | restricted |
|---|---|---|---|---|---|
| ClaudeBot | Anthropic | training | 121 | 86 (15.5%) | 21 |
| Claude-User | Anthropic | inference | 69 | 41 (7.4%) | 20 |
| Claude-SearchBot | Anthropic | search | 65 | 36 (6.5%) | 23 |
| GPTBot | OpenAI | training | 130 | 88 (15.9%) | 26 |
| OAI-SearchBot | OpenAI | search | 83 | 34 (6.1%) | 34 |
| ChatGPT-User | OpenAI | inference | 91 | 44 (7.9%) | 33 |
| PerplexityBot | Perplexity | search | 114 | 67 (12.1%) | 32 |
| Perplexity-User | Perplexity | inference | 61 | 39 (7.0%) | 14 |
| Google-Extended | Google | training | 112 | 75 (13.5%) | 21 |
| GoogleOther | Google | search | 39 | 30 (5.4%) | 5 |
| Google-CloudVertexBot | Google | inference | 39 | 34 (6.1%) | 5 |
| Bingbot | Microsoft | search | 61 | 5 (0.9%) | 48 |
| Amazonbot | Amazon | search | 72 | 55 (9.9%) | 13 |
| Applebot-Extended | Apple | training | 87 | 68 (12.3%) | 12 |
| meta-externalagent | Meta | training | 86 | 71 (12.8%) | 14 |
| CCBot | Common Crawl | training | 115 | 94 (17.0%) | 13 |
| Bytespider | ByteDance | training | 102 | 90 (16.2%) | 11 |
| DuckAssistBot | DuckDuckGo | inference | 52 | 37 (6.7%) | 14 |
| OAI-AdsBot | OpenAI | ad-verification | 4 | 1 (0.2%) | 2 |
| Google-Agent | Google | inference | 7 | 3 (0.5%) | 4 |
| MistralAI-User | Mistral AI | inference | 40 | 33 (6.0%) | 5 |
| Diffbot | Diffbot | data-aggregation | 66 | 62 (11.2%) | 3 |
| Diffbot-User | Diffbot | inference | 5 | 5 (0.9%) | 0 |
| ImagesiftBot | ImageSift (Hive) | data-aggregation | 52 | 52 (9.4%) | 0 |
| ICC-Crawler | NICT (National Institute of Information and Communications Technology) | training | 32 | 32 (5.8%) | 0 |
| cohere-ai | Cohere | inference | 68 | 59 (10.6%) | 7 |
| Meta-WebIndexer | Meta | search | 27 | 21 (3.8%) | 5 |
| Applebot | Apple | search | 37 | 7 (1.3%) | 25 |
| meta-externalfetcher | Meta | inference | 44 | 34 (6.1%) | 10 |
| meta-externalads | Meta | ad-verification | 10 | 0 (0.0%) | 5 |
| AI2Bot | Allen Institute for AI (Ai2) | training | 40 | 37 (6.7%) | 3 |
| anthropic-ai | Anthropic | training | 66 | 51 (9.2%) | 9 |
| Bravebot | Brave | search | 10 | 7 (1.3%) | 3 |
| kagi-fetcher | Kagi | inference | 2 | 2 (0.4%) | 0 |
| bedrockbot | Amazon | inference | 8 | 8 (1.4%) | 0 |
| cohere-training-data-crawler | Cohere | training | 38 | 35 (6.3%) | 3 |
| DuckDuckBot | DuckDuckGo | search | 20 | 2 (0.4%) | 17 |

## Per purpose (a domain counts once if it fully blocks at least one token of that purpose)

| purpose | tokens | domains blocking ≥1 | share of robots.txt |
|---|---|---|---|
| ad-verification | 2 | 1 | 0.2% |
| data-aggregation | 2 | 67 | 12.1% |
| inference | 12 | 74 | 13.4% |
| search | 10 | 93 | 16.8% |
| training | 11 | 123 | 22.2% |

## The question

- Training tokens: ClaudeBot, GPTBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot, Bytespider, ICC-Crawler, AI2Bot, anthropic-ai, cohere-training-data-crawler.
- User-fetch tokens (registry purpose 'inference'): Claude-User, ChatGPT-User, Perplexity-User, Google-CloudVertexBot, DuckAssistBot, Google-Agent, MistralAI-User, Diffbot-User, cohere-ai, meta-externalfetcher, kagi-fetcher, bedrockbot.
- Domains fully blocking at least one training token: **123** (22.2% of robots.txt).
- Of those, also fully blocking at least one user-fetch token: **73** (59.3%).
- Domains blocking a user-fetch token without blocking any training token: 1.
- Kill rule (README of this script): under 50% → claim stands; over 50% → dead; else partly.

## Domains that block a training token and a user-fetch token

| rank | domain | blocked tokens |
|---|---|---|
| 26 | amazon.com | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot, meta-externalagent, CCBot, Bytespider, DuckAssistBot, MistralAI-User, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, Meta-WebIndexer, meta-externalfetcher, AI2Bot, cohere-training-data-crawler |
| 54 | tiktok.com | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, CCBot, Bytespider, DuckAssistBot, MistralAI-User, AI2Bot, anthropic-ai |
| 60 | yahoo.com | ClaudeBot, GPTBot, ChatGPT-User, PerplexityBot, Google-Extended, CCBot, Bytespider, Diffbot, ImagesiftBot, cohere-ai, anthropic-ai |
| 66 | msn.com | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, DuckAssistBot, MistralAI-User, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, Applebot, meta-externalfetcher, AI2Bot, anthropic-ai, bedrockbot, cohere-training-data-crawler |
| 122 | amazonvideo.com | Claude-User, ChatGPT-User, PerplexityBot, Perplexity-User, GoogleOther, Google-CloudVertexBot, meta-externalagent, CCBot, Bytespider, DuckAssistBot, MistralAI-User, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, Meta-WebIndexer, meta-externalfetcher, AI2Bot, cohere-training-data-crawler |
| 128 | snapchat.com | ClaudeBot, GPTBot, Google-Extended, GoogleOther, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, meta-externalfetcher, AI2Bot, anthropic-ai, cohere-training-data-crawler |
| 153 | nytimes.com | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, Google-CloudVertexBot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, DuckAssistBot, Diffbot, Diffbot-User, ImagesiftBot, cohere-ai, Meta-WebIndexer, meta-externalfetcher, anthropic-ai |
| 177 | soundcloud.com | ClaudeBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Google-Extended, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, cohere-ai, anthropic-ai |
| 193 | cnn.com | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Amazonbot, Applebot-Extended, CCBot, Bytespider, DuckAssistBot, MistralAI-User, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, AI2Bot, anthropic-ai, cohere-training-data-crawler |
| 195 | theguardian.com | ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-CloudVertexBot, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, DuckAssistBot, anthropic-ai |
| 206 | forbes.com | ClaudeBot, GPTBot, PerplexityBot, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, Diffbot, ImagesiftBot, cohere-ai, anthropic-ai |
| 211 | bbc.com | ClaudeBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, Google-CloudVertexBot, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, Diffbot, Diffbot-User, cohere-ai, anthropic-ai |
| 224 | bbc.co.uk | ClaudeBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, Google-CloudVertexBot, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, Diffbot, Diffbot-User, cohere-ai, anthropic-ai |
| 262 | launchpad.net | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, PerplexityBot, Google-Extended, Google-CloudVertexBot, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, DuckAssistBot, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, meta-externalfetcher, AI2Bot, cohere-training-data-crawler |
| 281 | nature.com | ClaudeBot, GPTBot, ChatGPT-User, PerplexityBot, Google-Extended, GoogleOther, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, Diffbot, ImagesiftBot, cohere-ai, anthropic-ai |
| 287 | weibo.com | OAI-SearchBot, ChatGPT-User, Perplexity-User, meta-externalagent, Bytespider |
| 305 | weather.com | ClaudeBot, GPTBot, PerplexityBot, Google-Extended, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, Diffbot, ImagesiftBot, cohere-ai, anthropic-ai |
| 309 | ea.com | meta-externalagent, Bytespider, ICC-Crawler, cohere-ai, AI2Bot, cohere-training-data-crawler |
| 323 | amazon.co.uk | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot, meta-externalagent, CCBot, Bytespider, DuckAssistBot, MistralAI-User, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, Meta-WebIndexer, meta-externalfetcher, AI2Bot, cohere-training-data-crawler |
| 343 | trustpilot.com | Applebot-Extended, meta-externalagent, CCBot, Bytespider, Diffbot, ImagesiftBot, cohere-ai, meta-externalfetcher, anthropic-ai |
| 358 | washingtonpost.com | ClaudeBot, PerplexityBot, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, Diffbot, ImagesiftBot, meta-externalfetcher, anthropic-ai, cohere-training-data-crawler |
| 365 | bloomberg.com | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Google-Extended, Google-CloudVertexBot, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, DuckAssistBot, Diffbot, cohere-ai, meta-externalfetcher, anthropic-ai |
| 380 | amazon.de | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot, meta-externalagent, CCBot, Bytespider, DuckAssistBot, MistralAI-User, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, Meta-WebIndexer, meta-externalfetcher, AI2Bot, cohere-training-data-crawler |
| 416 | espn.com | GPTBot, ChatGPT-User, Google-Extended, CCBot, Bytespider, anthropic-ai |
| 434 | claude.ai | GPTBot, OAI-SearchBot, ChatGPT-User, Google-Extended |
| 470 | cnbc.com | ClaudeBot, Claude-User, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Google-Extended, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, Google-Agent, Diffbot, cohere-ai, anthropic-ai |
| 476 | statista.com | ClaudeBot, GPTBot, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, DuckAssistBot, Diffbot, ImagesiftBot, cohere-ai, meta-externalfetcher, AI2Bot, anthropic-ai, cohere-training-data-crawler |
| 487 | amazon.co.za | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot, meta-externalagent, CCBot, Bytespider, DuckAssistBot, MistralAI-User, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, Meta-WebIndexer, meta-externalfetcher, AI2Bot, cohere-training-data-crawler |
| 490 | quora.com | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Applebot-Extended, Bytespider, Diffbot, ImagesiftBot, Bravebot |
| 494 | aol.com | ClaudeBot, GPTBot, ChatGPT-User, PerplexityBot, Google-Extended, CCBot, Bytespider, Diffbot, ImagesiftBot, cohere-ai, anthropic-ai |
| 497 | amazon.co.jp | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot, meta-externalagent, CCBot, Bytespider, DuckAssistBot, MistralAI-User, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, Meta-WebIndexer, meta-externalfetcher, AI2Bot, cohere-training-data-crawler |
| 523 | ft.com | ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Applebot-Extended, meta-externalagent, CCBot, Bytespider, Diffbot, cohere-ai, meta-externalfetcher, anthropic-ai |
| 541 | patreon.com | ClaudeBot, GPTBot, PerplexityBot, Google-Extended, Google-CloudVertexBot, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider |
| 544 | wired.com | ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, DuckAssistBot, MistralAI-User, Diffbot, cohere-ai, Meta-WebIndexer, meta-externalfetcher, cohere-training-data-crawler |
| 562 | uol.com.br | ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, DuckAssistBot, MistralAI-User, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, Applebot, meta-externalfetcher, AI2Bot, anthropic-ai, cohere-training-data-crawler |
| 564 | tripadvisor.com | ClaudeBot, GPTBot, Google-Extended, Google-CloudVertexBot, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, cohere-ai |
| 567 | usatoday.com | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, DuckAssistBot, Google-Agent, MistralAI-User, Diffbot, Diffbot-User, ImagesiftBot, ICC-Crawler, cohere-ai, Meta-WebIndexer, Applebot, meta-externalfetcher, AI2Bot, anthropic-ai, Bravebot, cohere-training-data-crawler |
| 597 | telegraph.co.uk | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, GoogleOther, Google-CloudVertexBot, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, DuckAssistBot, MistralAI-User, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, meta-externalfetcher, AI2Bot, anthropic-ai, bedrockbot |
| 611 | amazon.fr | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot, meta-externalagent, CCBot, Bytespider, DuckAssistBot, MistralAI-User, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, Meta-WebIndexer, meta-externalfetcher, AI2Bot, cohere-training-data-crawler |
| 619 | sharethrough.com | ClaudeBot, Claude-User, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, GoogleOther, Bingbot, meta-externalagent, CCBot, Bytespider, Diffbot, cohere-ai, AI2Bot, DuckDuckBot |
| 624 | unesco.org | ClaudeBot, GPTBot, Google-Extended, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, Diffbot, cohere-ai, anthropic-ai, Bravebot |
| 629 | nintendo.com | ClaudeBot, GPTBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot, Bytespider, Diffbot, ICC-Crawler, cohere-ai, AI2Bot, cohere-training-data-crawler |
| 634 | sagepub.com | Amazonbot, meta-externalagent, CCBot, Bytespider, Diffbot, ImagesiftBot, cohere-ai |
| 636 | amazon.ca | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot, meta-externalagent, CCBot, Bytespider, DuckAssistBot, MistralAI-User, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, Meta-WebIndexer, meta-externalfetcher, AI2Bot, cohere-training-data-crawler |
| 643 | techcrunch.com | ClaudeBot, GPTBot, ChatGPT-User, Google-Extended, Applebot-Extended, CCBot, Bytespider, Diffbot, cohere-ai, anthropic-ai |
| 659 | elpais.com | ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Amazonbot, CCBot, Bytespider, DuckAssistBot, MistralAI-User |
| 660 | dailymail.co.uk | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, Google-CloudVertexBot, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, DuckAssistBot, MistralAI-User, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, meta-externalfetcher, AI2Bot, anthropic-ai, bedrockbot |
| 666 | primevideo.com | Claude-User, ChatGPT-User, PerplexityBot, Perplexity-User, GoogleOther, Google-CloudVertexBot, meta-externalagent, CCBot, Bytespider, DuckAssistBot, MistralAI-User, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, Meta-WebIndexer, meta-externalfetcher, AI2Bot, cohere-training-data-crawler |
| 668 | amazon.in | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot, meta-externalagent, CCBot, Bytespider, DuckAssistBot, MistralAI-User, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, Meta-WebIndexer, meta-externalfetcher, AI2Bot, cohere-training-data-crawler |
| 693 | lemonde.fr | ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, GoogleOther, Google-CloudVertexBot, Amazonbot, Applebot-Extended, CCBot, Bytespider, DuckAssistBot, Google-Agent, MistralAI-User, Diffbot, ICC-Crawler, cohere-ai, Meta-WebIndexer, AI2Bot, anthropic-ai, Bravebot, kagi-fetcher, bedrockbot, cohere-training-data-crawler |
| 694 | dailymail.com | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, Google-CloudVertexBot, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, DuckAssistBot, MistralAI-User, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, meta-externalfetcher, AI2Bot, anthropic-ai, bedrockbot |
| 699 | cnet.com | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, DuckAssistBot, MistralAI-User, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, Meta-WebIndexer, Applebot, meta-externalfetcher, AI2Bot, anthropic-ai, Bravebot, kagi-fetcher, cohere-training-data-crawler |
| 703 | amazon.es | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot, meta-externalagent, CCBot, Bytespider, DuckAssistBot, MistralAI-User, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, Meta-WebIndexer, meta-externalfetcher, AI2Bot, cohere-training-data-crawler |
| 707 | amazon.it | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot, meta-externalagent, CCBot, Bytespider, DuckAssistBot, MistralAI-User, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, Meta-WebIndexer, meta-externalfetcher, AI2Bot, cohere-training-data-crawler |
| 724 | spiegel.de | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, MistralAI-User, Diffbot, ImagesiftBot, cohere-ai, AI2Bot, anthropic-ai, cohere-training-data-crawler |
| 764 | ted.com | ClaudeBot, GPTBot, Google-Extended, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, Diffbot, ImagesiftBot, cohere-ai, anthropic-ai, cohere-training-data-crawler |
| 774 | bild.de | PerplexityBot, Perplexity-User, Amazonbot, Applebot-Extended, CCBot, Bytespider, DuckAssistBot, MistralAI-User, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, AI2Bot, bedrockbot, cohere-training-data-crawler |
| 780 | mercadolivre.com.br | ClaudeBot, Claude-User, GPTBot, ChatGPT-User, PerplexityBot, Perplexity-User, Amazonbot |
| 789 | apnews.com | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, PerplexityBot, Amazonbot, Applebot-Extended, CCBot, cohere-ai, anthropic-ai |
| 792 | welt.de | PerplexityBot, Perplexity-User, Applebot-Extended, CCBot, Bytespider, DuckAssistBot, MistralAI-User, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, AI2Bot, bedrockbot, cohere-training-data-crawler |
| 799 | people.com | ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Google-CloudVertexBot, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, DuckAssistBot, MistralAI-User, ImagesiftBot, cohere-ai, Meta-WebIndexer, meta-externalfetcher, anthropic-ai, Bravebot, cohere-training-data-crawler |
| 833 | theverge.com | ClaudeBot, Claude-User, Claude-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, Google-CloudVertexBot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, DuckAssistBot, MistralAI-User, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, meta-externalfetcher, AI2Bot, anthropic-ai, cohere-training-data-crawler |
| 845 | amazon.com.br | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot, meta-externalagent, CCBot, Bytespider, DuckAssistBot, MistralAI-User, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, Meta-WebIndexer, meta-externalfetcher, AI2Bot, cohere-training-data-crawler |
| 849 | amazon.com.au | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot, meta-externalagent, CCBot, Bytespider, DuckAssistBot, MistralAI-User, Diffbot, ImagesiftBot, ICC-Crawler, cohere-ai, Meta-WebIndexer, meta-externalfetcher, AI2Bot, cohere-training-data-crawler |
| 859 | nbcnews.com | ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, MistralAI-User, Diffbot, cohere-ai, anthropic-ai |
| 860 | figma.com | ClaudeBot, GPTBot, Google-Extended, CCBot, cohere-ai |
| 947 | mercadolibre.com.ar | ClaudeBot, Claude-User, GPTBot, ChatGPT-User, PerplexityBot, Perplexity-User, Amazonbot |
| 948 | investopedia.com | ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Google-CloudVertexBot, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, DuckAssistBot, MistralAI-User, ImagesiftBot, cohere-ai, Meta-WebIndexer, meta-externalfetcher, anthropic-ai, Bravebot, cohere-training-data-crawler |
| 958 | repubblica.it | GPTBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, DuckAssistBot, Diffbot, ImagesiftBot, cohere-ai, Applebot, meta-externalfetcher, anthropic-ai |
| 969 | corriere.it | ClaudeBot, GPTBot, Google-Extended, Google-CloudVertexBot, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, Diffbot-User, Applebot, anthropic-ai |
| 970 | healthline.com | ClaudeBot, GPTBot, Amazonbot, Applebot-Extended, meta-externalagent, CCBot, Bytespider, Diffbot, anthropic-ai, bedrockbot |
| 989 | pixiv.net | ClaudeBot, GPTBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot, Diffbot, cohere-ai, anthropic-ai |
| 992 | elmundo.es | GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Google-Extended, CCBot, anthropic-ai |