What is specific to Claude
Two things set Claude apart: the index behind its search and the separation of its bots, each with its own robots.txt name.
- Brave Search is built on its own crawl rather than on Google or Bing results, so coverage and ranking differ from both
- ClaudeBot gathers training data; blocking it does not stop Claude-SearchBot or Claude-User
- Claude-User fetches pages a user asks about and, per Anthropic, honours robots.txt, unlike some rival user agents
- Web search is a feature Claude uses when a question needs current information, so many answers are not grounded at all
- Citations appear inline next to the sentences they support, linked to the source page