Anthropic

Claude-Web / Claude-Usersearch-time from Anthropic

On-demand fetches from Claude during conversations.

Claude-Web and Claude-User are Anthropic's on-demand fetchers. They are triggered when a Claude user asks the assistant to read a specific URL, or when Claude decides to follow a link during a conversation. Distinct from ClaudeBot (training crawler).

Vendor
Anthropic
Category
Search-time browsing agents
User-Agent
Claude-Web, Claude-User
Documentation

robots.txt snippets

Allow
User-agent: Claude-Web
Allow: /
User-agent: Claude-User
Allow: /

FAQ

What is Claude-Web / Claude-User?
Claude-Web and Claude-User are Anthropic's on-demand fetchers. They are triggered when a Claude user asks the assistant to read a specific URL, or when Claude decides to follow a link during a conversation. Distinct from ClaudeBot (training crawler).
What is the user-agent string for Claude-Web / Claude-User?
Claude-Web / Claude-User identifies itself with the user-agent token "Claude-Web, Claude-User". You can match it in robots.txt with "User-Agent: Claude-Web" and route nginx / log-analyzer rules against that token.
How do I allow Claude-Web / Claude-User in robots.txt?
Add the following block to your /robots.txt — this explicitly grants Claude-Web / Claude-User access: User-agent: Claude-Web Allow: / User-agent: Claude-User Allow: /
How can I check whether my site is ready for Claude-Web / Claude-User?
Run a free check at https://agentics.page — it audits whether your robots.txt allows the right bots, whether you publish llms.txt and JSON-LD structured data, whether your content is server-rendered, and whether Claude-Web / Claude-User can actually consume your site.

Is your domain ready for Claude-Web / Claude-User?

agentics checks whether your robots.txt allows the right bots, your llms.txt is in shape, your JSON-LD and SSR content are visible, and whether Claude-Web / Claude-User can actually use your domain.

Run free check →

Related agents