ByteDance
Bytespider— crawlers from ByteDance
Crawler used for TikTok/Doubao AI products.
Bytespider crawls the web for ByteDance's AI products (Doubao, TikTok's recommendation pipeline, and other ByteDance LLMs). It has a reputation for aggressive crawling and not always honoring robots.txt — many site operators block it at the firewall level rather than relying on robots.txt alone.
Vendor
ByteDance
Category
Crawlers (training & indexing)
User-Agent
Bytespider
Documentation
robots.txt snippets
Disallow
User-agent: Bytespider Disallow: /
FAQ
What is Bytespider?
Bytespider crawls the web for ByteDance's AI products (Doubao, TikTok's recommendation pipeline, and other ByteDance LLMs). It has a reputation for aggressive crawling and not always honoring robots.txt — many site operators block it at the firewall level rather than relying on robots.txt alone.
What is the user-agent string for Bytespider?
Bytespider identifies itself with the user-agent token "Bytespider". You can match it in robots.txt with "User-Agent: Bytespider" and route nginx / log-analyzer rules against that token.
How do I block Bytespider in robots.txt?
Add the following block to your /robots.txt — note that well-behaved bots honor this, but not every crawler does:
User-agent: Bytespider
Disallow: /
How can I check whether my site is ready for Bytespider?
Run a free check at https://agentics.page — it audits whether your robots.txt allows the right bots, whether you publish llms.txt and JSON-LD structured data, whether your content is server-rendered, and whether Bytespider can actually consume your site.
Is your domain ready for Bytespider?
agentics checks whether your robots.txt allows the right bots, your llms.txt is in shape, your JSON-LD and SSR content are visible, and whether Bytespider can actually use your domain.
Run free check →