ITADN

Blocking CCBot

#211Closedwumpus 创建于 2026-01-22
W
wumpuscommented
The Common Crawl Foundation has an open-access repository, and we're the kind of organization which should be a member of COAR. Why is CCBot in the robots.txt in this repo?
关闭于 2026-01-22 2 条评论