Matching a string of "spider" catches too much
bughelp wanted
There's an entry in robots.json
```json
"Spider": {
"operator": "Unclear at this time.",
"respect": "Unclear at this time.",
"function": "AI Data Scrapers",
"frequency": "Unclear at this time.",
"description": "Description unavailable from darkvisitors.com More info can be found at https://darkvisitors.com/agents/agents/spider"
},
```
which turns out to turn into an overly restrictive regex match in nginx-block-ai-bots.conf, such that any agent with "spider" will get 403ed.
I discovered this when an update from your repo started making some of our automated tests using an agent of "OurCompanyName Test Spider" to fail.
4 条评论