nginx expressions escapes too much and matches case-insensitive
help wanted
The regular expression in the nginx conf escapes too much:
https://github.com/ai-robots-txt/ai.robots.txt/blob/198653b59a888d8c580b8566d305a44e184f5b33/nginx-block-ai-bots.conf#L3
That means the `-` characters don't need to be escaped, because outside of `[]` they have no special meaning.
And even in `[]` it would be natural to place them at the front or back of the character set.
Similarly, since the string is quoted the space characters don't need to be escaped like this `\ `.
---
Also, because of using the `~*` nginx operator instead of just `~`, the expression is evaluated case-insensitively which doesn't seem necessary - or may even has unwanted side-effects.
I know that robots.txt is also supposed to be matched case-insensitively, but in practice, if a crawlers bothers to set a differentiating user agent, it likely does so without varying the case.
For example, the `Spider` entry case-insensitively matches `Baiduspider` which seems to be a normal search engine crawler.
1 条评论