Hi.
I thought to contribute to the https://github.com/ai-robots-txt/ai.robots.txt
repo and would like to here your opinion to the Question.
Should the AI Crawler be
[ ] tarpitted or get directly a
[ ] deny?
Both lines are added in the example config to see that the config is almost the
same on HAProxy site.
https://github.com/git001/ai.robots.txt/blob/7169417be76d8f6e8ca69593f626ca24814cf3a2/haproxy-ai-crawler-block.config#L36-L37
I know that HAProxy Enterprise have a great bot-management Solution
https://www.haproxy.com/solutions/bot-management
which is mentioned in these Blog Posts
https://www.haproxy.com/blog/how-to-reliably-block-ai-crawlers-using-haproxy-enterprise
https://www.haproxy.com/blog/nearly-90-of-our-ai-crawler-traffic-is-from-tiktok-parent-bytedance-lessons-learned
but maybe the ai.robots.txt can be a poor person solution wich can also be used
for the Haproxy Ingress Controller?
Any Answer is highly appreciated.
Regards
Alex
