On 2021-07-14 14:51, Michael Niedermayer wrote:
Signed-off-by: Michael Niedermayer <mich...@niedermayer.cc>
---
htdocs/robots.txt | 13 ++++++++++++-
1 file changed, 12 insertions(+), 1 deletion(-)
diff --git a/htdocs/robots.txt b/htdocs/robots.txt
index eb05362..4bbc395 100644
--- a/htdocs/robots.txt
+++ b/htdocs/robots.txt
@@ -1,2 +1,13 @@
User-agent: *
-Disallow:
+Crawl-delay: 10
+Disallow: /gitweb/
+Disallow: /*a=search*
+Disallow: /*/search/*
+Disallow: /*a=blobdiff*
+Disallow: /*/blobdiff/*
+Disallow: /*a=commitdiff*
+Disallow: /*/commitdiff/*
+Disallow: /*a=snapshot*
+Disallow: /*/snapshot/*
+Disallow: /*a=blame*
+Disallow: /*/blame/*
LGTM based on my own personal experiences. But the robots.txt has to be
applied for git.ffmpeg.org as well, and not just ffmpeg.org. Or else
they will just do the same for git.ffmpeg since there are treated
separately.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
