Off the original subject but here’s Google’s description of robots.txt and how Google can show your page in results.
https://developers.google.com/search/docs/crawling-indexing/robots/intro Sent from my iPhone On Aug 18, 2023, at 13:39, Sellam Abraham via cctalk <cctalk@classiccmp.org> wrote: On Fri, Aug 18, 2023 at 12:35 PM Paul Koning via cctalk < cctalk@classiccmp.org> wrote: On Aug 18, 2023, at 12:48 PM, The Doctor via cctalk < cctalk@classiccmp.org> wrote: ------- Original Message ------- On Thursday, August 17th, 2023 at 13:29, Bill Gunshannon via cctalk < cctalk@classiccmp.org> wrote: On 8/17/2023 2:14 PM, John Herron via cctalk wrote: Why would you give all your data to Google? They pretty much have it all anyway. Even if you use /robots.txt to block them, they still spider and index pages, Really? It would be interesting to have evidence supporting that, because if so, they could be subjected to pain for violating an explicit order not to do so. Not to mention that that could well be a criminal act since it would constitute unauthorized access of a computer system. If anyone has actual evidence of this happening now then please show it to me, and I'll take the necessary legal steps to bring Larry and Sergey to justice. Sellam
