Cloudflare to block cynical search-and-scrape bots from ad-supported web pages

AI Summary

Cloudflare announced plans to block cynical search-and-scrape bots from ad-supported web pages to protect publishers' content and prevent disappearance from search results. The decision affects crawlers used for search and AI training data.

Some crawlers gather data for both search and AI training, so when publishers block them to protect content they risk disappearning from search results ...