Publishers Consider Blocking Google Search Crawler

Publishers are contemplating blocking Google's crawler, which could affect the quality of Google Search results. Cloudflare, a content delivery network, announced that starting September 15, new websites and those on its free tier will have settings to block crawlers that scrape for both search indexing and AI training unless site owners opt otherwise. Cloudflare's chief strategy officer, Stephanie Cohen, stated that they seek a solution that allows discoverability without giving away content for free. While Google has introduced options for publishers to opt out of AI training, concerns remain about potential penalties to search visibility. USA Today Inc. is among the publishers evaluating their options, with CEO Mike Reed indicating a possible delisting from Google within the next six to twelve months. The decision to block Google’s crawler is influenced by declining search traffic, prompting publishers to seek alternative revenue sources and licensing agreements with other tech companies.