Reddit, a cornerstone of online discourse and culture, has made a controversial move by restricting most search engines from accessing its content. This shift effectively leaves Google as the sole search engine capable of retrieving Reddit search results.
Recent changes have rendered alternative search engines like Bing and DuckDuckGo largely incapable of crawling and displaying new content from Reddit. While some of these search engines may present outdated or incomplete results, others might not show Reddit content at all. This development follows Reddit’s recent multi-million dollar agreement with Google, which includes provisions for using Reddit data in AI training.
This decision has sparked backlash from smaller search engine providers. Colin Hayhurst, CEO of Mojeek, expressed his concerns to 404 Media, stating, “They’re [Reddit] killing everything for search but Google.” As of now, Microsoft (which owns Bing), DuckDuckGo, and other search engine companies have yet to respond.
Reddit’s move is part of a larger trend of websites restricting access to automated data scraping bots used by AI companies. The platform recently updated its robots.txt file to block all web crawling, including that by search engines—a stark contrast to the permissions granted in an archived version from July 23rd. With Google holding the most prominent search engine position, it is uniquely poised to leverage this data for training its generative AI. This situation raises ethical questions about AI training practices, with Google potentially pushing the boundaries more aggressively than its competitors.
Related topics: