Reddit will continue blocking Microsoft and other search engines and AI models from crawling its content using robots.txt – unless they strike a content licensing deal. That’s what Reddit CEO Steve Huffman said in a new interview.
Why Reddit is blocking search engines. Noting it’s been “a real pain in the ass to block these companies,” here’s what else Huffman told technology news site The Verge:
“Without these agreements, we don’t have any say or knowledge of how our data is displayed and what it’s used for, which has put us in a position now of blocking folks who haven’t been willing to come to terms with how we’d like our data to be used or not used.”
“We’ve had Microsoft, Anthropic, and Perplexity act as though all of the content on the internet is free for them to use. That’s their real position.”
“I think the traditional value exchange from search engines has changed. Search and summarization and training are merging, and the value exchange of crawling in exchange for traffic back is becoming muddied.”
Freeware. Yes, Microsoft AI CEO Mustafa Suleyman actually called web content “freeware,” saying anybody can copy and use it:
“…With respect to content that is already on the open web, the social contract of that content since the ’90s has been that it is fair use. Anyone can copy it, recreate with it, reproduce with it. That has been freeware, if you like. That’s been the understanding.”
Google not blocked. Meanwhile, Reddit didn’t block Google. That’s because Google pays Reddit $60 million a year. That content licensing deal was announced in February.
Microsoft statement. Following the news about Reddit blocking search engines, a Microsoft spokesperson told Search Engine Land:
“Microsoft respects the robots.txt standard and we honor the directions provided by websites that do not want content on their pages to be used with our generative AI models. Bing stopped crawling Reddit after they implemented their updated robots.txt file on July 1, which prohibits all crawling of their site.“
Why we care. Reddit is in a powerful position, having a licensing deal with Google – not to mention the insane amount of organic visibility and traffic it’s getting due to its prominence in Google Search results. However, other content producers and publishers likely will need whatever visibility and traffic they can get from AI search and answer engines by incorporating generative engine optimization (GEO) strategies.
