At least 26 of the top 100 most popular websites – and 242 of the top 1,000 – are now blocking GPTBot, the web crawler OpenAI introduced Aug. 7, according to an updated analysis.
That’s a 250% increase since last month, when just 69 of the top 1,000 websites had blocked GPTBot, according to an updated analysis from AI content and plagiarism service Originality.ai.
Why we care. To block or not to block ChatGPT? That has been a big question for many SEOs because ChatGPT does not cite or link to its sources. We have let search engines crawl our content because there is a clear potential benefit – we get traffic through direct links/citations. Clearly, even more of the most popular websites have decided to block GPTBot, presumably because they don’t want OpenAI scraping their data to help train its models – at least not without some form of compensation.
12 popular websites now blocking GPTBot. Among the new additions from the top 100 most popular sites in the past month, the majority of which publish news and information:
pinterest.com
indeed.com
theguardian.com
sciencedirect.com
usatoday.com
stackexchange.com
alamy.com
webmd.com
dictionary.com
washingtonpost.com
npr.org
cbsnews.com
One big reversal. Interestingly, Foursquare, which was blocking GPTBot last month, no longer is.
What about CCbot? Common Crawl’s web crawler is still blocked less – by just 130 websites. As a reminder, Common Crawl provides part of the training data used by OpenAI, Google and others.
109 of the top 1,000 websites block both GPTBot and CCbot.
Limitations. 67 robots.txt files out of the 1,000 websites were not identified/inspected as part of this analysis. (That’s why I wrote “at least” in the opening sentence.)
Originality.ai’s updated analysis. Websites That Have Blocked OpenAI’s GPTBot – 1000 Website Study
Dig deeper. Should you block ChatGPT’s web browser plugin from accessing your website?
The post 26% of the top 100 websites are now blocking GPTBot appeared first on Search Engine Land.
