Google posts draft to formalize Robots Exclusion Protocol Specification

Google open sources its robots.txt parser as part of this announcement.

Please visit Search Engine Land for the full article.

Source: Search Engine Land
Link: Google posts draft to formalize Robots Exclusion Protocol Specification