robots.txt
The Robots Exclusion Protocol (or Standard) is a text file, accessible at the root of a website, that tells search engine crawlers which areas of a website should be ignored.
The Robots Exclusion Protocol (or Standard) is a text file, accessible at the root of a website, that tells search engine crawlers which areas of a website should be ignored.