--- title: "The robots.txt file" description: "The other day in How do you create a WordPress website with one or more domain name aliases?, we sketched out for you just a few lines about the robots.txt file, and today we want to go a little deeper" url: https://artprojectgroup.com/the-robots-txt-file date: 2012-04-08 modified: 2023-02-11 author: "Art Project Group" image: https://artprojectgroup.com/wp-content/uploads/2023/02/programacion-web-blog.webp categories: ["Tutorials"] tags: [".htaccess", "HTML", "robots.txt"] type: post lang: en-US --- # The robots.txt file The other day in [How do you create a WordPress website with one or more domain name aliases?](/wordpress-website-with-domain-name-aliases), we sketched out for you just a few lines about the robots.txt file, and today we want to go a little deeper into this small and, in appearance, insignificant, but extremely important file. But let us start at the beginning, *what is the robots.txt file?* The answer to this question is just as simple and straightforward as the file itself: it is a text file that serves to tell, mind you, only tell, the web search engines, well, in reality their robots, web bots, bots, spiders, indexers or crawlers, whichever you prefer to call them, which parts of our website they can crawl and which they cannot; it also indicates the address or addresses of our sitemap file or files. It is a tremendously useful file both for the web search engines’ own bots and for webmasters, since thanks to it: - *We will block certain robots*: spambots and malicious robots, although to make this block effective certain content has to be added to the .htaccess file, which we will explain carefully in a later post. - *We will save the crawlers time*: since we are clearly marking for them which parts of our content they must crawl, they will avoid the rest of the content, so we will make them work much faster. - *We will save bandwidth*: by directing the spiders’ traffic correctly, they will not only spend the least possible time reading our content, but, by reading only what we want, we save our server from serving unnecessary content, and therefore we save bandwidth. - *We will remove unnecessary content from the search results*: if they do not read it, they do not index it and therefore it does not appear in the search results. - *We will prevent private parts of our website from being crawled*: the normal thing is for the spiders to respect our instructions, but even so one of them may intend to read certain sensitive or forbidden content; for that we again have to make use of our best ally, the .htaccess file, but, again, that will be another story. - *We will keep duplicate content out of the results*: if our web pages can be called in different ways, for example: my-page.htm, my-page.php, page.php?identifier=my…, that is considered duplicate content, so hard work has to be done to avoid it. At **APG** we are keener on using the .htaccess file for these jobs, but the robots.txt file can also help us in some cases. - *We will tell the robots the address of the sitemap files*: thanks to them we can give them what we consider to be the only indexable content. And how does it work? It is very simple: it uses a few rules that are extraordinarily easy to learn and that are perfectly set out and documented at [The Web Robots Pages](https://www.robotstxt.org). And mind you, it is not the only way to make all this possible, since there are also some special tags that can be used directly in the HTML code of our web pages to do the same job. These tags are the META ones, and they basically take six forms: Copy to clipboardSyntax Highlighter Whose meaning is: - The META tags on **lines 1 and 3** are synonymous, and they tell the robots to index the web page (index) and its links (follow), that is, all the content (all). - On the contrary the META tags on **lines 2 and 4,** which are also synonymous, tell the spiders exactly the opposite, not to index the web page (noindex) nor its links (nofollow), that is, no content at all (none). - The META tag on **line 5** tells the bots to index the web page (index), but not its links (nofollow). - And lastly, the META tag on **line 6** tells the web bots not to index the web page (noindex), but to index its links (follow). The big difference between using the META tag and using the robots.txt file lies in the fact that the former only affects one particular web page, so it has to be added to each and every one of the web pages of a website that we need, whereas with the robots.txt file we can do the same in one go. Using the META tag is indicated for those particular cases in which we need one specific web page to carry out a particular order that is contrary to the generality of the rest of the website and in which we do not want to get tangled up with the robots.txt file. As for the great protagonist of this post, the robots.txt file, it basically comes down to 3 statements, thanks to which we can create the orders we need: Copy to clipboardSyntax HighlighterUser-agent: Crawl-delay: Allow: Disallow: Sitemap: Let us see how they work, bearing in mind that the hash (#) marks the beginning of a comment, so all the text that follows it up to the line break is not read by the bots: - *User-agent*: indicates the robot we want to act on, although it can refer generically to all of them. They have to be given one by one, always before using *Disallow*, and a blank line has to be put between each of them. The forms it can take are: - `User-agent: ````bingbot # to point to one specific crawler, in this case Bing's` - \`User-agent: * # to refer to all the bots, without exception\`\`\` - *Crawl-delay*: tells the indexers given above the time in seconds they have to wait between each request, although not all of them support it: - Crawl-delay: 10 # forces a 10 second wait between each access - *Allow*: allows the web bots given above access to certain content in a forbidden folder, although, again, it is not supported by all of them. It must be given before the *Disallow* that forbids access: - Allow: /folder/file.ext # allows access to one specific file - *Disallow*: forbids or allows access to the spiders given above. It allows the use of the wildcards * (a sequence of at least 1 character) and $ (end of character string or end of line). As many as we need can be given for each robot. Some of the forms it can take are: - Disallow: # allows access to all the content - Disallow: / # forbids access to all the content - Disallow: /folder/ # forbids access to one specific folder - Disallow: /file.ext # forbids access to one specific file - Disallow: /*.ext$ # forbids access to any file with a given extension - Disallow: /*.ext?*$ # forbids access to any file that contains a query string - *Sitemap*: gives the complete URL of the sitemap file we want to be indexed: - Sitemap: http://www.domainname.ext/sitemap.xml A good example of a robots.txt file is the one we can find on this very website: Copy to clipboardSyntax HighlighterUser-agent: asterias Disallow:/ User-agent: BackDoorBot/1.0 Disallow:/ User-agent: Black Hole Disallow:/ User-agent: BlowFish/1.0 Disallow:/ User-agent: BotALot Disallow:/ User-agent: BuiltBotTough Disallow:/ User-agent: Bullseye/1.0 Disallow:/ User-agent: BunnySlippers Disallow:/ User-agent: Cegbfeieh Disallow:/ User-agent: CheeseBot Disallow:/ User-agent: CherryPicker Disallow:/ User-agent: CherryPickerElite/1.0 Disallow:/ User-agent: CherryPickerSE/1.0 Disallow:/ User-agent: CopyRightCheck Disallow:/ User-agent: cosmos Disallow:/ User-agent: Crescent Disallow:/ User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow:/ User-agent: DittoSpyder Disallow:/ User-agent: EmailCollector Disallow:/ User-agent: EmailSiphon Disallow:/ User-agent: EmailWolf Disallow:/ User-agent: EroCrawler Disallow:/ User-agent: ExtractorPro Disallow:/ User-agent: Foobot Disallow:/ User-agent: Harvest/1.5 Disallow:/ User-agent: hloader Disallow:/ User-agent: httplib Disallow:/ User-agent: humanlinks Disallow:/ User-agent: InfoNaviRobot Disallow:/ User-agent: JennyBot Disallow:/ User-agent: Kenjin Spider Disallow:/ User-agent: Keyword Density/0.9 Disallow:/ User-agent: LexiBot Disallow:/ User-agent: libWeb/clsHTTP Disallow:/ User-agent: LinkextractorPro Disallow:/ User-agent: LinkScan/8.1a Unix Disallow:/ User-agent: LinkWalker Disallow:/ User-agent: LNSpiderguy Disallow:/ User-agent: lwp-trivial Disallow:/ User-agent: lwp-trivial/1.34 Disallow:/ User-agent: Mata Hari Disallow:/ User-agent: Microsoft URL Control - 5.01.4511 Disallow:/ User-agent: Microsoft URL Control - 6.00.8169 Disallow:/ User-agent: MIIxpc Disallow:/ User-agent: MIIxpc/4.2 Disallow:/ User-agent: Mister PiX Disallow:/ User-agent: moget Disallow:/ User-agent: moget/2.1 Disallow:/ User-agent: mozilla/4 Disallow:/ User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95) Disallow:/ User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95) Disallow:/ User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98) Disallow:/ User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT) Disallow:/ User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP) Disallow:/ User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000) Disallow:/ User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows ME) Disallow:/ User-agent: mozilla/5 Disallow:/ User-agent: NetAnts Disallow:/ User-agent: NICErsPRO Disallow:/ User-agent: Offline Explorer Disallow:/ User-agent: Openfind Disallow:/ User-agent: Openfind data gathere Disallow:/ User-agent: ProPowerBot/2.14 Disallow:/ User-agent: ProWebWalker Disallow:/ User-agent: QueryN Metasearch Disallow:/ User-agent: RepoMonkey Disallow:/ User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow:/ User-agent: RMA Disallow:/ User-agent: SiteSnagger Disallow:/ User-agent: SpankBot Disallow:/ User-agent: spanner Disallow:/ User-agent: suzuran Disallow:/ User-agent: Szukacz/1.4 Disallow:/ User-agent: Teleport Disallow:/ User-agent: TeleportPro Disallow:/ User-agent: Telesoft Disallow:/ User-agent: The Intraformant Disallow:/ User-agent: TheNomad Disallow:/ User-agent: TightTwatBot Disallow:/ User-agent: Titan Disallow:/ User-agent: toCrawl/UrlDispatcher Disallow:/ User-agent: True_Robot Disallow:/ User-agent: True_Robot/1.0 Disallow:/ User-agent: turingos Disallow:/ User-agent: URLy Warning Disallow:/ User-agent: VCI Disallow:/ User-agent: VCI WebViewer VCI WebViewer Win32 Disallow:/ User-agent: Web Image Collector Disallow:/ User-agent: WebAuto Disallow:/ User-agent: WebBandit Disallow:/ User-agent: WebBandit/3.50 Disallow:/ User-agent: WebCopier Disallow:/ User-agent: WebEnhancer Disallow:/ User-agent: WebmasterWorldForumBot Disallow:/ User-agent: WebSauger Disallow:/ User-agent: Website Quester Disallow:/ User-agent: Webster Pro Disallow:/ User-agent: WebStripper Disallow:/ User-agent: WebZip Disallow:/ User-agent: WebZip/4.0 Disallow:/ User-agent: Wget Disallow:/ User-agent: Wget/1.5.3 Disallow:/ User-agent: Wget/1.6 Disallow:/ User-agent: WWW-Collector-E Disallow:/ User-agent: Xenu's Disallow:/ User-agent: Xenu's Link Sleuth 1.1c Disallow:/ User-agent: Zeus Disallow:/ User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow:/ User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Sitemap: https://temporal.artprojectgroup.es/sitemap_index.xml Sitemap: https://temporal.artprojectgroup.es/sitemap-image.xml Sitemap: https://temporal.artprojectgroup.es/sitemap-mobile.xml Sitemap: https://temporal.artprojectgroup.es/sitemap-video.xml Sitemap: https://temporal.artprojectgroup.es/sitemap-news.xml Although you can also take a look at [Google](http://www.google.com/robots.txt)‘s robots.txt file, where you will see what level of complexity can be reached in this fantastic file.