So what should I put in my Robots.txt file you might ask? Good question and as with most things, “doctors differ, patients die” well maybe that’s a little bit extreme!
Yoast’s Recommendation for Robots.txt
The recommendation from Yoast for the robots.txt file is very simple and as follows:
I have been using something similar for some time now, but I add a Disallow folder for all of my affiliate links, so mine used to read like this:
What is in My Robots.txt Now
So, after a lot of reading on the internet from my authorities on SEO, as usual, I have gone my own way and I am now using the following robots.txt:
If you do not have affiliate links in a “/recommends/ separate folder, then all you need is:
Again, I use one xml sitemap and an html sitemap and you should list the one(s) you use as shown.
Why Include the Sitemap in the Robots.txt?
A lot of SEO authorities say it is not necessary to list the sitemap(s) in the robots.txt file, but in my digging around to find the best setup for SEO, I found this from sitemaps.org who as you would recognise are the absolute authority on sitemaps on the internet and they recommend the following:
You can specify the location of the Sitemap using a robots.txt file. To do this, simply add the following line including the full URL to the sitemap:
This directive is independent of the user-agent line, so it doesn’t matter where you place it in your file. If you have a Sitemap index file, you can include the location of just that file. You don’t need to list each individual Sitemap listed in the index file.
You can specify more than one Sitemap file per robots.txt file.
In my experience this strongly confirms that the sitemap(s) should be in your robots.txt file.
I hope this help to clear up any confusion there might be out there as to what should I put in my Robots.txt file and by all means give it a test on your own website to see what a difference it makes to SEO traffic.
If you have any queries on the above, just get in touch.