What Should I Put in My Robots.txt File?
So what should I put in my Robots.txt file you might ask?
Good question and as with most things, “doctors differ, patients die” well maybe that’s a little bit extreme!
Read on a bit further to see your options and find the best code to suit your on website.
Optimizing WordPress Settings for Enhanced Performance and SEO
Yoast’s Recommendation for Robots.txt
Yoast is an authority on SEO and you would do well to read their website.
Yoast’s SEO for WordPress Plugin is also a must for any website.
Recommended Reading
The recommendation from Yoast for the robots.txt file is very simple and as follows:
User-Agent: *
Disallow:
I have been using something similar for some time now, but I add a Disallow folder for all of my affiliate links, so mine used to read like this:
User-Agent: *
Disallow: /recommends/
What is in My Robots.txt Now
So, after a lot of reading on the internet from my authorities on SEO, as usual, I have gone my own way and I am now using the following robots.txt:
User-agent: *
Disallow: /recommends/
Sitemap: https://www.brianhillier.co.uk/sitemap.xml
If you do not have affiliate links in a “/recommends/ separate folder, then all you need is:
User-agent: *
Disallow:
Sitemap: https://www.brianhillier.co.uk/sitemap.xml
Again, I use one xml sitemap and an html sitemap and you should list the one(s) you use as shown.
Why Include the Sitemap in the Robots.txt?
A lot of SEO authorities say it is not necessary to list the sitemap(s) in the robots.txt file, but in my digging around to find the best setup for SEO, I found this from sitemaps.org who as you would recognise are the absolute authority on sitemaps on the internet and they recommend the following:
You can specify the location of the Sitemap using a robots.txt file.
To do this, simply add the following line including the full URL to the sitemap:
Sitemap: http://www.example.com/sitemap.xmlThis directive is independent of the user-agent line, so it doesn’t matter where you place it in your file.
If you have a Sitemap index file, you can include the location of just that file.
You don’t need to list each individual Sitemap listed in the index file.
You can specify more than one Sitemap file per robots.txt file.
Sitemap: http://www.example.com/sitemap-host1.xmlSitemap: http://www.example.com/sitemap-host2.xml
In my experience this strongly confirms that the sitemap(s) should be in your robots.txt file.
What Should I Put in My Robots.txt File? Hopefully It’s A Bit Clearer
I hope this help to clear up any confusion there might be out there as to what should I put in my Robots.txt file and by all means give it a test on your own website to see what a difference it makes to SEO traffic.
If you have any queries on the above, just get in touch.
