So what should I put in my Robots.txt file you might ask?  Good question and as with most things, “doctors differ, patients die” well maybe that’s a little bit extreme!

What Should I Put in My Robots.txt File robot txt crazy

Yoast’s Recommendation for Robots.txt

Yoast is an authority on SEO and you would do well to read their website. Yoast’s SEO for WordPress Plugin is also a must for any website.

The recommendation from Yoast for the robots.txt file is very simple and as follows:

User-Agent: *
Disallow:

I have been using something similar for some time now, but I add a Disallow folder for all of my affiliate links, so mine used to read like this:

User-Agent: *
Disallow: /recommends/

What is in My Robots.txt Now

So, after a lot of reading on the internet from my authorities on SEO, as usual, I have gone my own way and I am now using the following robots.txt:

User-agent: *
Disallow: /recommends/
Sitemap: https://www.brianhillier.co.uk/sitemap.xml

If you do not have affiliate links in a “/recommends/ separate folder, then all you need is:

User-agent: *
Disallow:
Sitemap: https://www.brianhillier.co.uk/sitemap.xml

Again, I use one xml sitemap and an html sitemap and you should list the one(s) you use as shown.

Why Include the Sitemap in the Robots.txt?

A lot of SEO authorities say it is not necessary to list the sitemap(s) in the robots.txt file, but in my digging around to find the best setup for SEO, I found this from sitemaps.org who as you would recognise are the absolute authority on sitemaps on the internet and they recommend the following:

You can specify the location of the Sitemap using a robots.txt file. To do this, simply add the following line including the full URL to the sitemap:
Sitemap: http://www.example.com/sitemap.xml
This directive is independent of the user-agent line, so it doesn’t matter where you place it in your file. If you have a Sitemap index file, you can include the location of just that file. You don’t need to list each individual Sitemap listed in the index file.

You can specify more than one Sitemap file per robots.txt file.
Sitemap: http://www.example.com/sitemap-host1.xml

Sitemap: http://www.example.com/sitemap-host2.xml

In my experience this strongly confirms that the sitemap(s) should be in your robots.txt file.

I hope this help to clear up any confusion there might be out there as to what should I put in my Robots.txt file and by all means give it a test on your own website to see what a difference it makes to SEO traffic.

If you have any queries on the above, just get in touch.

You might also be interested in reading about: What is Anchor Text and Do I need to Use It? and Make Keywords rank in search engines.