In this WordPress tutorial, we will learn in detail about how to create and optimize WordPress robots.txt for SEO. It is very important for your website.
Robots.txt is really a very powerful and important SEO tool that every serious website owner or blogger should have. It plays a very significant role in your site’s SEO but if only configured correctly.
But having and making this file is not enough for enjoying a complete benefit of this amazing and powerful tool but you have to configure and optimize this in order to get full reward by Google search crawlers.
Don’t worry in this detailed article we will teach you how to create an awesome robots.txt file for SEO.
What Is The Robots.txt File?
Robots.txt is a simple yet powerful text file that contains a piece of code that guides the bots or crawlers of search engines on how to crawl or index posts or pages on a website.
Every time search engine bots visit a website they have a limited memory that is reserve to crawl a website at a time.
If they crawl all the important as well as not important things they can run out of given memory resources and stop crawling. This will decreases the speed of indexing your site in search engines.
As the search engine crawlers come to your site it locates the robots.txt file follows the instruction of how-to and what to crawl and indexed.
Remember! Non Availability of robots.txt file does not stop search engine bots from crawling and indexing your website but it does in an undefined way.
Like every file in a computer stored in some folder, the robots.txt file is typically in the root directory of your website. Typically it looks like this.
User-agent: [User agent name as google or bing ]
Disallow: [URL link which we don't want to crawl or index]
User-agent: [User agent name as google or bing ]
Allow: [URL link which we want to crawl or index]
Sitemap: Here comes the URL of your website sitemap
Other than the above file we can have multiple allow or disallow statements and more than one sitemap statement as shown below.
In the above robots.txt file the crawlers are instructed to crawl and indexed all the files in the upload folder from the website directory and meanwhile, we have given instruction not to crawl the plugin files and WordPress admin folder.
Why Robots.txt Is So Much Important?
If you are really concern about the higher search ranking and good SEO then keep in mind that this file is a must-have file for you. Now we will see why it is so much important?
As we discussed earlier that all search engine bots have some resources quota to crawl website but in the absence of robots.txt file we have no control over the bots to restrict them from crawling unnecessary files or folder that will eventually cut down or crawling quota.
But generally, if you have a small one-page blog then this will not hinder your crawling rate because you don’t have a lot of content to crawl.
If you have a large website with lots of different types of posts and pages then it is a necessary step to have better control over your website behavior and crawled and indexed and this is where the robots.txt file comes to rescue.
You can disallow unnecessary pages from crawling and this also saves crawl quota and as a result search engines, bots or spiders can crawl even more posts or pages on your website and ultimately indexed them as quickly as possible.
How To Create And Optimize Robots.txt?
Luckily Robots.txt automatically created with the WordPress and lies in the root directory of your website but it is a raw not optimized fully. So we have to edit it in order to enjoy all the benefits.
You can two options to edit this file either by using FTP(File Transfer Protocol) Manager or using cPanel File Manager. But these both methods are time-consuming and a bit difficult.
So we take an easy approach to this.
If you are using Yoast SEO plugin then very well and if not then I highly recommend you to install that plugin because it gives to the ability to modify your robots.txt file from your WordPress admin area.
Now I assume that you have installed and activate Yoast SEO plugin. Follow the steps below.
Now Once you’ve installed and activated Yoast SEO plugin, your first step to go to the WordPress Admin Panel > SEO > Tools.
Now click on “File Editor”
Click on “Create robots.txt file”.
After you click this button a window will open. In this, you can actually edit and optimize your code as per your desire.
Above robots.txt file is generated by WordPress automatically but it does not fulfill our requirements. So we need to edit and optimize this according to our requirements.
But before dive into the optimization process what a good practice to know about the terms or commands use in creating this file.
So let’s learn about these commands first so we can understand clearly what we are doing.
User-Agent: By using this command we can allow or restrict any search engine bot like Googlebot, Bingbot, Baidubot from crawling and indexing our website. But generally, we use an asterisk (*) to allow all search engine bots to come and crawl our website.
Disallow: This command instructs the search engine bots from not crawling and indexing the specified part of your website.
Allow: It instructs the search engine spiders to crawl and index parts of your website that you specified.
Now, these are some important commands that we use in creating and optimizing WordPress robots.txt file.
Create And Optimize WordPress Robots.txt File For SEO
First, you should delete all the code available in the auto-generated robots.txt file which looks something like this.
Now you should copy and paste the code we have provided below for how an ideal robots.txt file should look like. But if you want to do some changes according to your desire you are welcome to do so.
Let’s see how to create a WordPress robots.txt file for SEO
- First of all, you need to add User-Agent commands with an asterisk (*) to allow all the search engine bots to crawl and index your website.
- Now you should determine which part of your website you don’t want index by search engine bots. According to our recommendation, you should restrict /wp-admin/, /wp-content/plugins/, /readme.html, /referer.
- In this part, we instruct the search engine bots what to crawl and index on your website. You should add /wp-content/uploads. In case you don’t want to add something bots will crawl your website anyway but it is good practice to specify it.
- Now, in the end, you add your website sitemap at the end of this file.
Our Ideal WordPress Robots.txt File For SEO Recommendation
Below is our ideal WordPress robots.txt file recommendation for you to use for your website.
You can use this robots.txt file as it is or can make changes according to your needs.
There are some misconfigurations about robots.txt file that you should have in your mind and try to avoid them if you want to get higher SEO ranking.
- It is not a good practice to stop search engine bots from crawling and indexing your website Categories, Tags, Archive pages, etc. If you want to do so then you should add nofollow and noindex meta tags to these pages.
- Do not allow or use your Robots.txt file to manage your duplicate content. You can use other ways for it.
- Good SEO practice is that you should not hide low-quality content in the robots.txt file.
How To Test Your WordPress Robots.txt?
Once you have created and optimize your robots.txt file completely now it’s time to test it for any possible errors. It is always a good practice to check your code after you have made it.
There are other ways to test your robots.txt file but the best and easiest way is by using the Google Official Tester Tool. Once you are one URL click on Open robots.txt Tester. Should be logged in to your Gmail account for this.
Once you are on the specified link a window of Google testing tool opens.
Now you do as mentioned in the image above. After you submit your code the following screen will appear.
Now Click on the “Submit” button for submitting your robots.txt file. Now you should wait for few minutes for the changes to take effect. To check your robots.txt file you should type the following link in the search bar of your browser.
It will show your submitted robots.txt code. Don’t forget to replace “example.com” with your website name.
I am pretty much sure that this detail WordPress Tutorial helps you a lot in creating and optimizing your best robots.txt file for SEO. This will also help search bots in crawling and indexing your website faster.