Robots. Txt file play very important role whenever we speak about SEO. I came across this Robot.txt when I used adsense for first time on my WordPress blog. I got one notification saying Adsense bots are unable to crawl my blog, with red warning to edit Robot.txt. My Robots txt configuration were not allowing Google Bots to crawl my blog.
After this Small incident happened, I started searching for the solution for WordPress blogs. After lots of search on my dashboard I was unable to find any option to edit robots txt file. On surfing internet I found the solution that we can edit easily using WP Robots txt plugin in WordPress.
If your file has wrong configuration in this file can let your blog out of search engine results. By modifying this Script you can avoid some private part of your blog from getting indexed. As well as you can direct spiders to crawl the right part of your site which are useful.
You still would be wondering why would you put no index to any part of your website. When you know if you get more pages indexed and ranked resulting in more traffic. But truth is when you allow spiders to index your weak or private part of your blog, you are not gaining anything but telling them to crawl weak or useless part of your blog/website. This would not result good in any way but it would be a negative point.
What is Robots .txt file and how to edit it in WordPress?
Robots txt files are help spiders or directs them in indexing your website or avoiding some weak part of your website. Basically spiders while crawling your website they read this file for index or no index any page of your website.
How to Edit Robots. txt file by using WordPress Plugin?
WordPress is having very flexible features as I have been saying it from the beginning. In same way you can easily edit your file by using plugin because if you are not using plugin, it would be complicated.
Download WP Robots txt.
Go to Dashboard<<< Plugins<<< Add new<< Upload<<Install< Activate.
After you activate your plugin go to Settings<< Reading.
After you install this plugin and activate it will look like below image.
Copy and Paste it by changing URL of your blog.
This is one of the most SEO friendly configured file which you must use. This file directs spiders not to crawl private part of your blog which adds no value.
This file also allows Google Adsense Bots to crawl your blog and avoids any kind of error.
If you want to allow or disallow anything of your own, you can make changes according to your requirements.
If you have any doubts please ask me via comment.