Edit Robots.txt For Optimizing, Sitemap, Allow & Disallow All
Home » Edit Robots.txt For Optimizing, Sitemap, Allow & Disallow All
Blogging SEO

Edit Robots.txt For Optimizing, Sitemap, Allow & Disallow All

Robots. Txt file play very important role whenever we speak about SEO. I came across this Robot.txt when I used adsense for first time on my WordPress blog. I got one notification saying Adsense bots are unable to crawl my blog, with red warning to edit Robot.txt. My Robots txt configuration were not allowing Google Bots to crawl my blog.

After this Small incident happened, I started searching for the solution for WordPress blogs. After lots of search on my dashboard I was unable to find any option to edit robots txt file. On surfing internet I found the solution that we can edit easily using WP Robots txt plugin in WordPress.

If your file has wrong configuration in this file can let your blog out of search engine results. By modifying this Script you can avoid some private part of your blog from getting indexed. As well as you can direct spiders to crawl the right part of your site which are useful.

Wordpress-Robots.txt-file

 

You still would be wondering why would you put no index to any part of your website. When you know if you get more pages indexed and ranked resulting in more traffic. But truth is when you allow spiders to index your weak or private part of your blog, you are not gaining anything but telling them to crawl weak or useless part of your blog/website. This would not result good in any way but it would be a negative point.

 What is Robots .txt file and how to edit it in WordPress?

Robots txt files are help spiders or directs them in indexing your website or avoiding some weak part of your website. Basically spiders while crawling your website they read this file for index or no  index any page of your website.

How to Edit Robots. txt file by using WordPress Plugin?

WordPress is having very flexible features as I have been saying it from the beginning. In same way you can easily edit your file by using plugin because if you are not using plugin, it would be complicated.

Download WP Robots txt.

Go to Dashboard<<< Plugins<<< Add new<< Upload<<Install< Activate.

After you activate your plugin go to Settings<< Reading.

After you install this plugin and activate it will look like below image.

Robots.txt-file-allow-disallow

 

Copy and Paste it by changing URL of your blog.

Sitemap: https://www.chrometechny.com/sitemap.xml
sitemap: https://www.chrometechny.com/sitemap-image.xml

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /recommended/
Disallow: /comments/feed/
Disallow: /trackback/
Disallow: /index.php
Disallow: /xmlrpc.php
Disallow: /wp-content/plugins/

User-agent: NinjaBot
Allow: /

User-agent: Mediapartners-Google*
Allow: /

User-agent: Googlebot-Image
Allow: /wp-content/uploads/

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Mobile
Allow: /

This is one of the most SEO friendly configured file which you must use. This file directs spiders not to crawl private part of your blog which adds no value.

This file also allows Google Adsense Bots to crawl your blog and avoids any kind of error.

Try: Complete guide to optimize your blog using WP super cache.

If you want to allow or disallow anything of your own, you can make changes according to your requirements.

If you have any doubts please ask me via comment.

Best Regards.

Paul

Tags

About the author

Paul billygraham Reang

Hello Friends, I am Paul billygraham Reang CEO and Founder of ChromeTechny Networks Pvt. Ltd. company. I started blogging since 7th June, 2013 and took blogging as my full time career.
Apart from blogging I love visiting new places, listening musics, reading motivational books and talking to those peoples who wants to do something big in their life.