How Robot.txt,Meta Robots and rel= nofollow Effect SEO

Lots of SEO misuse the Robots.txt, Meta Robots and rel=nofollow attributes, most of mix their role in SEO with each other. All of these three attribute effect the crawling, indexing and Page Rank of website,so you should have understanding when and where we should use these attributes. Let’s take each one by one and understand them in brief way.

Robot.txt File
Robot.txt file a special file which is placed at root folder of website. It looks like It tells search engine robots which file you need to crawl in website. If you don’t want some of the pages or folder of your website crawled, you can use the robots.txt file. But there are few area where most of SEO made mistake in understanding the functionality of Robots.txt file

Q. I used robots.txt file to block a page or folder of my website. Will it come in search results?
Ans. Yes, it will come. Surprised, as i said it robots.txt file will tell search robots which page not being crawled but it does not mean they will not come in search results. These pages can come in search results if Google found the infomation about pages in high authority website pages links like in DMOZ, CNN etc. search engine will show your pages in search results.
Title and description found in SERP is based on the data search engine collects from other websites.

Q. I blocked a page of my website with Robot.txt file will it receive Page Rank
Ans. Yes, If some other pages of website are giving link to robot.txt page it will receieve the link juice from these pages.Page Rank algorithm will not consider whether page is robot.txt or not.

Meta Robots
Meta robots tags tell the search bots which pages need to crawled and which pages should get page rank. Some of the values of meta robots are index, follow, no follow, none and all. Let’s understand the Meta Robots by answering some questions

Q. I blocked page using Meta tag noindex value, will search engines crawl it?
Ans. Yes, Search engines will crawl the page, search bots extract the all links web page had, but it will not come in search engine results. After crawling the web page if search engines found the Meta robots noindex value, they will not index the page, means your page will not appear in search results.

Q. What follow” and nofollow values tell?
Ans.Follow value tell the search engines that links found on this page will flow the link juice or page rank to given link. nofollow value means no link juice or page rank will pass from given page to those links.

The rel=”nofollow” attribute is used in anchor text, it is used when we don’t want to give link juice or page rank to page mentioned in anchor text. Means rel=”nofollow” links do not pass Page Rank to the target page. Search engines crawled and index the page having rel=”nofollow” links.

Here is summary of how search engines handle robot.txt, Meta Robots and rel= nofollow attributes:

Appears in SERP
Page Rank
Yes, if it receives enough links. Yes, robots.txt Pages get PR
META-robots noindex
No, if the links of the page are followed.
META-robots nofollow Yes, if other followed links are found pointing to the target page. Yes, if the target page receives links from other webpages. Yes, all of the PageRank of the page is evaporated.
rel=”nofollow” Yes, if other followed links are found pointing to the target page. Yes, if the target page receives links from other webpages.

Article by

Dinesh Thakur is editor of Click Conversion Blog. Dinesh write about Digital Marketing, SEM , SEO and Web Analytics. You can follow him at Google+


  1. John says:

    Awesome stuff .Thank you for share very nice knowledges.You actually placed right stuff on the table.I hope more users will get the information that i receive from you.

  2. Dinesh Thakur says:

    Thanks a lot John.

  3. Ancy Antony says:

    My XML Sample file are being indexed by search engines,i used robot.txt file to block, Can anyone give me a permanent solutions for my xml flash files not been indexed from google

  4. Dinesh Thakur says:

    Hello Ancy,

    If I'm not wrong, you mean to say your XML sample files are appearing (index) in SERP, which tell your xml files are getting good authority links from some other sites (chances are less but can be). By using the robots.txt file you can't block the page in showing the SERP.

    But it will not cause any negative impact in your website SEO. Can you please specify your problem in brief.

0 Trackbacks

Leave a Reply

Your email address will not be published. Required fields are marked *