Robot.txt file a special file which is placed at root folder of website. It looks like www.yourwebsite.com/robots.txt. It tells search engine robots which file you need to crawl in website. If you don’t want some of the pages or folder of your website crawled, you can use the robots.txt file. But there are few area where most of SEO made mistake in understanding the functionality of Robots.txt file
Q. I used robots.txt file to block a page or folder of my website. Will it come in search results?
Ans. Yes, it will come. Surprised, as i said it robots.txt file will tell search robots which page not being crawled but it does not mean they will not come in search results. These pages can come in search results if Google found the infomation about pages in high authority website pages links like in DMOZ, CNN etc. search engine will show your pages in search results.
Title and description found in SERP is based on the data search engine collects from other websites.
Q. I blocked a page of my website with Robot.txt file will it receive Page Rank
Ans. Yes, If some other pages of website are giving link to robot.txt page it will receieve the link juice from these pages.Page Rank algorithm will not consider whether page is robot.txt or not.
Meta robots tags tell the search bots which pages need to crawled and which pages should get page rank. Some of the values of meta robots are index, follow, no follow, none and all. Let’s understand the Meta Robots by answering some questions
Q. I blocked page using Meta tag noindex value, will search engines crawl it?
Ans. Yes, Search engines will crawl the page, search bots extract the all links web page had, but it will not come in search engine results. After crawling the web page if search engines found the Meta robots noindex value, they will not index the page, means your page will not appear in search results.
Q. What follow” and nofollow values tell?
Ans.Follow value tell the search engines that links found on this page will flow the link juice or page rank to given link. nofollow value means no link juice or page rank will pass from given page to those links.
The rel=”nofollow” attribute is used in anchor text, it is used when we don’t want to give link juice or page rank to page mentioned in anchor text. Means rel=”nofollow” links do not pass Page Rank to the target page. Search engines crawled and index the page having rel=”nofollow” links.
Here is summary of how search engines handle robot.txt, Meta Robots and rel= nofollow attributes:
Appears in SERP
|Yes, if it receives enough links.||Yes, robots.txt Pages get PR|
|No, if the links of the page are followed.|
|META-robots nofollow||Yes, if other followed links are found pointing to the target page.||Yes, if the target page receives links from other webpages.||Yes, all of the PageRank of the page is evaporated.|
|rel=”nofollow”||Yes, if other followed links are found pointing to the target page.||Yes, if the target page receives links from other webpages.||