- The Adventures of SEO Boy® - http://www.seoboy.com -

Update on How To Use XML Sitemaps and Robots.txt Files

Posted By Amber On September 7, 2009 @ 7:43 pm In Crawlability | 1 Comment

Just a few weeks ago I attended the SEOmoz conference in Seattle.  I learned a few things about and robots files that I wanted to bring to everyone’s attention.

Robots.txt Regarding PageRank

According to Rand, do not to add pages to your robots.txt file that have generated PageRank, or else you’re blocking that page from passing any of its juice along. Essentially, any page that is added to the Robots.txt file can still be indexed.  However if that page has generated any PageRank it’s like the search engines have arrived but can’t pass on any PR to any other page, so you’re technically blocking PR. What you should do instead of adding a page to the robots.txt is to add a meta no index, follow so that page can still pass on PageRank but not be indexed. If you don’t want a page to generate any PageRank and not be indexed [1] then you need to add a meta no follow, no index.

This may explain why you might see an error in your Google Webmaster tools account saying a certain file is being blocked by the robots.txt file. It’s not a good idea to tell the search engines not to crawl a page that has generated PageRank.

Now for the Sitemap.xml file, I have mentioned before in a previous post that sitemaps contain priority and frequency levels [2] that you can set to guide the search engines toward more important pages of your site. While that information is true, Rand did mention that the search engines at this time don’t pay any attention to priority levels or frequency settings in your XML sitemap.  Why those two items exist in an XML sitemap, I’m not really sure.


Article printed from The Adventures of SEO Boy®: http://www.seoboy.com

URL to article: http://www.seoboy.com/update-on-how-to-use-xml-sitemaps-and-robots-txt-files/

URLs in this post:

[1] If you don’t want a page to generate any PageRank and not be indexed: http://www.seomoz.org/blog/headsmacking-tip-13-dont-accidentally-block-link-juice-with-robotstxt

[2] sitemaps contain priority and frequency levels: http://www.seoboy.com../../../../../does-setting-priority-and-frequency-in-your-sitemap-help-increase-rankings/

[3] The Differences Between Meta NoIndex, NoFollow and Robots.txt File: http://www.seoboy.com/the-differences-between-noindex-nofollow-and-robotstxt-file/

[4] Take Control of Your SEO Destiny with a Robots.txt File: http://www.seoboy.com/take-control-of-your-seo-destiny-with-a-robotstxt-file/

[5] Unleash Your Website’s Hidden SEO Potential by Optimizing Your PDF Files: http://www.seoboy.com/unleash-your-websites-hidden-seo-potential-by-optimizing-your-pdf-files/

[6] Need Another Reason to Use XML Sitemaps? Compare Index Stats in Google Webmaster Tools: http://www.seoboy.com/need-another-reason-to-use-xml-sitemaps-compare-index-stats-in-google-webmaster-tools/

[7] How and Why You Should be Using Google Webmaster Tools – Part 2: http://www.seoboy.com/how-and-why-you-should-be-using-google-webmaster-tools-part-2/

Copyright © 2008 The Adventures of SEO Boy. All rights reserved.