Archive for the ‘Crawlability’ Category

Just a few weeks ago I attended the SEOmoz conference in Seattle.  I learned a few things about and robots files that I wanted to bring to everyone’s attention.

Robots.txt Regarding PageRank

According to Rand, do not to add pages to your robots.txt file that have generated PageRank, or else you’re blocking that page from passing any of its juice along. Essentially, any page that is added to the Robots.txt file can still be indexed.  However if that page has generated any …

So I’m sure you’ve heard of the news by now, the new Google Caffeine is a new search engine index which includes crawling, indexing and ranking changes.  Currently Google is providing webmasters and searchers to try it out and give Google feedback.  When Google Caffeine goes officially live, the new search infrastructure will replace what exists today.

According to Search Engine Land, and Google’s blog post about Caffeine, the new infrastructure may include crawling of the web more comprehensively, determining …

If you want to understand someone, you should walk a mile in their shoes. When it comes to how search engines understand your website, you should work mile in their shoes. Okay, maybe this is a stretch. What I am saying is that if you want to understand how search engine spiders view your site, you should utilize a spider simulator tool. This way you’ll get an idea of what they’re seeing.

There are numerous spider simulators out there. Each pulls …

Last week I wrote on priority levels that you can add or change within your XML sitemap in order to increase the number of pages that can get indexed by the search engines. Today, I’ll dive into frequency levels in your sitemap and how you change them.

Just to re-cap on last week’s post, setting priority levels within your XML sitemap will tell the search engines which pages you deem more important to be crawled and indexed. It’s …

A few weeks ago SEO Zombie wrote an interesting post on the flow of internal link juice.  The post itself goes into ways to prevent duplicate content issues on your blog if you’re running WordPress. However, what I found most helpful was the explanation from Matt Cutts on the differences between the NoIndex tag, NoFollow tag and the Robots.txt file. So if you’re confused at all about what the three of these do and when you need to …

What is a 404 page?

The dreaded 404 error pages are not only annoying to users and create a bad experience, but they’re also not good for the search engines and crawlability of your site either.  The 404 or Not Found error message is an HTTP response code that says the client was able to communicate with the server but the server could not find what was requested, or it was configured not to fulfill the request….

As we’ve discussed before, submitting an XML Sitemap to Google (and the other search engines) is an openly debated practice. Proponents of sitemaps tout benefits to indexation and visibility with the search engines. Those against hold a more principled stance – that your site should be optimized in such a way that submitting an XML Sitemap isn’t needed. I for one fall under the proponent category, and I’ve found another reason to back up my case: …