Welcome to SEO Boy, the authority on search engine optimization -- how to articles, industry news, insider tips, and more! If you like what you see, you can receive free and daily updates via email or RSS.

Archive for the ‘Crawlability’ Category

Update on How To Use XML Sitemaps and Robots.txt Files

September 7th, 2009 | | Crawlability

Just a few weeks ago I attended the SEOmoz conference in Seattle.  I learned a few things about and robots files that I wanted to bring to everyone’s attention.

Robots.txt Regarding PageRank

According to Rand, do not to add pages to your robots.txt file that have generated PageRank, or else you’re blocking that page from passing any of its juice along. Essentially, any page that is added to the Robots.txt file can still be indexed.  However if that page has generated any …

Read Full Article Read Full Article  |  Comments 1 Comment  

Google Caffeine & How It Could Affect Your Rankings

August 24th, 2009 | | Crawlability

So I’m sure you’ve heard of the news by now, the new Google Caffeine is a new search engine index which includes crawling, indexing and ranking changes.  Currently Google is providing webmasters and searchers to try it out and give Google feedback.  When Google Caffeine goes officially live, the new search infrastructure will replace what exists today.

According to Search Engine Land, and Google’s blog post about Caffeine, the new infrastructure may include crawling of the web more comprehensively, determining …

Read Full Article Read Full Article  |  Comments Comments  

How Do Search Engines See Your Site? Try Using a Spider Simulator to Learn More

July 29th, 2009 | | Crawlability

If you want to understand someone, you should walk a mile in their shoes. When it comes to how search engines understand your website, you should work mile in their shoes. Okay, maybe this is a stretch. What I am saying is that if you want to understand how search engine spiders view your site, you should utilize a spider simulator tool. This way you’ll get an idea of what they’re seeing.

There are numerous spider simulators out there. Each pulls …

Read Full Article Read Full Article  |  Comments Comments  

Does Setting Priority & Frequency in Your Sitemap Help Increase Rankings? Part 2

July 6th, 2009 | | Crawlability

Last week I wrote on priority levels that you can add or change within your XML sitemap in order to increase the number of pages that can get indexed by the search engines. Today, I’ll dive into frequency levels in your sitemap and how you change them.

Just to re-cap on last week’s post, setting priority levels within your XML sitemap will tell the search engines which pages you deem more important to be crawled and indexed. It’s …

Read Full Article Read Full Article  |  Comments 6 Comments  

The Differences Between Meta NoIndex, NoFollow and Robots.txt File

April 6th, 2009 | | Crawlability

A few weeks ago SEO Zombie wrote an interesting post on the flow of internal link juice.  The post itself goes into ways to prevent duplicate content issues on your blog if you’re running WordPress. However, what I found most helpful was the explanation from Matt Cutts on the differences between the NoIndex tag, NoFollow tag and the Robots.txt file. So if you’re confused at all about what the three of these do and when you need to …

Read Full Article Read Full Article  |  Comments 15 Comments  

3 Quick Tips for Improving Your Site Architecture for SEO

March 11th, 2009 | | Crawlability, Information Architecture

SEO Architecture isn’t a trick or slick method to get more visitors.  It’s actually good for both search engines and visitors in a win-win situation that is actually surprising.

When I talk to business owners about SEO options and tips, this is the one topic that makes them uncomfortable, squirmy and withdrawn.  One day I’ll just stop them and say “Ok, what? What is it? What’s wrong? Why are you edging to the door?”  I would expect that reaction from …

Read Full Article Read Full Article  |  Comments Comments  

Ugly 404 Pages: How to Find Them on Your Site & Why They Should be Removed ASAP

March 9th, 2009 | | Crawlability

What is a 404 page?

The dreaded 404 error pages are not only annoying to users and create a bad experience, but they’re also not good for the search engines and crawlability of your site either.  The 404 or Not Found error message is an HTTP response code that says the client was able to communicate with the server but the server could not find what was requested, or it was configured not to fulfill the request….

Read Full Article Read Full Article  |  Comments Comments  

Learn How to Use the New Canonical Tag and Other META Tags to Control Your SEO Search Results

February 20th, 2009 | | Crawlability, Nuts & Bolts of Optimization

Learn more about the new Canonical Tag and refresh your memory and skills on the META NOODP (dmoz Open Directory Project) and META NOYDIR (Yahoo! Directory) tags to take control of your search results and SEO success.

Read Full Article Read Full Article  |  Comments 14 Comments  

Need Another Reason to Use XML Sitemaps? Compare Index Stats in Google Webmaster Tools

January 29th, 2009 | | Crawlability

As we’ve discussed before, submitting an XML Sitemap to Google (and the other search engines) is an openly debated practice. Proponents of sitemaps tout benefits to indexation and visibility with the search engines. Those against hold a more principled stance – that your site should be optimized in such a way that submitting an XML Sitemap isn’t needed. I for one fall under the proponent category, and I’ve found another reason to back up my case: …

Read Full Article Read Full Article  |  Comments 3 Comments  

How to Set Up and Use MSN’s Live Search: Webmaster Central

January 5th, 2009 | | Crawlability

Like that of Google Webmaster tools and Yahoo! site explorer, MSN also has a tool that can help people find and enhance their site to increase their SEO performance.  As I have in my Google Webmaster Tools post and my Yahoo! Site Explorer post, today I’ll go into detail on how to set up and use the MSN live search webmaster central (MSN WMT) tool.

The set up is identical to that of GWMT and YSE.  You must first …

Read Full Article Read Full Article  |  Comments Comments