And as January 2010 rolled around, a funny thing happened:  we got Caffeine-deprived. The launch date was pushed back. But it gave us a time to plan for what was to come.

Now it’s June and Google Caffeine is out…

And so far, it was much worth the wait.

What is Google Caffeine?

Google Caffeine is the new way the search engine indexes web traffic. To paraphrase Matt Cutts at 2010 SMX Advanced: the old index was like waiting at the bus station for the next bus; Caffeine is like getting off the plane and having a limo waiting for you at the door.

Here’s the video from SMX Advanced.

How Does Google Caffeine Work?

To understand how Caffeine works, it’s best to remember how the old(er) system worked…

Google would send out its bots to take text copies of batches of the website. The content would then be processed on the servers and the changes would be pushed live in one shot. Now, Googlebots indexes a page, processes it, and ranks it nearly instantly. And it doesn’t stop at websites. Social Media sites and video are all fair game. If it’s publicly accessible, it’s liable to get indexed before you know it.

Initial Tests Show…

I’ve seen a couple sites pushed live this week. The site had no promotion, no backlinks, no url history, and we just put it in Google Webmaster before we checked and found the site already indexed.

I also did one of those things where I published a blog post before I edited it, then realized I forgot Caffeine was live, went to check to see (hope it wasn’t) indexed, and found it was already indexed.

I also wrote another post and set a search to see how fast it would indexed. The result:  it literally took seven seconds to have the post indexed and ranked in Google.

What this means.

1. Make sure you edit your content first.
2. If you can do it this quickly, so can your competitors. – Keep on top of your keyword rankings.

If I had to guess…

I’d say Google reestablished its relevance with Caffeine. Like Cutts says in the above video, Google’s showing ALL content – real time and relevant, in a blend that keeps Google authentic, established, and trustworthy.

Have you played with your site’s SEO since Caffeine went live?

What do you think?

And there were a lot of issues to sort out. Duplicate content, 100s of links indexing the same content (in the days before canonicals), stuff that wouldn’t index at all…

And let’s not mention the lack of on-site SEO.

In the end, I was lucky to have worked at a place that dealt with commerce sites that were all in https because I had a baptismal by fire.

Anyway, we ran the textbook process for site submission. Many of the commerce sites had but few pages, but XML sitemaps were a great way to get the search engines to find the site right away.

But after Google indexes your site, do you still need the XML Sitemap?

It’s a great question.

And here’s the thing:  when you’re starting out, sitemaps are great. Submit your site, verify them in search engines, add an XML sitemap, run some promotion and a few links, and your site is up and running.

But whether or not you need it again is another story. I used to run sites where I’d take off the XML sitemap just to see what it would do. I’d pick sites with similar traffic, similar competition, and similar promotion. My results were negligible.

Why?

Because if you’ve built your site with and real forethought regarding SEO. If there’s internal linking, bread crumbs, canonicals, proper naming, decent traffic, and – am oldie but a goodie – a sitemap, then Google’s not going to forget about the site. Google monitors traffic through the search engines.

Google’s large and has an elephant’s memory.

Even with blog sites. Because typically with blogs, the fresh content comes from the index page, and if the site’s verified in search engines, Google will find it. Yes – or from .com/blog or blog.url.com…the point is that the fresh content is coming from the same spot. Google’s learning to drink from that watering whole. And it’ll crawl the tags. Take it from someone who took the XML sitemap off blogs just to check. The fresh content was still found rather quickly – WordPress is voodoo like that.

Where XML Sitemaps are most important

Where XML Sitemaps are important throughout the life of the site is in shopping cart sites and sites where the content could be added anywhere. With shopping cart sites, products could go in under the manufacturer category, the product category, the specials, the features, any one of a few places – pending on how the site is laid out.

And at that point, you should be using a dynamic XML sitemap. If you’re using static XML Sitemaps,you’re wasting time because you’ll have to make a new one every time you add products. And metaphysics forgive the days you forget to recreate them.

But that’s my experience on XML sitemaps.

If you have another, I’d be glad to hear it.

After a bit (okay, a lot) of research, this is what I found for preserving SEO during a domain redirect and I have successfully implemented this for my client. So far, so good. One-week in, the new domain has been indexed in Google Webmaster Tools and we are appearing for highest priority main keywords in the Top 10 for SERPs.

1. Register Both Domains in Google Webmaster Tools. Registering your domain under Google Webmaster Tools (GWT) is the most efficient way to get your website crawled, indexed and cached.  When transferring your website to a new domain, it is important to register both domain within this service. If you don’t already have your old domain setup in GWT, your first step is to add the old domain, verify the site, and submit the current xml sitemap. Next, add your new domain under the same login and follow the same steps to verify the website.
2. Outline Pages on Both Domains. Create a comprehensive list of the URLs on the old domain and the new domain. Determine what pages you plan to keep on the new domain, and assign them a URL. For the most part, every page on the old domain should have a counterpart on the new domain. If a page is not going to be carried over, designate that page as a redirect to a custom 404 error page (see #5). Creating this list will no only get your content organized for the domain transition, but also get your URLs ready for 301 redirects (see #7).
3. Submit an XML Sitemap for the new domain. To get your website crawled and indexed faster, it is important to submit an xml sitemap for your new domain. This will provide Google with a direct list of the new pages on our website, which in the end will increase the ease and time required to get your new website crawled and listed in the index. Once you have mapped the URLs for your new domain and site structure, create an xml sitemap and upload it to your new domain at www.newdomain.com/sitemap.xml. Submit this sitemap to GWT under your new domain profile.
   
4. Create an HTML Sitemap for the new domain. Along the same vein as the xml sitemap, you also need to add an html sitemap for your new domain. This should be located at www.newdomain.com/sitemap.html. Again, including an HTML sitemap is another method to increase the crawling ease and efficiency for search engine bots on your website.
5. Create Custom 404 Error Pages. A custom 404 error message is very important because when a URL error occurs (which in most cases is inevitable), it redirects visitors to a page on your domain versus a “Page Not Found” error message. When you transfer a new domain, chances are you will have URL errors. Creating a custom 404 error page is required to preserve user experience. The page will keep your visitors on your website, let them know that they are in the right place and give them options to continue browsing your website.
6. Build Links to the New Domain. Yes, 301 redirects do pass on PageRank, but some juice is lost. Studies have shown that between 1%-15% of a web page’s PageRank is lost in a 301 redirect. So, chances are that a domain-level 301 redirect will hurt your PageRank, and in turn, SERPs rankings.  Once your website is launched, start a strong link building initiative to attract fresh links to your domain. If you have strong links to your old domain, reach out to the linked site’s webmaster (if appropriate) and request a URL change.
7. Implement Permanent 301 Redirects. Once your have fully tested your new domain in a dev environment, then you should be safe to launch. At this point, implement permanent 301 redirects from your old domain to the new domain.  Old website pages should be redirected on a 1-to-1 basis— every page from www.olddomain.com should not redirect to the homepage of www.newdomain.com. Use the website URL outline created in Step 2 to determine the most relevant content to redirect old web pages to. If a relevant page does not exist on the new website, then redirect that page to a custom 404 error page.
8. Submit a “Change of Address” Request in GWT. “Officially” changing the address within GWT is just another way to signal to search engines that your domain changed. Implement the “Change of Address” after your new domain has been launched. The “Change of Address” section can be found within Google Webmaster Tools under the “Site Configuration” section. If you have added and verified your new domain within the same GWT login, you should see your new domain listed under “Tell us the URL of your new domain.” Select the URL and click Submit.
9. Monitor Errors Pages in GWT. It can take anywhere between 1 day and one month for Google to visit, crawl, and index your new website. Check GWT everyday to see if any pages on your old or new domain are showing a 404 error. Fix any errors as soon as possible.
10. List Your New Domain in Web Directories. Don’t forget about directory listings! You probably have your old domain listed in all of the important directories. Take the time to resubmit (or update current listing) the new domain to both paid and free directories.

Did I miss something? Please share additional tactics or strategies you have successfully implemented when redirecting a domain.

Here’s Web Pro New’s interview with Matt Cutts regarding the topic.

SSO Fears

Google already uses site speed as determining factor in AdWords so the news should not have come as a big surprise. Yet when news broke of the potential algorithm tweak, not only did it upstage Google’s Caffeine buzz, it also stirred up some blogosphere controversy.

Some people believe that SSO – Site Speed Optimization – will diminish the need for quality content.

Others fear that site speed will hurt small companies who can’t afford the same quality of webhosting as medium to large companies.

SEO/SSO Common Sense

I typically jump on the small company bandwagon. I even share the concerns of those who fear the impact of site speed on SERPs. But in this case I give Google the benefit of the doubt:

1) Google understands that User Experience is the most important factor in SEO. Quick, accurate load times enhance the user experience.

2) Site Speed could become “a” factor, not “the” factor in SERPs. Google’s secret indexing recipe considers many, many factors so if your sites don’t load lightning-quick today, it probably won’t be the end of their good rankings.

3) The Twitter effect. Google’s stranglehold on the Search Engine market stems from their ability to give accurate, relevant listings. With the rise of Twitter Search, users not only find relevant results, but real time results from their peers. Because of this, Google is more than likely trying to give users both relevant and real time results.

Your Neighbor Google Likes to Share Tools

Cutts mentioned how Google wants a “faster web” (source, webpronews video). A fast web comes from not only faster webhosting but from cleaner code. With this in mind, Cutts mentioned a few of the many tools webmasters and site owners can use to check their site speed.

Google Speed
A link from Google with tools, tips and strategies to help make your site(s) quicker

Google Closure Compiler and Page Speed Firefox Plugin
Page Speed helps give you analytics on how your page loads. Closure Compiler helps minify your javascript code.

WebPageTest.org
A third-party site that gives nice, second-by-second waterfall date regarding site page loading.

Google probably doesn’t have a hidden agenda. They have to catch up to a 2.0 world. And the rest of us are going to have to catch up along with them.

The answer to the million dollar question is yes. Technically, the search engines can crawl dynamic URLs and content that is dynamically generated. However, they rarely do. Did you think it was going to be that easy?!!

In a post from the Official Google Webmaster Central Blog back in 2008, they claim that yes, search engines can and will crawl dynamic URL’s:

Myth: “Dynamic URLs cannot be crawled.”
Fact: We can crawl dynamic URLs and interpret the different parameters. We might have problems crawling and ranking your dynamic URLs if you try to make your urls look static and in the process hide parameters which offer the Googlebot valuable information. One recommendation is to avoid reformatting a dynamic URL to make it look static. It’s always advisable to use static content with static URLs as much as possible, but in cases where you decide to use dynamic content, you should give us the possibility to analyze your URL structure and not remove information by hiding parameters and making them look static.

I definitely agree with the thought behind using static pages with static URL’s as much as possible if it’s important to get those pages crawled and indexed.  Another recommendation is if you do have dynamic URL’s, if possible include those dynamic URL’s, or as many as you can in your XML and HTML sitemaps. This way the search engines won’t have to stumble upon them, they can simply find them via your sitemap.

Another response I received from SEOmoz was this:

Unless the search engines have HTML links through which to get to a page, it is quite unlikely that they will spider that page. They have been known to execute some form functions in the past, but rarely do it, as it usually results in them creates loads of duplicate or garbage content for themselves. If you need search engines to find those pages, create a static version of the pages (or of as many of them as is sensible and does not result in duplicate content) and link to these pages without the necessity of a form.

The point I think is people should try not to ‘hide’ important information from the search engines. You may not be trying to hide your content that is on these dynamic URL’s but to the search engines they are still harder to find and more difficult to crawl. Point being, if the page is that important make it a static page with a static URL in order for the search engines to find that data more easily.

However, one thing I heard at the recent SEOMoz conference in Seattle was that once this news broke, people were going back and removing all their nofollows on their homepage, and were actually seeing their rankings and traffic drop as a result.

Recently, I actually removed miscellaneous links from my homepage altogether instead of trying to nofollow them. These miscellaneous links lead to content that should be placed on other pages over my homepage. Instead, I focused only my top keywords on my homepage and tried to link to only those pages that contained those top keywords. This proved to help fairly well, seeing an increase in traffic by 10.25% after the change was made.

In theory, you should link your homepage only to the most important pages of your site. If you want to add content to your site to help your SEO efforts, don’t add links to this content from your homepage, find a more appropriate place for that content on other pages of your site or even in your blog. Choose 3 – 5 top keywords you want your homepage to rank for and include those keywords and links in your homepage.

Another thing you can do is if you have a privacy policy, terms and conditions page and an About Us page that all link from your homepage, combine the privacy policy and terms and conditions page into your About Us page. This will allow two less links on your homepage and more PageRank to your other links.

Keep in mind that each page of your site should only focus on 3 – 5 core keywords you wish that page to rank for. If one page is bombarded with anchor text and links pointing away from that page it could prevent you from ranking as well as you should be.

So within a week and a half we are going to start seeing documents within search results.  My first thought was “oh great, a whole lot of content clogging my results.”  It could happen.  One way to tone down the clutter is to have a tab of “documents” to go along side Web, Images, Video, etc.

But here are some ramifications of letting documents becoming crawl-able.

1. Faster Published Content

People have developed content management systems to try to get as many people online as possible.  “A website can be made in 3 easy steps!” they tout, but they don’t.  No matter how easy you make a website, some computer users won’t bother, but everyone seems to know how to write documents.  Copious amounts of documents containing ideas and notes and conceptual pieces can be flung into the Internet between the websites and blogosphere.

2. Formatting for optimal results

Documents will beat out other documents for rankings and SEO people, like myself, will analyze what is making one pop over another.  “There won’t be linking inside the document will there be? Are there headings involved? How important are the keywords? Interesting!”  Once trends of indexing behavior become theorized, expect a wave of documents exploiting those trends.

3. Better Content

If documents are free from Adsense Opportunities, Adwords, banners and paid word thingys, there will be less interested for those who are on the web for quick results that bring in ad revenue money and more for information.  It is possible that someone might actually have something to say that’s not an ad-laden article or brief blog post.

4. Less duplicate content

Ever look for information and wind up going to several websites that held the same article? Well, here is a venue where it will be a LOT harder to come across duplicate content.  Since documents are tied to public websites, it’s more than a submit button of copied and pasted content.

5. The Rise of Thought Leadership

Though blogs were to bring thought leaders to the front, blogs are more complicated than just posting your latest observations – there’s linking, imaging, publicizing, widgets and other things that take a LOT of time to pull together and straighten out.  If a site has the most insightful documents, then they may be declared a thought leader, pure and simple.

Watch and See

It’s hard to guess what actually may happen when this starts, but there’s one thing we do know and that’s you always need to pay attention! Simple things like these can change a world’s searching habits.

What do you think about this new turn?

I use Google Webmaster tools (GWMT) almost on a daily basis. Each time I login and click around I find something new about my website that I didn’t find before.  Most people use GWMT to find crawler errors, files restricted by the robots.txt file, 404 pages, etc.  And all of that is great and very noteworthy because you need to have the least possible crawler errors in order to get higher rankings.

1. Crawl Stats: Something that helped me more recently, was when I clicked on ‘crawl stats’, and scrolled al the way to the bottom of the page to find my page with the highest PageRank for the past three months. To my surprise it wasn’t my homepage!  It was another page that has been around a little bit longer than my homepage and got transferred to another section of my website.  Since this odd page had the highest PageRank in the past three months, I realized there was room to add in some anchor text and point to other important pages of my site from this page.  This inevitably will help increase PageRank to those other important pages.

2. HTML Suggestions: Another stat I like to look at in GWMT is HTML suggestions. Here it will tell you information related to your meta descriptions and page titles, whether there are duplicates of either, whether they’re too short or too long or even ‘non-informative’ or not. Now, to be honest I’m not sure how much duplicate meta descriptions or page titles can hurt your SEO, but it’s worth fixing if Google is calling you out on it right??

3. Mobile Search Reports: If you do a lot with Mobile ads and marketing, you can click on the ‘your site on the web’ link and go to the top search queries report. Towards the middle of the page you can select the dropdown to show searches coming from the web, mobile, images and all searches.  It’s really interesting to see raw search data on people coming from their mobile browsers.  If there is one keyword in particular that is driving a majority of your mobile traffic, then perhaps add content around that keyword and optimize it for mobile users.

Point being, don’t forget about the free tools you have from the actual search engines like Google Webmaster tools, Yahoo Site Explorer and MSN webmaster tools. They can identify all different kinds of problems with your site which can prevent pages from being crawled and indexed properly.

Robots.txt Regarding PageRank

According to Rand, do not to add pages to your robots.txt file that have generated PageRank, or else you’re blocking that page from passing any of its juice along. Essentially, any page that is added to the Robots.txt file can still be indexed.  However if that page has generated any PageRank it’s like the search engines have arrived but can’t pass on any PR to any other page, so you’re technically blocking PR. What you should do instead of adding a page to the robots.txt is to add a meta no index, follow so that page can still pass on PageRank but not be indexed. If you don’t want a page to generate any PageRank and not be indexed then you need to add a meta no follow, no index.

This may explain why you might see an error in your Google Webmaster tools account saying a certain file is being blocked by the robots.txt file. It’s not a good idea to tell the search engines not to crawl a page that has generated PageRank.

Now for the Sitemap.xml file, I have mentioned before in a previous post that sitemaps contain priority and frequency levels that you can set to guide the search engines toward more important pages of your site. While that information is true, Rand did mention that the search engines at this time don’t pay any attention to priority levels or frequency settings in your XML sitemap.  Why those two items exist in an XML sitemap, I’m not really sure.

According to Search Engine Land, and Google’s blog post about Caffeine, the new infrastructure may include crawling of the web more comprehensively, determining reputation and authority and returning more relevant results more quickly.  Currently building authority and reputation is building quality links to your website from other high authority websites. Although Matt Cutts says changes are “primarily in how we index”.  They also go on to say that a search for ‘buffy the vampire slayer’ in Google Caffeine returns video and news results midway down the page. In my opinion, this may be good for users, but bad for advertisers.  If I am ranking organically at the bottom of page one, adding videos and news blurbs may push my rank to page 2.  To some clients, their ultimate goal is to be on page one. I hate being that close to page one but occasionally being bumped to page two depending on additional information like news and videos that may be available during the time of the search.

A friend of SEOboy, Shimon Sandler says not to simply build links to your website, but the shift is moving more towards building great content. Shimon goes onto to say that building great content includes more than just adding more webpages to your site.  You need to build content that people will want to link to. And that combination of great linkable content and quality inbound links is the key.

Shimon’s three biggest tips for on-going SEO tactics for the current infrastructure and the new Google Caffeine is to:

1. Continually build content silos

2. Continually participate in social media like Facebook, Twitter and LinkedIn

3. Continually build links using a complex link structure optimized for the social web. Don’t just buy links.

Mashable.com took part in testing the new Google Caffeine, and here are some of their results:

The speed test of Caffeine passed 100% as they claim it is “lightning fast”.  Comparing Caffeine to the current Google search it went from a .25 seconds to .12 seconds for returning search results.

Accuracy of Google Caffeine shows improvements too. They go on to say that the new search engine cares more about keywords than the current.  The search results for ‘are social media jobs here to stay’ brought up more results with those keywords.

They also go into temporal relevancy by saying each perform about the same when it comes to breaking news.

And perhaps most importantly, the index size of the new Google Caffeine provides more results over the current.  However an interesting an non bias fact they found is that Bing brought up even more search results listings than the current Google, new Google and Yahoo with 2 million for Bing and only 183,000 for Ben Parr’s name.

My thoughts are personally, it seems as though the new Google Caffeine is going to show a lot of improvements, but I think Bing may as well. I don’t know about you, but I like the constant competition among the top 3 search engines.  That’s how it should be for every company, a constant struggle to get more customers by providing the best experience. This in turn means more benefits and convenience for us!