Welcome to SEO Boy, the authority on search engine optimization -- how to articles, industry news, insider tips, and more! If you like what you see, you can receive free and daily updates via email or RSS.

«

»

Print This Post Print This Post

Get Hip to SEO Lingo by Distinguishing Between Crawl, Index and Cache
December 11th, 2008 | John | Nuts & Bolts of Optimization

For all of those out there that suffer from arachnophobia, this post may offend you. Today I’m going to talk all about search engine spiders. These creepy-crawlers (well, maybe not creepy) are the backbone of the world-wide-web. Whether you’re discussing GoogleBot or SLURP (Yahoo!’s bot), they are the tools by which the search engines discover new websites, new content and ultimately play a huge role in the health of your SEO campaign. But more to the point, when talking about the spiders, crawlers and bots of the world, it’s important to know the lingo! Get ready to learn what crawl, index and cache mean to you and your SEO campaign.

The Crawl

Search engines are built around the ability of search spiders to discover new websites and quickly (and accurately) save those pages for future reference. This process in a nutshell is known as “crawling” or the “web crawl.” Many SEOs refer to their website’s crawl rate, or the rate at which the search engine spiders are returning to pull fresh copies of the content. If you launch a brand new website, the crawl is the first sign that you’re on your way to being indexed by the search engines (refer to the next section!).

So, this begs the question: “How do I tell Google to come crawl my website?” There are a few ways you can do this.

  1. Create accounts at Google Webmaster Tools and Yahoo! SiteExplorer, verify your site and create a sitemap. This is like raising a big red flag saying “I’m over here!”
  2. But if you really want to get a jump on having your website crawled, start building links from established sites. You can start with strong directories or use any number of link building strategies – the important point is getting links!

After your website is established, you may want to increase or decrease your crawl rate. For Google, the first place to look is in your Webmaster Tools account. Under “Statistics” you can view activity from Googlebot on your website for the past 90 days. If you see a problem and need to speed things up or slow them down, go to the “Settings” section. From here you can take control and set a custom crawl rate. Google recommends you only set a custom crawl rate if you are having “traffic problems on your server.”

While that’s all good for Google, what about all the other search engines? Take a gander at this great post from Search Engine Journal that details 10 ways to increase your crawl rate. I won’t repeat them all here, but my favorite? “Update your content often and regularly.” Good advice!

…the most efficient way to get frequent and deep crawls is to develop a website that search engines see as important and valuable.

The Index

To say that the search engines “index” your website’s content is a fancy way of stating that they have your stuff saved on their servers. After one of the search engine spiders has crawled a page on your website, that page’s textual content and other important data is handed off to the “indexer” which stores those pages in a database. You can check how many pages from your website have been indexed in a few different ways:

  1. Visit Google, Yahoo! or MSN and enter the query – site:mydomain.com. This search will show you the pages that are contained in each search engine’s index for your root domain. You can also check sub-domains by entering – site:www.mydomain.com or site:blog.mydomain.com, etc. (There are issues with primary vs. supplemental index, but that’s a blog post for another day!)
  2. Utilizing Google Webmaster Tools and Yahoo! SiteExplorer will also give you results for pages in each engine’s index.

So, your crawl rate is tied to how fast and how many pages from your site will be included in the index. Crawl rate will also determine how fast changes you’ve made to a particular page will show up in each search engine’s index. And just because you’re indexed doesn’t mean you’ll rank in the SERPs. You’ve got to do your homework and perform the SEO basics, too.

The Cache

I like cash, cash is good. Wait, I don’t think we’re talking about the same cash. Oh, you mean CACHE – as in the archived copy of a webpage as indexed by a search engine! If you enter the search query cache:mydomain.com, this will show you the last version of your webpage that was downloaded. The cached version of your webpage is a literal copy of the page that is saved on Google or Yahoo!’s server. So, there is a major difference between index vs. cache – index is text and data, cache is a literal copy. Remember that.

Hopefully, I’ve schooled you on the hip SEO lingo today as it regards to the crawl, index and cache of your website. These are important terms to remember and will help you to navigate through search results and to troubleshoot issues with your SEO campaign.

Do you have anything to add or have a question? Leave me a comment!

Related posts:

  1. Take Control of Your SEO Destiny with a Robots.txt File
  2. My Favorite SEO Argument: Sitemap vs. No Sitemap?
Facebook   IN   Stumble Upon   Twitter   Sphinndo some of that social network stuff.

This entry was posted on Thursday, December 11th, 2008 at 5:00 pm and is filed under Nuts & Bolts of Optimization. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

  • http://www.ibntech.com Shubhangi

    Great Information.The given information cleared my doubts, querries about index and cached.Thanks for sharing .

  • http://www.100seotips.com seo tips

    Thanks for telling the difference between cache and index. This is really a post to understand the difference between cache and index pages. Now i can explain some one that what is google cache and index pages.

    Thanks a lot.

  • http://www.avanindra.com Avanindra

    Nice post on differencing cache – index.
    hope to find some more informative post soon.

  • Robert

    Hi,

    Thanks for the nice post and for clearly differentiating index and cache. I have one query for you, how will I know the date the page was last indexed.

    Thanks!

  • http://www.yourstyledoors.co.uk Jenny

    Hello John,

    Feel great after read you article about deference Between Crawl, Index and Cache. i have another question like which one occurs first cache or index.

  • http://www.chooseusfirst1177.com indianapolis contractor

    A question if I may. when the site is cache on their server and then they in the future crawl the site again does it keep the old version and the new or does it replace the old version.

  • http://www.blogelement.com babor_7uiu

    hey man u send nice information. And i know something:
    Indexing is a process to make a webpage searchable on search engine whereas the process of caching refers to providing a reprinting content snapshot.

    thanks for sharing

  • http://www.watsonpropertygroup.com pinjarra land development

    Google takes a picture of each page, examines and caches (stores) that version as a back-up in the Database. The cached version is what Google makes use of to judge if a page is a good match for a searched query. Google index refers to getting website listed in the Google’s database index for SERPs.Crawling is just the spider visiting and taking info from your pages.

  • Chakri

    Dear John,

    I am also having same question like jenny. First cache or index?

    Thank you,
    Chakri.

  • kaka

    Mobile  plays a more and more important role in our daily life. To achieve its communication function, cell phones have to receive and send powerful HP Laptop Battery ing us great examples. Let’s take a look together.anuary Jones and her black Prada bagDo you see what January Jones was wearing in the picture? HP Laptop Battery acing activities. From the launching of the first racing watch PR516 in 1965 to become the 70s cooperative partner of F1 Team Lotus, Tisso Hermes handbags ce for regular usage. And applied ladies Rolex horologes significantly satisfy their needs.Although the appplied horologes finds it hard Compaq Laptop Battery acing activities. From the launching of the first racing watch PR516 in 1965 to become the 70s cooperative partner of F1 Team Lotus, Tisso rolex replica showing period. A lot of them are viewed since diamond jewelry merchandise that can characterize the person’s manner flavor and public reputa Hermes Replica supports many other services, one of which is Internet. Perhaps when you see me looking at my phone, most of the times, I’m not texting, but surfing laptop battery mechanic movement. The face is made of black steel, which has a punched strap design and style. This has its obvious trace inside throug IBM Laptop Battery looked very attractive. The black Chanel is classic so that it has become a legend of designer bags.aty PerryKaty Perry is always serious. I gucci handbags but far lower expenses. I believe, workable utilized Rolex woman wristwatches don’t have any distinction with those authentic ones. Both of gucci handbags ? Kitty watch collection that contains around 6 variants in different colors. All watches are plated with rose gold and fitted with eye-catching IBM Laptop Battery For this year’s Christmas Day, I have a suggestion of presents for you: why not try to pick a pair of shoes and send them to your loved ones? gucci handbags xperienced a fast developing period in which science and technology bring a lot of progress and changes in every field of our life. Mobile ph IBM Laptop Battery your ideal choice. In one word, with the quality replica bags, you can still go after the fashion and trend. In order to be special and stand out in the rolex replica There are different colors and tattoos of the caps in the market, such as white, black, green, pink, red, yellow, brown, and coffee. laptop battery

  • Ming Wang

    Best handbags that we offer here are so much more than affordable fake designer handbags. And our quality assurance starts way before that. From the very beginning, each of our replica handbags is designed to mimic the exact style you would expect to find from your favorite designer. Gucci replica handbags have the trademark design that will fool even the most discerning eye. Our Replica Louis Vuitton Handbags will have that French inspired flare that you would receive had you purchased it from an expensive designer store. Bottom line. From the very beginning, each of our Louis Vuitton handbags is designed to mimic the exact style you would expect to find from your favorite designer. Gucci handbags have the trademark design that will fool even the most discerning eye. Our Chloe handbags will have that French inspired flare that you would receive had you purchased it from an expensive designer store. Bottom line, we duplicate, down to the tiniest detail, all of the charming aspects of an authentic designer handbag and pass that along to you at a great savings.you will find Balenciaga Handbags,chanel handbags, Replica Prada Handbags ,you can Miumiu Handbags, My friends buy Fendi Handbagsfrom here. i very like knock off designer handbags.My mothe want me help me buy replica designer handbags.Why many men like replica designer bags.you should see Hermes Handbags and alexa mulberry bag replica.you will like the Gucci Baby Items. you should buy louis vuitton handbags.they are best replicas. my friend like Valentino Handbags.

blog comments powered by Disqus