But why?

Indexing frequency can be explained as the count or the number of times Google or the other search engines, visit your website to index your content.

And, the higher the frequency, the better you chances for a better rank or your importance in the SERPs.

Can’t believe? Neither did I, once upon a time.

Well, do you know that the best and the top blogs on the internet gets the best indexing frequencies? That is the content published on them gets indexed quicker that anyone else and gets more importance on the SERPs. Examples are TechCrunch.com and Problogger.net.

Content published on TechCrunch and Problogger gets indexed in minutes. The closest we have measured is like around 8 minutes or so. Yeah that’s right, when a content is published on Problogger, it is very likely that the next moment it is on the SERP’s.

At DailySEOblog, I’ve seen content indexed in 30 minutes or so, which is good.

Now the point here is that, if you are writing the same content as the above blogs, like for say example a movie review or so, it is highly likely that even if you have better content, TechCrunch like blogs are likely to get higher on the SERPs.

So they clearly have an edge of having the crawlers index them faster.

Now, how can you get this edge? How can you make crawlers index your content quicker as well?

Several factors.

1. Rapidly changing content.

2. A good sitemap structure.

3. Good pinging tools.

4. Good meta tags that tell the crawlers to visit more often.

5. Easy navigation structure.

Well, so many other factors as well. But these are the best of them I’d say.

Essentially, if you have rapidly changing content, that is made available to the crawlers through a well maintained and updated sitemap, that itself should take care of the problem.

For maintaining the sitemap, of course you’d be using the AllinOne SEO plugin right? That’s a good one.

Also, make sure that you have a regularity in posting. If the crawlers find that your content is being updated on a regular basis, like daily or  hourly, then they will set their crawl frequency(if not manipulated by META tags) to hourly or daily basis, however it is followed on your site.

This is exactly the same reason why blogs have an edge over static websites. It is easy for crawlers to find a pattern in your posts and hence they find it easy to follow the pattern.

You can find more information about this concept over here and here.

Related SEO Tips and Articles:

  1. [...] Mani Karthik tells us about our indexing frequency. [...]

  2. dear Mani
    thank you for useful information
    best regard for you

    [Reply to this comment]

  3. Indexing in google is speed not only for the big blogs but my blog content also got indexed within 30 minutes.. Indexing is different from cache recording. I will show you a screenshot of the quick indexing in google next time when i publish my next post.

    Chetan’s last blog post..Participating in the seocontest2008 ?

    [Reply to this comment]

  4. Dude,

    I have a question on sitemaps created by Google Sitemap XML plugins. Well, in the options page, we are allowed to change the weight placed on posts, home page, category page, etc. Do you think that the default values are good enough?

    Wayne Liew’s last blog post..Wordpress Plugin: Clean Up Broken Images

    [Reply to this comment]

  5. great info buddy. Are there any tools through which we can find out how much time does it take for our content to get indexed or we have to find it manually by visiting google.

    Madhur Kapoor’s last blog post..Zeiss Cinemizer video glasses for iPod

    [Reply to this comment]

  6. Its not just the big names that get indexed fast, the Blogs that use FeedBurner as their RSS provider are getting into to Google with lighting speed…

    I can send you a screen shot that says,”wrote 1 minute ago”… that’s right, 1 Minute.

    But in real time it took about 7 or * minutes to show up.

    This lighting speed indexing has just begon after Google bought FeedBurner, go figure :-)

    [Reply to this comment]

  7. Chetan, That’s exactly the point I’m trying to make. If “ONLY” big blogs were able to make it, then I wouldn’t discuss it for others here. In fact, if you follow certain preliminary stuff, the “so called” small blogs can also achieve the same indexing frequency.

    The default values can be changed for your advantage. Having said that, if SE’s find that the page weightage(which can be easily guessed from your posting frequency) is not the same as what’s given in the sitemap details, then the whole exercise is fatal.

    Madhur, doing it manually will be more precise than using any tool, which I’m yet to find. Makes Sense? :D
    Mani Karthik’s last blog post..Which is the best satellite television in Chennai?

    [Reply to this comment]

  8. Madhur, I found one!
    You can find a tool that will help you find the crawl frequency here - http://www.seometer.com/

    Mani Karthik’s last blog post..Increase your indexing frequency - get the edge over others.

    [Reply to this comment]

  9. Nice post…I do agree that posting frequency has a lot to do with indexing frequncy…

    Techblissonline Dot Com’s last blog post..MySpace to launch developer platform

    [Reply to this comment]

  10. Mani,

    I get indexed in under 10 minutes and my blog doesnt have any gang bang content or doesnt have super links.

    I believe its all about getting your content “travel” and make it travel real fast.

    Think about it laterally, there are two ways in which a bot crawls you:

    1. It DECIDES to crawl you.
    2. It follows your link from elsewhere.

    Make use of that thing called the “RSS Feed” :D ( That was a hint )

    Ashwin’s last blog post..Is It Okay To Submit Your Own Articles To Content Voting Sites

    [Reply to this comment]

  11. That’s a good point there Ashwin.

    Mani Karthik’s last blog post..Increase your indexing frequency - get the edge over others.

    [Reply to this comment]

  12. I have heard this same concept on courtneytuttle as well….

    btw, missed you at the barcamp in cochin….was hoping to meet you up. ;)

    [Reply to this comment]

    Mani KarthikNo Gravatar Replied:

    Hope it was a nice one Anand. There’s always another time or another conference.

    [Reply to this comment]

  13. Even though I don’t post that frequently, my posts get indexed within 20 minutes. It looks like the Google indexing speed has improved a lot.

    [Reply to this comment]

  14. For a second there I thought you were going to say to set a higher priority on your XML sitemaps lol

    Very good points there. I believe the two main factors are PR (not toolbar PR, but actual PR or “link juice”) and frequently updated content.

    Josh Spaulding’s last blog post..How Often Should I Update my Web Pages?

    [Reply to this comment]

  15. Sorry about the bragging, but just came back to tell that my last post: Taking your users for granted was indexed in less than 2 minutes. Wow!!!

    [Reply to this comment]

  16. Nice blog.. Nice information.. nice contents..
    I got lots of knowledge from here..

    Cheers guys.. :)

    [Reply to this comment]

  17. Nice Blog and Simply Nice Information Dude. Thanks :)
    Visit my Blog for More Latest Movies and Games @ http://directlinksheaven.blogspot.com

    [Reply to this comment]