Home
About me
Sitemap
Wordpress SEO
Contact
SEO jargon busters
RSS feeds
Daily SEO Blog
Jul
27

10 Tips to ensure better search engine crawling of your site

Even if you have a kickass site, sometimes you have to wait for ages to get some pages indexed and crawled on Google.  Why does this happen ? I’ve found many people complain that after building a great site, and doing the standard chores, waiting so long for the results doesn’t make any sense. Certain things just don’t happen as planned. Like certain pages on a site, doesn’t get indexed and if indexed doesn’t get the right visibility on the search engines. Strange.

I can probably attribute all these problems to a couple of SEO factors. Let me try and explain them below.

1. Set a sitemap, with the right priorities

Setting a proper sitemap solves most of the problems often. But don’t take this lightly as we have many automated plugins to do the job. Many a times, the automated plugins need good tweaking to ensure that they have the right settings for your site. I have written several posts about sitemaps, like this one, check it out.

2. Publish articles regularly in a predictable fashion

This is not a  way to “control” how often the bots visit you. Google has its own algorithm to find out when it should visit you, but publishing more articles in a regular fashion sure does give Google clues on what your posting frequency is like. Nothing wrong in publishing articles in random, but I’d suggest you keep it pretty much in a predictable fashion so we make the process easy.

3. Link well contextually and often between posts

Cross linking between pages contextually is a great way to make sure that the bots visit all those pages. Contextual links are given more importance probably than any other link on the site, so make use of them.

4. Keep the directory structure simple

If you have a self designed website, make sure you keep the directory structure simple and not too deep. Having more directories to crawl will make the job difficult for the bots. Keep it simple.

5. Block the unnecessary pages with proper SEO techniques

Along with making sure that all the pages are crawled, also make sure that the ones you don’t need to show up in the search results are blocked. (Like the TOS page.) You could use the Meta Noindex and the robots.txt exclusion protocol to get things done.

6. Make the navigation bar simple and accessible

Often I see websites with complex, fancy navigation bar annoying. I mean, it doesn’t have to be fancy to impress your visitors right ? You could use classy CSS styling to get elegant looks and still impress them. The problem with fancy navigation bars are that they often do more harm than good like not allowing proper crawling of the pages linked. There are various techniques you can use to make the navigation bar attractive and still allow smooth crawling, employ them.

7. Practice deep linking

Always practice deep linking (you shouldn’t be having a deep structure in the first place.) if you have a big and deep directory structure. Do not miss out linking to any directory/pages. Use clever linking strategy to link all the directories/categories and pages each other. If possible link to all the main category indexes right from the homepage.

8. Make use of proper anchor texts and internal text links

Linking to pages with images and other elements may also result in poor crawling of those pages. Use proper anchor texts to share the contextual factor among your pages and directories.

9. Keep the URLs simple and easy to remember

Confusing URLs are neither good for Google bots as well as visitors. Google bots love simple, meaningful URLs and visitors love URLs with recall value and those are easy to remember. If you don’t want to screw up things, keep the URLs simple and easy to remember (both for bots and humans). P.S – Google does not have any problems with seemingly confusing or meaningless URLs. They still make out the content from the page, however having meaningful URLs are a definite plus.

10. Use the homepage very well

The homepage as you know is likely to gather the most link juice and importance compared to other pages on the site. (This is just a general observation, but if a page gathers more links to it, it could also get more authority and link juice than the homepage.) Use the homepage well, and add more links to the inner pages from it, so they share the authority.

Essentially, factors like how fast to index, how often to index and how deep to index are decided by the search engines. But to a certain level you can control their crawling speed and frequency. The points above are good pointers to this, hope they helped you.

SEO Auditor Features
- Complete SEO audit
- Competitor Analysis
- Report generation
Try it today !
Link Assistant Features
- Easy Link Building
- Finds link partners
- Get backlinks regularly
Try it today !
Rank Tracker Features
- Rank Check reports
- 558 Search Engines
- Keyword Research
Try it today !



Possibly related SEO & Social Media Articles

  • 3 Effective Ways to block Google from crawling parts of your website
    Sometimes, we try hard to welcome Google, and try best to make it index the pages from a website, but sometimes we don’t want them to crawl the pages too. What do we do in those cases ? Can we control Google the way we want ? Well, sometimes yes, sometimes not. Let’s see how [...]...
  • SEO Tips Day 9 – Optimizing URLs for Search Engines
    Are there, at all, any optimizing procedures for URLs on a website, or is it okay to leave them just like that, they way they are ? That’s the question many webmasters ask these days regarding URL optimization, because Google had cleared once that they didn’t have too much trouble crawling through those long URLs, and [...]...
  • Handy Tips on Internal Linking and SEO : Where, When and How to link
    Do I have to explain what internal linking is to you ? Guess not. Am pretty sure that you know already much about it, however here goes my two cents. Internal Linking is the process when you place links within your site, between pages and thus control the Google Link juice (to a level) via text/image [...]...
  • Search Engine Optimization tips for MSN Live
    Search Engine Optimization techniques are pretty much similar for all search engines like Yahoo and MSN. If you optimize your site focusing Google, then others follow – that’s what we’ve seen so far. However, the MSN Live team too have some specific SEO techniques they suggest to webmasters that help in search engine specific metrics. [...]...
  • Ensure effective Google crawling on blogs
    If you don’t know nothing about robots.txt and controlling robots through Meta tags -and don’t want to take the pain of editing the txt file in your root folder and disallowing each search engine crawler manually – worry not! There is an easy way to get this done in Wordpress. Goto Dashboard> Options> Privacy settings. You will [...]...

Category: SEO Tips | Author: Mani Karthik
SEO Wordpress Themes
  1. Anish K.SNo Gravatar

    Nice tips mani

    Reply

  2. JaredNo Gravatar

    Sometimes its the simplest things that really make a difference. Thanks for the info.

    Reply

  3. TimNo Gravatar

    Thanks for the tips!!

    Reply

  4. suganindiaNo Gravatar

    Great guidance for a starter like me
    yet to start a blog…
    I hope these tips help me…

    Reply

  5. HerTechNo Gravatar

    Your points have been observed and dually noted. Thanks for the tips! This will help me and other (neophyte) bloggers get that much closer to our goals… Much appreciated!

    Reply

  6. Jacques Snyman | Web DevelopmentNo Gravatar

    Don’t forget to use the no-follow attribute to sculpt the way your redistribute PageRank from your high ranking pages.

    Reply

  7. NiharNo Gravatar

    Simple but great and important points

    Reply

  8. Diez consejos para asegurarte que tu sitio sea bien indexado | Reparacion de PC a Domicilio - Blog

    [...] Daily Seo Blog AKPC_IDS += "5748,"; [...]

  9. ChristianNo Gravatar

    Excellent suggestions… Hard to keep myself in line with…lol
    Will start adding contectual links since I’ve really slacked on that one. Mind if I post and attribute this to your authors? I like it a lot.

    Reply

  10. Great Suggestions For Improve Search Engine Crawl-from DailySEOBlog | Naper Design

    [...] crawling speed and frequency. The points above are good pointers to this, hope they helped you. Daily SEO Blog Share and [...]

  11. Improve Search Engine Crawl-from DailySEOBlog | Naper Design

    [...] crawling speed and frequency. The points above are good pointers to this, hope they helped you. Daily SEO Blog Share and [...]

  12. leeNo Gravatar

    hi i have had my blog on word press for a week now http://www.diyanswerdirect.com/blog but nothing is showing it up in google, aswell as my site http://www.diyanswerdirect.com has been crawled but nothing has been updated can anyone tell me when google updates its pages thsnka lee

    Reply

  13. Janesville SEONo Gravatar

    It took me about three days to get my blog initial indexed by google. I even wrote a few posts about the whole process. If you have a new site, I suggest submitting it directly to Google. As for me, I am working on getting it crawled faster and more often, this will take some time, since it is a new blog/site. I need to build the importance of the site in Google’s eyes.

    Reply

  14. TwittLink - Your headlines on Twitter

    [...] 10 Tips to ensure better search engine crawling of your site – Daily SEO blog [...]

  15. Ecommerce Development BrightonNo Gravatar

    Applying relevence to your keywords in your content, urls and internal links is the biggest lesson to take away here :)

    Reply

  16. đá ốp látNo Gravatar

    if SE can’t crawling some page in my site. what i need should?

    Reply

  17. New site not indexed on Google yet ? Here’s what to do | Peer Marketing Group

    [...] 10 Tips to ensure better search engine crawling of your site [...]





Click to cancel reply

  • Create Flash Websites
  • Knownhost
  • Recent SEO Blog Articles

    • Migrating from shared hosting to dedicated servers – Things to consider
    • 5 Ways to find out if an SEO is lying to you
    • Upload any file of upto 250MB to Google Docs ( Yeah, any file type ! )
    • New site not indexed on Google yet ? Here’s what to do
    • Confused over .com and regional domain SEO Advantages ?
    • How to detect mobile browsers & Redirect mobile users to another URL
    • 3 Effective Ways to block Google from crawling parts of your website
    • News from Google: Fix duplicate content issues using canonical tag across websites
    • Header Tags – What are they ? Where and How to Use Them Effectively
    • Speed up your pages like right now !
    • 5 Routine SEO house keeping tasks to check site health regularly
    • 4 SEO Factors Search Engines might talk about more in 2010
    • 3 Simple Ideas that will get backlinks, even if you don’t ask
    • SEO Tips Day 14 – How to submit a new website to search engines ?
    • SEO Tips Day 13 – 3 Crucial things you should avoid in SEO
    • SEO Tips Day 12 – Top 3 Influencing factors in backlinks
    • Google now indexes pages via RSS/Atom feeds on your site
    • SEO Tips Day 11 – How to deal with duplicate content issues
    • SEO Tips Day 10 – Domain Age and SEO – How important is it ?
    • SEO Tips Day 9 – Optimizing URLs for Search Engines
    • SEO Tips Day 8 – Will linking to other sites, reduce your Page Rank ?
    • SEO Tips Day 7 – Would buying links get you banned from Google ?
    • SEO Tips Day 6 – Use of Keywords in Page Title
    • SEO Tips Day 5 – Don’t pick the keyword with highest search volume
    • SEO Tips Day 4 – Make an attractive page title for better click through rate
  • Popular at Daily SEO blog

    • 10 Twitter tools to effectively manage your followers (130)
    • How to get indexed by Google in 48 hours (102)
    • How to get a Google Wave invite (even if you didn’t sign up earlier) (92)
    • How to get free backlinks (90)
    • Ultimate list of Dofollow Social Bookmarking sites (80)
    • Free SEO Wordpress theme – SEO Blog (76)
    • 9 Twitter Tips that will help you gain respect in the Twitterverse (Like @Zaibatsu) (73)
    • 25 SEO Gurus you should follow on Twitter (61)
    • New Wordpress theme - "SEO Green" from DailySEOblog (58)
    • SEO for Wordpress - A quick guide (57)
  • SEO Tools & Resources

    • 6 Twitter Analytics Tools to measure your Social Media presence
    • SEO Tool: Find out if you’re doing Black Hat SEO, check for SEO errors
    • Track content changes on any page even without an RSS Feed
    • Google introduces “Browser Size” tool for webmasters
    • 10 Essential Adobe Air Applications for Social Media Addicts
    • Integrity – Broken Link Finder SEO Software for Mac
    • Find out people who’re not following you back on Twitter
    • Google Advanced Search Operators right in your browser (So you don’t have to memorize them)
    • You can actually make money on Twitter, A method that really works !
    • Firefox Plugin – Monitor AdSense earnings from Firefox statusbar
  • Social Media Articles

    • 6 Twitter Analytics Tools to measure your Social Media presence
    • What is Twitter for ? Link Building – In case you didn’t know !
    • New to Twitter ? 7 Tips to make Twittering a rewarding experience
    • Why is it easy to convince your client of Social Media Marketing but not implementing it
    • 4 Cool Twitter Applications to help you monitor weight, health and stay fit !
    • Why I Tweet Frequently (and still don’t count it as a mistake)
    • 5 Tips to make Twittering more meaningful by increasing the signal to noise ratio
    • StumbleUpon gets a makeover – Gets cool new features !
    • Top 25 Most Popular PodCasters on Twitter you shouldn’t miss following
    • 5 Features I’d like to see on the Su.pr URL sharing service
    • Top 35 News Sources on Twitter you should follow : Keep updated with the latest news
    • 8 Habits of Successful & Popular Twitter users for Inspiration
    • 10 Tricks to search Twitter better, Beyond your timeline – Around the world
    • How to make money on Twitter ? Some ideas that work ( without screw ups ) !
    • 9 Mistakes I committed on Twitter and could’ve avoided !
  • Recent SEO Tips

    • 6 Twitter Analytics Tools to measure your Social Media presence
    • SEO Tool: Find out if you’re doing Black Hat SEO, check for SEO errors
    • Google thinks Bing, Dogpile & Ask.com are better search engines than itself ! (Proof)
    • Track content changes on any page even without an RSS Feed
    • Migrating from shared hosting to dedicated servers – Things to consider
    • Orkut is 6 year old today (and there’s no surprise gift for ya) !
    • How to get certified as an SEO ? Take the Google SEO test !
    • 5 Ways to find out if an SEO is lying to you
    • 20 Awesome Workspaces with Multi Monitor Setups for inspiration
    • What is Twitter for ? Link Building – In case you didn’t know !
    • We love you Google – From your fans in China !
    • Upload any file of upto 250MB to Google Docs ( Yeah, any file type ! )
    • New to Twitter ? 7 Tips to make Twittering a rewarding experience
    • New site not indexed on Google yet ? Here’s what to do
    • Confused over .com and regional domain SEO Advantages ?
  • Top SEO Tags

    ads adsense backlinks blog blogger Blogging blogosphere blogs categories check crawler crawling crowd Excel google image images indexing india indian instances internet javascript keywords links pages rank robots search engine optimization seo tips serps sitemap submit tags templates theme themes upload video wordpress Wordpress SEO wordpress seo wordpress tips yahoo youtube

Basic SEO Tips

  • Change the way your site appears on Google SERPs
  • Control Google's crawl frequency to your site
  • How does Google see your site?
  • Importance of footer text in SEO
  • Optimizing header images
  • SEO tips for MSN search engine
  • Optimizing blog titles for Google
  • SEO friendly layout
  • What is an SEO friendly site structure?
  • Importance of keywords - SEO Tips
  • Anchor text importance - SEO Tip
  • SEO Tips for blogs and bloggers
  • Absolute links or relative links is good for SEO?
  • Important SEO tips for wordpress

SEO Tips for blogs

  • 24 Must have SEO plugins for Wordpress
  • SEO Tips - Copy writing guide
  • How to get quality backlinks?
  • Social media and link building tips
  • SEO Tips - Landing page Optimization
  • How to increase traffic to your blog?
  • How to increase your Page Rank?
  • Image optimization tips for blogs
  • 7 steps to your blog's SEO
  • How and where to find incoming links?
  • SEO tips for Google
  • SEO metrics to track
  • How to build a sitemap for blogger
  • How to get Google sitelinks?
  • Find out when Google indexed you
Mani Karthik Hi, I'm Mani Karthik, SEO, Blogging & Social Media Enthusiast. I primarily blog about SEO and Social Media on this blog, basically How-to articles and tutorials to help the SEO learner. Feel free to have a look around and drop comments. Hope you enjoy your stay.
>> More about me
© 2009 DailySEOblog.com Privacy policy