Even if you have a kickass site, sometimes you have to wait for ages to get some pages indexed and crawled on Google. Why does this happen ? I’ve found many people complain that after building a great site, and doing the standard chores, waiting so long for the results doesn’t make any sense. Certain things just don’t happen as planned. Like certain pages on a site, doesn’t get indexed and if indexed doesn’t get the right visibility on the search engines. Strange.
I can probably attribute all these problems to a couple of SEO factors. Let me try and explain them below.
Setting a proper sitemap solves most of the problems often. But don’t take this lightly as we have many automated plugins to do the job. Many a times, the automated plugins need good tweaking to ensure that they have the right settings for your site. I have written several posts about sitemaps, like this one, check it out.
This is not a way to “control” how often the bots visit you. Google has its own algorithm to find out when it should visit you, but publishing more articles in a regular fashion sure does give Google clues on what your posting frequency is like. Nothing wrong in publishing articles in random, but I’d suggest you keep it pretty much in a predictable fashion so we make the process easy.
Cross linking between pages contextually is a great way to make sure that the bots visit all those pages. Contextual links are given more importance probably than any other link on the site, so make use of them.
If you have a self designed website, make sure you keep the directory structure simple and not too deep. Having more directories to crawl will make the job difficult for the bots. Keep it simple.
Along with making sure that all the pages are crawled, also make sure that the ones you don’t need to show up in the search results are blocked. (Like the TOS page.) You could use the Meta Noindex and the robots.txt exclusion protocol to get things done.
Often I see websites with complex, fancy navigation bar annoying. I mean, it doesn’t have to be fancy to impress your visitors right ? You could use classy CSS styling to get elegant looks and still impress them. The problem with fancy navigation bars are that they often do more harm than good like not allowing proper crawling of the pages linked. There are various techniques you can use to make the navigation bar attractive and still allow smooth crawling, employ them.
Always practice deep linking (you shouldn’t be having a deep structure in the first place.) if you have a big and deep directory structure. Do not miss out linking to any directory/pages. Use clever linking strategy to link all the directories/categories and pages each other. If possible link to all the main category indexes right from the homepage.
Linking to pages with images and other elements may also result in poor crawling of those pages. Use proper anchor texts to share the contextual factor among your pages and directories.
Confusing URLs are neither good for Google bots as well as visitors. Google bots love simple, meaningful URLs and visitors love URLs with recall value and those are easy to remember. If you don’t want to screw up things, keep the URLs simple and easy to remember (both for bots and humans). P.S – Google does not have any problems with seemingly confusing or meaningless URLs. They still make out the content from the page, however having meaningful URLs are a definite plus.
The homepage as you know is likely to gather the most link juice and importance compared to other pages on the site. (This is just a general observation, but if a page gathers more links to it, it could also get more authority and link juice than the homepage.) Use the homepage well, and add more links to the inner pages from it, so they share the authority.
Essentially, factors like how fast to index, how often to index and how deep to index are decided by the search engines. But to a certain level you can control their crawling speed and frequency. The points above are good pointers to this, hope they helped you.
SEO Auditor Features - Complete SEO audit - Competitor Analysis - Report generation Try it today ! |
Link Assistant Features - Easy Link Building - Finds link partners - Get backlinks regularly Try it today ! |
Rank Tracker Features - Rank Check reports - 558 Search Engines - Keyword Research Try it today ! |
Possibly related SEO & Social Media Articles
Nice tips mani
Reply
Sometimes its the simplest things that really make a difference. Thanks for the info.
Reply
Thanks for the tips!!
Reply
Great guidance for a starter like me
yet to start a blog…
I hope these tips help me…
Reply
Your points have been observed and dually noted. Thanks for the tips! This will help me and other (neophyte) bloggers get that much closer to our goals… Much appreciated!
Reply
Don’t forget to use the no-follow attribute to sculpt the way your redistribute PageRank from your high ranking pages.
Reply
Simple but great and important points
Reply
[...] Daily Seo Blog AKPC_IDS += "5748,"; [...]
Excellent suggestions… Hard to keep myself in line with…lol
Will start adding contectual links since I’ve really slacked on that one. Mind if I post and attribute this to your authors? I like it a lot.
Reply
[...] crawling speed and frequency. The points above are good pointers to this, hope they helped you. Daily SEO Blog Share and [...]
[...] crawling speed and frequency. The points above are good pointers to this, hope they helped you. Daily SEO Blog Share and [...]
hi i have had my blog on word press for a week now http://www.diyanswerdirect.com/blog but nothing is showing it up in google, aswell as my site http://www.diyanswerdirect.com has been crawled but nothing has been updated can anyone tell me when google updates its pages thsnka lee
Reply
It took me about three days to get my blog initial indexed by google. I even wrote a few posts about the whole process. If you have a new site, I suggest submitting it directly to Google. As for me, I am working on getting it crawled faster and more often, this will take some time, since it is a new blog/site. I need to build the importance of the site in Google’s eyes.
Reply
[...] 10 Tips to ensure better search engine crawling of your site – Daily SEO blog [...]
Applying relevence to your keywords in your content, urls and internal links is the biggest lesson to take away here
Reply
if SE can’t crawling some page in my site. what i need should?
Reply
[...] 10 Tips to ensure better search engine crawling of your site [...]