We had seen earlier that Google is now indexing your site quick and fast as never before. More of this has got to do with the frequency of content change that’s happening in your site. And blogs are at an advatange here compared to other static paged websites.

This is a screenshot from my indexing results I took this morning. Soon after I had published an article (Reader’s Questions), the article was absorbed/indexed by the Google bot. It took only 39 minutes.(See the screenshot) The indexing must have been done earlier but I checked it only at 39 minutes late.

One more proof that you can decide yourself how often the crawler should visit your site. Just give it food every often and the poor crawler will visit frequently.

Google has announced officially that it has now tweaked it’s settings that if you have more content updates, the crawlers will absorb your data more frequently.

The quicker the content update, the quicker the crawling. This is with respect to the data from your verified blog in the Google Webmaster’s dashboard.
I’m assuming that this might have some effect on the data collected by the Google search bot too. Anyways, the more the content update, the better - Thumbrule!

Photo courtesy - Somma

Here is a compilation of all the search engine optimization tips for bloggers ( Wordpress, Blogger, Typepad and other platforms) featured here on DailySEOblog. Bookmark this page and you can refer to them any time - honestly quite some of them are really good tips particularly catering to Google, which are not featured elsewhere. Some of them are basic SEO stuff, you may want to have a look.

Importance of primary and secondary keywords - What are primary and secondary keywords? Why should you select them, and how they will help you rank high on search engines.

Five sure shot tweaks to rank high on Google - Must read for bloggers - The top 5 things you should do to ensure high ranks on Google. Only the relevant SEO tips for Google.

How interlinking your pages will help you rank higher on the SERPs - Importance of interlinking, why and nhow should you interlink your pages so that the google bot catches it.

Optimize your Wordpress categories to avoid duplicate content - Wordpress creates a lot of duplicate content by default. Here are tips on how you can reduce it.

Create a user-navigation sitemap for your blog - Crawlers like it - Sitemaps are not only for crawlers, here is how you can create a manual sitemap with ease.

How to avoid duplicate content on Wordpress? - All the tips you need to know on avoiding duplicate content on Google.

User server location advantage to rank high on Google - If you are hosting a website, did you know that the location of your server will help you gain edge on regional search engines?

Optimize the robots.txt file for Wordpress, allow your blog to rank high - Robots.txt is an often ignored file, which is actually an excellent tool that will help you get more files indexed on google and thus rank high. Here ate the tweaks.

Create and submit a sitemap for Yahoo - Sitemaps are different for each search engine, here’s how you can create a sitemap customized for Yahoo in easy steps. 

Tweak title tags of your blog to rank high - Title tags are very important to get search engines attention. here are tips on how you should write an attractive title.

Highlighting your content with a SEO friendly layout - Having great content is not enough. Here are tips on how to decorate it to cater to the spiders.

Importance of footer text in SEO - Footer text is an ignored element which can be used effectively to feed information to the search engines.

How to build a sitemap for large websites and blogs - Building a sitemap for a small blog or site is easy. But if you have a large website, it turns messy. Learn how you can still get a great sitemap ready without mess.

Importance of allinanchor text - Why is allinanchor text and how you should take care of them.

What are supplementary results? Do they affect you? - Everyones, night mare once upon a time.

5 vital SEO stats that you should keep track of - If i were to suggest you 5 SEO metrics that you should constantly keep track of so that you can keep your positions in tact, here are they.

Does Google page rank affect your rankings? - Google page rank is often given more importance than what it actually deserves. Read this article to study the facts.

Importance of incoming links - Why is an incoming link important and how they will help you rank high.

Keyword usage in site content - facts and myths - How should you use keywords in your site content. Do’s and dont’s.

How to SEO on an Ajax-ified site? - When your site is AJAX-ful SEO becomes tough, but here’s how you can effectively harness the power of AJAX as well as SEO together.

How image ALT tags helps you to rank high - Why should you use image ALT tags, where you should use and how?

Flash and SEO - moving together the web2.0 times - How to optimize pages in flash.

Creating SE friendly post titles - What is the importance of titles, how should you write titles to get the edge over other blogger who has the same content.

How to get indexed by Google in 48 hours - Getting indexed on Google is not easy, but here’s a tip that will ensure you indexing in 48 hours or less.

Selecting keywords for SEO - What are keywords and what keywords should you select for your blog?

SEO friendly layout - Ensuring a SEO friednly layout will help you rank high on search engines automatically. See what factors makes your blog SEO friendly.

Let’s talk today about good and bad menus first. Good menus are those that are SEO friendly, bad ones are those that are not. Simple logic.

When dealing with clients, one thing that’s common to all, is about the use of fancy javascript navigation menus, that troubles me most. If you use one, please refrain from using it. If you are a blogger, there is very little chance that you are using one. But the JS enabled menus are mostly found in portfolio like pages.

A few words about portfolio pages. They are a gold mine for onsite optimization, there would be many wrongs. They are all frills and no stuff. They are stupid. They make lot of noise.

Many a times, they have this animated JS enabled menus that are not crawlable by bots. Big mistake and out subject of talk today.

Crawlers by default, avoid anything that’s Javascript enabled and Ajax. This is one fallback of modern crawlers, even the google one. They haven’t deviced a technology that will help them crawl the contents of JS enabled elements on a webpage and AJAX. They had a problem with Flash, but it is almost rectified now.

• Javascript/AJAX powered navigation menus may look cool and attractive to the eyes, but avoid them whereever possible. Instead use CSS and you could create almost similar spectacular and eye-catching menus.
• Don’t use images for making menu images, instead try CSS, it’s cleaner, quick and easy to manage.
• If it’s completely necessary, keep the JS menus, but submit the linking pages to google.
• Avoid sound effects please, it’s so 90’s!
• Use text to link to pages in the navigation menus if possible.
• Place the navigation menus wither on the top, or the left hand table, or at a place where crawlers won’t miss it.
• Don’t embed them in frames - big mistake!

So the idea is that Navigation menus should be simnple, crawlable, easy to navigate and thus serving the purpose. Let them not be too fancy, complex and JS enabled. It defeats the purpose.

You can find excellent resources on making CSS Navigation menus here
CSS Tutorials on Navigation menus CSS Menu help High quality CSS Menus

With the advent of Wordpress and other arrays of blogging softwares, poor old forums have lost their charm. 3 years earlier, PhpBB was the most used software in forums, almost every webmaster had a forum about something. It was an instant hit since it’s launch because of it’s folksonomical nature, but in a more un refined way as compared to the latest softwares.

One major reason why a forum failed is because of it’s non SEO friendly nature.

• Forums did not have SEO friendly urls
• They did not allow bots to crawl the inner pages
• They had SEO un friendly titles
• Forums in general did not look at SE’s with importance at all, so they did not leave any room for tweaks as well.

With all these problems in front of us, let’s see how we can make the forums (PhpBB mainly) SEO friendly?

• Optimizing the page titles
  Page titles are a crucial factor for search engine optimization. Crawlers pick up the titles from every page and it’s important to give maximum information in this area in a non-spammy way. Here’s how you do it by some quick edits.

  - Edit file ‘templates/subSilver(or your template)/overall_header.tpl’

  Replace -

<title>{SITENAME} :: {PAGE_TITLE}</title>

with

<title>{PAGE_TITLE}</title>

or

<title>{PAGE_TITLE} :: {SITENAME}</title>

Edit Viewtopic.PHP

Replace -

$page_title = $lang['View_topic'] .’ - ‘ . $topic_title;  with
$page_title = $lang['View_topic'] .’ - ‘ . $topic_title;

• Tweaking the robots.txt file to allow better Search Engine indexing.
  Robots.txt can be used to control what pages is available to search engine robots and what pages are not. Generally on PhpBB, there are lot of sub directories that are allowed to crawl by default. But actually this is not required. We only need to allow certain folders and files to be crawled - so that only the necessary information is available to the crawler.

  Including the below code in to your robots.txt will disallow all the unwanted folders from being crawled by search engines.
  This way only the necessary information (those in the posts) are available for crawlers and the junk is filtered out.

User-agent: *
Disallow: /admin/
Disallow: /db/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /privmsg.php
Disallow: /profile.php
Disallow: /search.php 
Disallow: /templates/
Disallow: /common.php
Disallow: /config.php
Disallow: /faq.php
Disallow: /viewonline.php
Disallow: /groupcp.php
Disallow: /login.php
Disallow: /memberlist.php
Disallow: /modcp.php
Disallow: /posting.php

• Plugins to help create sitemaps for PhpBB
  Here are some cool plugins/mods for PhpBB that will help you create SEO friendly urls and even sitemaps that will help you get more pages indexed on Google. Here goes -

1. Google, MSN and Yahoo sitemap generator mod for PhpBB - This mod will create a Google SitemapIndex and auto generated Google Sitemaps for phpBB
2. Google Puller - This hack creates pages with 500 links to posts on your community. The words are taken from actual posts on your community (from the search_word table) and they link to the post that contains that word.
   The aim of the hack is to help search engines like Google or MSN to index your pages.

Part 2 of this post will follow with more methods on how to optimize your forum software for more search engines.

I had to write about this. There are lot of fraud SEO’s around. No doubt. Many of them somehow downloads a pirated copy of some popular SEO book ( there are many around ) and becomes SEO’s overnight (That’s not my idea - I read it on the DP forums ). Now they go on to set up a one - page website with a Google, MSN and Yahoo logo with a dart picture (un-licensed) saying something like “The world’s best SEO - Rank No.1 guaranteed “. Have you seen one?

It’s easy to make them out as frauds from their overly detailed testimonials and super-duper client list (Some even have Yahoo on the client list - what the..?). It’s essential for genuine SE optimizers to make aware the public on these frauds and hence this post.

This post is inspired from this article by Philip Lenssen. I thought this post might enlighten you to make out between fraud SEO’s and genuine one’s from their claims. Though Philip has mentioned many white-hat terms here, I think it’s OK to assume that many of them are used by fraud SEO’s than genuine ones.

He says

In the SEO industry agencies, experts and even bloggers have adopted a special mode of speech not to say slang that might be misunderstood by outsiders like clients, website visitors or the general public. To help you understand what search engine optimization experts really mean I devised this real glossary of SEO speak:

And here is the article -

What they say…
What they mean…

We offer Search Engine Optimization/SEO
We assume you are the Google bot and want you to index this page for both keywords

We offer Search Engine Optimisation
Our SEO company is based in the UK

Guaranteed top positions
We place Google Adwords for you

We do SEO, SEM, PPC to increase your ROI
We do not want you to know what we do

We stick to the Google Webmaster Guidelines
We only break them in a way that we assume Google won’t notice

We tell you how to make money online
We want you to click on our ads

10 ways of making money online
Those are our 10 affiliates, please click on the respective undisclosed ads

We offer social media optimization
We got several accounts banned at Digg

We offer link baiting services
We want to put those drunk naked ladies video on your site

Our network
Our link farm

Authority sites
Sites that do no SEO

Black hat SEO
We do anything to get rich quick, even if your site gets banned

White hat SEO
We only cheat Google where we have to, others do it too, come on!

We optimize for Google, Yahoo, MSN, Ask
If we fail in Google you still have to pay

Search Engine Submission
We need your mail address, those guys offered us $$$ for each 1000 verified addresses

Partners
People we never heard of until we exchanged links

PageRank optimization
Sorry, we just started doing SEO and do not have a clue

SEO India
We offer 1000 links for 30$

Alexa optimization
All our employees have the Alexa Toolbar installed, it really works!

Did you know that your wordpress blog is not completely SEO friendly?

Of course, wordpress is 90% SEO friendly with it’s title tags, seo friendly templates and the like. But there is one loose point in wordpress that spoils the whole show.

I’m talking about duplicate content. This matters most with Google.It does not like duplicate content on any site. And Google’s handcuff to this problem is Supplemental index.

Let’s see how Wordpress is responsible for creating duplicate content on your blog and ultimately your blog ending up in the supplementary index.

Villain No.1 - Archive pages

Villain No.2 - Categories

Archive pages
Have you set your default archive settings to daily, weekly or monthly? If yes, you are in trouble. When google bots visit your site, it sees the same content first on the post page, then on the archive page, and third on the index page. So it is duplicate content three times.

Categories
Do you have the habit of tagging a particular post in more than one category? Trouble again!

How to overcome?

Solution 1
- Do not archive pages. If at all you are archiving pages, make sure that robots/crawlers does not crawl those pages.
- How to ensure robots don’t crawl archived pages.
- Use nofollow robots meta tags on the archive template.Here’s how you do it -

<meta name="robots" content="noindex,follow">

-Or use this Duplicate content cure plugin - It automatically adds the nofollow robots meta tag on to the archive pages, so that archived pages are not crawled by google.

Solution 2
- Do not categorize posts in more than one category. One post - one category. Mixing up categories makes duplicate content for the crawlers as well as creates a non-user friendly navigation structure as users are likely to see the same posts on each category.

Google penalizes blogs and sites for a number of reasons. John Chow is the recent victim. He was taken off the google index recently for doing something that violated Google webmaster guidelines.

John had been running his “Link to me - to get linked back” scheme, whereby if you write a small review on his blog and link to him with the anchor text,”Make money online”, he would link back to you. Many bloggers linked to him and got linked back from John.

Now that John Chow is kicked out of Google Index, what does it mean to other bloggers who have linked to him?

Google says it clearly in it’s guidelines that -

Don’t participate in link schemes designed to increase your site’s ranking or PageRank. In particular, avoid links to web spammers or “bad neighborhoods” on the web, as your own ranking may be affected adversely by those links.

So that means anyone who linked to John Chow is going to get the penalty this Google PR update. Yikes! Are you one of those?

Does it affect John Chow?

Nope. The man had made the blog hugely popular and enjoys a steady source of traffic.He’s even declared that banning from Google has done no detrimental effects on his income.

Does it affect you?

If you’ve linked to John, and he’s linked back - well yes may be. Only this Google PR update will tell you.

So what are the things that I shouldn’t do to get a bad reputation with Google?

Yeah, the guidelines are a bit boring to read through. so let’s me list out the most overlooked things by most of the people.

1. Choose a good server that has no downtime - Google does not like servers that has frequent downtimes. Worse - when the Google bots visit your site, if your server is down, they flag you red!
2. Don’t do those black hat SEO stuff like keyword stemming, cloaking, doorway pages etc.
3. Don’t use a hosting that has a shared IP of a banned user on Google - Google may think you’re friends?
4. Too much similar anchor text linking to you - Google guesses it out that you’ve done something artificial to get those links, hence banned.
5. Stupid link building campaigns - Like John’s stay away from them. They are temporary.

Since the Google PR update is nearing, make sure that your blog is safe of these troubles. Remember, no matter how huge or popular you are - if you are not white hat and genuine - Google is going to get you one day - once and for all!

The robots.txt file is used to control the crawlers activity on a website/blog. It will help you to keep some directories away from crawling while allowing some. For example if yu have two folders 1.Articles and 2.Javascripts - and if you wish to exclude Javascripts from crawling by robots, then you can command it on the robots.txt file.

A few basics about what the robots.txt file is -
- It is found in the root folder, Ex:-www.google.com/robots.txt
- It’s a text file and can be edited
- It is used to command the robots what to crawl and what not
- It is used to help the crawlers locate the sitemap on your site

If you are on blogger platform, then you can’t upload the robots.txt file. Panic not - there is another option which you can utilize. I’ll discuss it towards the end of this article. First let’s discuss a normal robots.txt implementation on a hosted site.

Implementing the Robots.txt file on a web-hosted site(Wordpress)

Pre-requisites - I assume that you have a wordpress hosted site with Cpanel/FTP access.

- Find the file at your public_html folder. If it isn’t there, create a blank text document.

Excluding a folder from crawling by SE bots.
Suppose you don’t want Google to index one of your folders.
In the robots.txt file, you have to specify two things - which crawler agent(Google, Yahoo, MSN) do you want to keep out and - which folder/folders you want to exclude.

The general syntax to be written in the Robots file is this.

User-agent: *
Disallow: /yourfolder/

Here, user-agent:* means all search agents(Google,MSN,Yahoo etc).
/yourfolder/ restricts that folder from crawling. Note that the sub-folders will not be crawled too.

In order to keep all agents away from crawling ALL folders, use this code.

User-agent: *
Disallow: /

You can specify individual crawler agents with their names(replacing *) like google bot,lycra etc.If you are following a general command to all search engine crawlers, keep the * in the user-agent line.

Specifying a sitemap with the Robots.txt file
Due to the recent agreement with the major search engines, they have come up with a common command that they will follow to detect sitemaps from robots.txt file. The command is -

Sitemap: Sitemap url here

Robots.txt for Blogger users.

Blogger users cannot upload the robots.txt file instead, they can use the robots meta tag to control the crawling of bots on particular files.

These codes should be included in the HEAD section of the particular page template.(Enclosed in arrow brackets)

META NAME=”ROBOTS” CONTENT=”NOINDEX”

This command will not index the current page in which this code is included.

META NAME=”ROBOTS” CONTENT=”NOFOLLOW”

This command will not follow/parse the links present on the particular page where this code is present in the head section.Blogger users can use this option to their advantage when making posts.If you want every new page to be crawled by the bots, include the following code to head section of your blogger template.

meta name=”robots” content=”index, follow”

Happy driving the robots.

Today, let’s see more about sitemaps. Every webmaster must have a sitemap ready for his site and submit it to google in order to get all the pages listed on Google.Sitemaps are of two types, as you know the HTML sitemap you use to navigate a site and second the sitemap used to help crawlers crawl the pages more effectively.

Why are they necessary?

Sitemaps are not necessary.(Yep i said that) Even if you don’t have sitemaps the crawlers will crawl your pages and find the content. But, it is like letting them crawl in a dark room. What if you had a well lit room with all navigation and helpers around which will take them to each room? It will be more effective right? Sitemaps serve this purpose.

It has the site structure ready giving indication to the crawlers as to which are the folders/files that are important, which are not, which are the folders/files that are to be visited frequently, and which are the ones to be visited only once. This helps the crawlers to undersand your site more effectively.

Now, how to build a sitemap for blogger?

It’s very simple in Blogger. It only requires you to go to the Google Webmaster Central and ass your site feed and the sitemap is automatically created. You can get detail instruction on this here.Make sure that you submit your full feed and not partial one.

Which is the best sitemap generator program around?
There are lot of free online and downloadable sitemap generators.
Here’s a simplified listing of what is best.

1- Python Scipt - This is the most difficult one to install. But if you are familiar with python, then this is the best one around.It’s automated and requires no additional support.I don’t recommend it for a beginner.Requires technical knowledge.

2 - Online sitemaps - This is best for small websites. It’s easy, simple and online.Just go to this site and submit your url.Fill in some basic details like time and priority settings for the files and click go!The whole sitemap will be generated online.You will get both ROR file and the Google sitemap XML file.If you are interested only in Google, use the XML sitemap.The format is according to Google sitemap protocol and is faultless.
Best choice for beginners and small websites of less than 500 pages.

3 - Gsite Crawler - This is a downloadable application. If your website is a bit large and you have time to tweak some settings and is serious about sitemap, then i would recommend this guy for you.
It requires you to give the website url, then select the types of files to be scanned from it, priority settings are automatically detected, and you can create bot Google sitemap and Yahoo url.
It has report generation as well that will give you an idea of how many urls were crawled and broken links etc.This is very useful while handling large sites.

How to make sitemap for large sites?

If you have really large websites for instance a one million page one, then it’s really going to be tough creating a sitemap. Practically this is possible with the Python script but if you are not okay with the technical stuff then you got to depend on sitemap generator programs.(If you don’t have a really large website the follwing piece of information may not help you.)

Step 1 - Download a free sitemap generator program like Gsite crawler.
Step 2 - Use it to crawl each folder of your website as separate projects.Make sure that you create a new database each time a new project is opened.
Step 3 - Now you have separate sitemaps for each folder.
Ex:- yourdomain.com/folder1 has a sitemap called folder1.xml and yourdomain.com/folder2 has a sitemap called folder2.xml
Step 4 - Download this simple index generator program.
Step 5 - Copy paste all the folders (containing the sitemaps) from thh projects folder of Gsite crawler(C:program files…) and put it into one single folder.
Step 6 - Run the index generator program against this parent folder.
Step 7 - Now a sitemap index would be created with links to all the child sitemaps but one problem, since in Gsite Crawlers projects folder(C:Program Files) each crawled folder will be named with underscore replacing the forward slash.
Ex:- yourdomain.com/folder will be named as yourdomain.com_folder
Therefore the sitemap index produced will have the links too this way.
Step 8 - Use notepad/wordpad to open the sitemap index file. Find and replace all the underscores with forward slash.
Step 9 - Upload the child sitemaps in the respective folders online.
Ex: - yourdomain.com/folder1..folder2 etc.
Step 10 - Upload the sitemap index file to the root folder and submit it to google.

Bingo! There you go you have now created a sitemap index and child sitemaps for a large website. Now submit it trough the webmaster central window and keep waiting!