Google started the trend when they announced the new attribute. Along with them hoped onboard also Yahoo and MSN and a lot of other blog-related sites. The original idea was to stop the comment spam on blogs. Well, it had an impact, but not the one desired, because spam still exists even with the nofollow-ed links.

Another use that a lot of the SEO world saw, was controlling the outbound links on a page, by adding rel="nofollow" to the links you want not to hand over a part of your page rank. And another way of using nofollow, which I myself used a few times, is to give nofollow to a link which performs an operation which has an effect on the user's browsing session, but would not affect the search engines' bots. I know that theoretically this shouldn't be done by GET methods, as the HTTP RFC specifies. It is said that for actions the POST method should be used. But sometimes, just sometimes it looks better with a link than a button to submit a form.

One day a few months ago I wanted to see which search engines obey to this rule, and which don't, which follow links which are given rel="follow" and which don't. I put a special file on my site fur testing purposes only, which I marked it with nofollow. A couple of months later here we are, and I have an interesting piece of data I want to share.

So when we add them up, in a three months interval, the page was crawled as such:

It seems that Google alone did the most crawling, from those which seem to be a search engine crawler. So what does this mean? Are the search engines not respecting their own imposed crawling filter?

Comments