Understanding How Search Engines Rank Web Pages

When you do a search using your favorite search engine, the engine quickly sorts through the millions of pages in it's database and provides you with what it feels most closely matches your topic. The matches are also ranked, so that the most relevant ones come first. However, the search engines aren't always "on the money". Often times, non-relevant sites will show up, and it may take a little more to find what you are looking for. But all in all, search engines are still pretty amazing.


So why do search engines continue to occasionally return irrelevant material? Often times it is because the searcher has not provided enough information. If you just search for a single word like "vacation," you will more than likely receive a broad spectrum of results. Search engines don't have the ability to ask a few questions to focus the search. They're not intelligent beings who can rely on judgement and past experience to rank web pages. Although they are moving in that direction, we probably won't see it anytime soon. For that reason, it is important to provide more focus to find what you are looking for.


But what is it that determines relevancy for a search engine? Search engines are set up to follow what is called an algorithm. An algorithm is a mathematical equation which provides the engine's "spider" or "robot" with a set of rules, in essence. These rules mainly involve the location and frequency of keywords within a web page or within certain HTML code. Although the algorithms continually change, there is one thing that stays the same... location and frequency.


Location and Frequency


If you were looking through a stack of books for a specific topic, it stands to reason that you would probably look in the title first. That is basically the first thing a search engines does. If it finds sites with keywords appearing in the title, those sites are more likely to be found relevant in a search query.

Search engines also check to see if keywords appear near the top in the page text. For this reason, headlines or headers can be very important. The engines assume that any page relevant to the topic will mention those words right from the beginning. And if your site is to the point, that should be case.

Frequency is the other major factor in how engines determine relevancy. A search engine will analyze how often keywords appear in relation to other words in a web page. Those with a higher frequency are often deemed more relevant than other web pages. However, there can be a fine line between high frequency and spamming. Spamming is what search engines deem as excessive or overuse of keywords. For instance, if you were to say "vacation, vacation, vacation" multiple times, it would most assuredly be seen as spam and the page would either be given less relevancy or be dropped altogether.


Optimizing Your Results


As we mentioned above, location and frequency are an important aspect of getting good listings in the major search engines, but there is more to do. You may recall that engines utilize "algorithms" to assign relevancy to sites. Well, no engine uses the same algorithm and every one of them changes their algorithms on a regular basis. That's why you receive different results on the same search from time to time.

It is also important to know that several search engines index more web pages than others. Some search engines also index web pages more often than others. The result is that no search engine has the exact same collection of web pages to search through.

Search engines may also give web pages higher rankings for unique reasons. For example, WebCrawler uses link popularity as part of its ranking method. This means they track how many links a specific page in their index has pointing at them. The more links a site has to their page, the better the ranking they would receive in a search query, since it is probable that they are more well-regarded on the Internet. That's one of the reasons why it is important to be listed in more than just the top search engines and directories.

Some search engines also incorporate directories within their site which are separate from their spidered database. These directories are more difficult to get listed in because there is a review process sites must go through to get listed. Only the "best" sites get listings in these areas. Engines that have these separate directories may also give a relevancy boost in a search query to sites they've reviewed. The logic is that if the site was good enough to earn a review, chances are it's more relevant than an unreviewed site.

META tags can be somewhat important for obtaining higher relevancy in certain search engines, as well. However, many people make the mistake of assuming META tags are the "secret" to propelling their web pages to the top of the rankings. The fact is, Alta Vista, HotBot and Infoseek are the only engines that do give a boost to pages with META tags. But engines like Excite, WebCrawler, Lycos and Northern Light don't read them at all. In fact, research has even shown that some pages without META tags still get highly ranked in the engines that do utilize the code. So, they can be important but are by no means the end all be all of getting good rankings.

Search engines may also penalize pages or exclude them from their index if they detect search engine spamming. An example is when a word is repeated hundreds of times in a row on a page or in META tags, to increase the frequency and propel the page higher in the listings. Search engines constantly watch for common spamming methods and often receive complaints from users that spot pages that may have slipped through. In fact, heavy offenders can even have their domain banned from the engine for life.