Every time you do a search on a crawler-based search engine, its spider crawls through millions of listings, displaying the ones that best match your search query. The matches will even be ranked, so that the most relevant web sites appear first. Of course, the search engine doesn't always get things right. Sometimes irrelevant pages slip through, and other times, it may take a little more hunting to find what you are looking for. On average though, search engines do an amazing job of locating information.

But how exactly is search engine ranking determined? The answer is, by a set of instructions, called algorithms. Each search engine uses a different set of algorithms, but they all follow the same general principles.

Because search engines want repeat customers, they try to return relevant results. One way they do this, is by looking at the location and density of search terms or keywords, on a web page. Search engines check to see if keywords appear in the title tag, in the header, or in the beginning of paragraphs. They assume that relevant pages will feature these keywords more prominently than irrelevant ones. Search engines will also analyze how many times keywords appear in relation to other words on a page. Web sites with a higher density of keywords are often deemed more significant than other web pages. Meta tags may also play a part in determining relevance. Many search engines do not read them however, and those that do, may weigh them differently.

Search engines may penalize web sites or ban them altogether for spamming. Keyword stuffing, repeated keywords at the bottom of a page, and submitting a page that automatically redirect visitors to a page with different content, are all forms of spamming.

Lastly, search engines also use link analysis to determine search engine ranking. By analyzing how pages link to each other, a search engine can determine what a page is about and whether that page is important enough to merit a boost in ranking. It is important to remember that while the major search engines all follow the same general principles, their algorithms are different. That is why doing the same search on different search engines may generate vastly different results. For instance, some search engines may index more web pages than others, while some may index web pages more often. The end result is that no two search engines have exactly the same catalog of web pages, and thus search engine ranking may differ greatly among them.