How Search Engines Work BY GURU ( DAVE SMITH )
Whether you want surfers to find your site fast or you just want to speed up your own Web searches,
you need to know how search engines work.
People often mistakenly think search engines respond to your queries by searching the entire Web each time.
Oh, would that were true. The Web grows too fast for any search engine to keep up.
Spiders crawl across the Web. Also called crawlers and sometimes called 'bots, these suckers travel across the Web
collecting keywords and links from webpages. Spiders navigate the Web the same way we do. Each time a spider finds a hypertext link on a page, it moves on to that page.
Search engines then take all the links and index them with their own proprietary algorithms. When you search, your results come from a search engine's index, not the entire Web.
To get your site listed in search engines, it's important to put useful keywords into your meta tags. However, if you really want surfers to find you fast, it's much more important to have content good enough so that other sites will link to you.
How to Get Your Site Listed in Search Engines
If you're a webmaster, you want to get listed in as many search engine indexes as possible. Some search engines will let you submit your site, but that means a human has to add the site manually. That's not nearly as fast as attracting the attention of a spider.
According to Search Engine Watch, a leading authority on search engines, the biggest indexes contain well over 500 million pages. So, the key is not just that you're listed, but where you're listed.
You can get spider attention and come up more often in some search engines by using meta tags, but the better and newer search engines don't rely on those. You're better off trying to get other sites to link to yours.
The Myth of the Mighty Meta Tag
Let's say I have a site about sock monkeys. A search for the phrase "sock monkeys" in Google turns up over 3,000 results. If you wanted to see all of those results, you'd have to click through over 40 pages.
Who's going to do that?
Studies have shown that most surfers never click past the first few pages of results. So, your real goal is not
to just be listed in a search engine, but to be listed first or at least on the first page.
Spiders focus on location and frequency of words on a page. Any good webmaster knows to put descriptive
keywords in her page's meta tags (the tags that don't get seen by the viewer). In the past, if you were sneaky enough, you could type words over and over again in your meta tags and you'd be guaranteed high rankings in every search engine.
Not so any longer. Older search engines like HotBot and Infoseek still depend heavily on meta tags, but most of the new generation are smart enough to detect cheats like this, and they'll actually punish your site by sending it to the end of the list.
Raise Your Site's Search Engine Ranking
Newer search-engine technology uses unique relevancy rankings based on the link structure of the Web. Google's patented PageRank technology actually judges sites by how many other sites link to it.
Think of it as a democracy. Take my site about sock monkeys. Say my page is so informative that 100,000
people want to link to me on their site. My ranking in one of the new search engines will increase because
so many people are linking to me.
This link popularity even goes a step farther. If I decide to link to your page, your ranking will increase
because I'm so popular. Get it? It's basically a popularity contest and just by hanging out with me you're more popular because I'm more popular.