Understanding search engine crawlers

Did you ever wonder how all those pages got into the search engines in the first place? There's a magic search engine genie that flies from server to server waving a magic wand; not really but close. Actually, there is a computer program called a crawler (or sometimes a spider or robot or'bot) that lives on the search engine's servers. Its job is to surf the Web and save anything it finds. It starts by visiting sites that it already knows about and after that, follows any links that it finds along the way. At each site that it visits, it grabs all the HTML code from every single page it can find and saves that on its own servers.

Later, an indexing server will take that HTML code, examine it, parse it, filter it, analyze it, and some other secret stuff (a lot like waving that magic wand). Finally, your site is saved into the search engine's index. Now, it's finally ready to be served up as a search result. Total time elapsed? About two minutes.

One important thing to note here is that search engine crawlers follow the same links that you do. That means that if you can't click the link, then there's a good chance that the crawler can't click the link either. Fortunately Google does a great job of following JavaScript links, but if you're using JavaScript for your Drupal navigation menus then chances are good that other search engines can't see much past your front page. That's where some creative techniques can really come in handy. Breadcrumbs to show navigation or an XML sitemap (refer to Chapter 5, Sitemaps) can help the crawler find out where to go next. That's why those tools are sometimes called spider food.