A robots.txt is a useful file that keeps search engines from indexing pages you do not want spidered. But why would you not want a page indexed by a search engine? This tutorial will explain why, and how to do it.

More >