When search engine crawlers (robots) look at a website, the first file they will look at is not your index.html or index.php page. It is your robots.txt file.
My Blog Posts
Every day there are millions of searches conducted on World Wide Web using big search engine sites such as Google and Yahoo!
Have you ever wondered what they are searching for?