Understanding the Importance of the robots.txt file
When search engine crawlers (robots) look at a website, the first file they will look at is not your index.html or index.php page. It is your robots.txt file.
When search engine crawlers (robots) look at a website, the first file they will look at is not your index.html or index.php page. It is your robots.txt file.