Blog Post Tag:

robots.txt

Understanding the Importance of the robots.txt file

By Wil / September 10, 2008

When search engine crawlers (robots) look at a website, the first file they will look at is not your index.html or index.php page. It is your robots.txt file.

Read More