A Robots.txt file is a simple file containing information for search engine crawlers visiting your site. You can exactly what parts of your site should be out of bounds for certain bots. This also ensures that page content is indexed faster and more accurately. This has been around for a while now.
Well, breaking news… Google has tightened the way it looks at robots.txt files, if not validated, a lot of site owners could face getting their websites banished from the Google index.
You can easily validate your robots.txt file with robots.txt validator offered in Google’s webmaster console
I found this really interesting thread on these latest robots.txt developments, called “validate your robots.txt, Google might deindex your site” by Sebastian’s Pamphlets.
Anyway, I better double-check my clients robots.txt files now.
