PHP Classes

File: robots.txt

Recommend this page to a friend!
  Classes of Pierre-Henry Soria   PHP Website Informer Tools   robots.txt   Download  
File: robots.txt
Role: Documentation
Content type: text/plain
Description: Documentation
Class: PHP Website Informer Tools
Tool to show relevant information about Web sites
Author: By
Last change:
Date: 9 months ago
Size: 219 bytes
 

Contents

Class file image Download
# Rules for Google User-agent: googlebot Crawl-delay: 0 Disallow: # Rules for All Other Engines User-agent: * Crawl-delay: 10 Disallow: # Disallow Internet Archiver Wayback Machine User-agent: ia_archiver Disallow: /