PHP Classes

File: robots.txt

Recommend this page to a friend!
  Classes of Pierre-Henry Soria   PHP Website Informer Tools   robots.txt   Download  
File: robots.txt
Role: Documentation
Content type: text/plain
Description: Documentation
Class: PHP Website Informer Tools
Tool to show relevant information about Web sites
Author: By
Last change:
Date: 9 months ago
Size: 219 bytes
 

Contents

Class file image Download 
# Rules for Google
User-agent: googlebot
Crawl-delay: 0
Disallow:

# Rules for All Other Engines
User-agent: *
Crawl-delay: 10
Disallow:

# Disallow Internet Archiver Wayback Machine
User-agent: ia_archiver
Disallow: /