File: |
Robots.txt.class.php |
Role: |
Class source |
Content type: |
text/plain |
Description: |
Core file |
Class: |
Robots_txt
Test if a URL may be crawled looking at robots.txt |
Author: |
By Andy Pieters |
Last change: |
Bug fix for regexp problem |
Date: |
16 years ago |
Size: |
12,447 bytes |
|
|
|