PHP Classes
elePHPant
Icontem

Spider website: Crawl a site and retrieve the the URL of all links

Recommend this page to a friend!

  Author Author  
Picture of Karol Janyst
Name: Karol Janyst <contact>
Classes: 2 packages by
Country: Poland Poland


  Detailed description   Download Download .zip .tar.gz   Install with Composer Install with Composer  
This class can be used to crawl a site and retrieve the the URL of all links.

It can retrieve a page of a site and follow all links recursively to retrieve all the site URLs.

The class can restrict the crawling to URLs with a given extension and avoids accessing pages listed in the site robots.txt file, or pages set with the no index or no follow meta tags.

  Classes of Karol Janyst  >  Spider website  >  Download Download .zip .tar.gz  >  Support forum Support forum (2)  >  Blog Blog  >  RSS 1.0 feed RSS 2.0 feed Latest changes  
Name: Spider website
Support forum Support forum
Base name: spider
Description: Crawl a site and retrieve the the URL of all links
Version: 0.1
PHP version: 5.0
License: GNU General Public License (GPL)
 
  Groups   Applications   Files Files  

  Groups  
Group folder image HTML HTML generation and processing View top rated classes
Group folder image PHP 5 Classes using PHP 5 specific features View top rated classes
Group folder image Searching Search engines, crawling and indexing View top rated classes


  Applications that use this package  
No pages of applications that use this class were specified.

Add link image If you know an application of this package, send a message to the author to add a link here.

  Files folder image Files  
File Role Description
Plain text file spider.class.php Class Main class file
Accessible without login Plain text file example.php Example Example file

Install with Composer Install with Composer - Download Download all files: spider.tar.gz spider.zip
NOTICE: if you are using a download manager program like 'GetRight', please Login before trying to download this archive.