PHP Classes

PHP Google Search Crawler: Perform searches and retrieve results from Google

Recommend this page to a friend!
  Info   View files Documentation   View files View files (32)   DownloadInstall with Composer Download .zip   Reputation   Support forum   Blog    
Ratings Unique User Downloads Download Rankings
Not enough user ratingsTotal: 397 All time: 6,593 This week: 80Up
Version License PHP version Categories
google-crawler 1.1.11GNU General Publi...7Searching, Web services, PHP 7
Description 

Author

This class can perform searches and retrieve results from Google.

It can send requests to Google search pages Web server to perform searches for given keywords.

The requests may be sent using proxy servers.

The package parses the result pages and extracts the results as a result object that provides an array interface so it can be traversed using regular array functions.

Picture of Vinicius Dias
Name: Vinicius Dias <contact>
Classes: 2 packages by
Country: Brazil Brazil

Documentation

Google Crawler

Latest Stable Version Build Status Code Coverage Scrutinizer Code Quality License

A simple Crawler for getting Google results.

This component can be used to retrieve the 100 first results for a search term.

Since google detects a crawler and blocks the IP when several requests are made, this component is prepared to use some online proxy services, such as hide.me.

Installation

Install the latest version with

$ composer require cviniciussdias/google-crawler

Usage

Crawler class constructor prototype

CViniciusSDias\GoogleCrawler\Crawler::__construct(
    SearchTermInterface $searchTerm, GoogleProxyInterface $proxy = null,
    string $googleDomain = 'google.com', string $countryCode = ''
)

Parameters

  • $searchTerm Term that will be searched on Google
  • $proxy Online proxy service that will be used to access Google [optional]
  • $googleDomain Your country specific google domain, like google.de, google.com.br, etc. [optional]
  • $countryCode Country code that will be added to `gl` parameter on Google's url, indicating the location of the search. E.g. 'BR', 'US', 'DE' [optional]

Examples

Without proxy

<?php
use CViniciusSDias\GoogleCrawler\{
    Crawler, SearchTerm
};

$searchTerm = new SearchTerm('Test');
$crawler = new Crawler($searchTerm); // or new Crawler($searchTerm, new NoProxy());

$resultList = $crawler->getResults();

With some proxy

<?php
use CViniciusSDias\GoogleCrawler\{
    Crawler, SearchTerm, Proxy\CommonProxy
};

$searchTerm = new SearchTerm('Test');
$commonProxy = new CommonProxy('https://us.hideproxy.me/includes/process.php?action=update');
$crawler = new Crawler($searchTerm, $commonProxy);

$resultList = $crawler->getResults();

More details on proxies

To know more details about which proxies are currently supported, see the files inside tests/Functional folder. There you'll see all the available proxies.

Iterating over results

foreach ($resultList as $result) {
    $title = $result->getTitle();
    $url = $result->getUrl();
    $description = $result->getDescription();
}

About

Requirements

  • This component works with PHP 7.2 or above
  • This component requires the extension php-ds to be installed

Author

Vinicius Dias (ZCE) - carlosv775@gmail.com - https://github.com/CViniciusSDias/ - http://www.zend.com/en/yellow-pages/ZEND030134

License

This component is licensed under the GPL v3.0 License - see the LICENSE file for details


  Files folder image Files  
File Role Description
Files folder image.phan (1 file)
Files folder imagesrc (5 files, 2 directories)
Files folder imagetests (2 directories)
Accessible without login Plain text file .travis.yml Data Auxiliary data
Accessible without login Plain text file composer.json Data Auxiliary data
Accessible without login Plain text file composer.lock Data Auxiliary data
Accessible without login Plain text file Dockerfile Data Auxiliary data
Accessible without login Plain text file infection.json.dist Data Auxiliary data
Accessible without login Plain text file LICENSE Lic. License text
Accessible without login Plain text file phpstan.neon Data Auxiliary data
Accessible without login Plain text file phpunit.xml Data Auxiliary data
Accessible without login Plain text file README.md Doc. Documentation

  Files folder image Files  /  .phan  
File Role Description
  Accessible without login Plain text file config.php Aux. Auxiliary script

  Files folder image Files  /  src  
File Role Description
Files folder imageException (3 files)
Files folder imageProxy (4 files)
  Plain text file Crawler.php Class Class source
  Plain text file Result.php Class Class source
  Plain text file ResultList.php Class Class source
  Plain text file SearchTerm.php Class Class source
  Plain text file SearchTermInterface.php Class Class source

  Files folder image Files  /  src  /  Exception  
File Role Description
  Plain text file InvalidGoogleHtmlException.php Class Class source
  Plain text file InvalidResultException.php Class Class source
  Plain text file InvalidUrlException.php Class Class source

  Files folder image Files  /  src  /  Proxy  
File Role Description
  Plain text file CommonProxy.php Class Class source
  Plain text file GoogleProxyInterface.php Class Class source
  Plain text file KProxy.php Class Class source
  Plain text file NoProxy.php Class Class source

  Files folder image Files  /  tests  
File Role Description
Files folder imageFunctional (3 files)
Files folder imageUnit (4 files, 1 directory)

  Files folder image Files  /  tests  /  Functional  
File Role Description
  Plain text file AbstractCrawlerTest.php Class Class source
  Plain text file DefaultCrawlerTest.php Class Class source
  Plain text file PersonalizedCrawlerTest.php Class Class source

  Files folder image Files  /  tests  /  Unit  
File Role Description
Files folder imageProxy (3 files)
  Plain text file CrawlerTest.php Class Class source
  Plain text file ResultListTest.php Class Class source
  Plain text file ResultTest.php Class Class source
  Plain text file SearchTermTest.php Class Class source

  Files folder image Files  /  tests  /  Unit  /  Proxy  
File Role Description
  Plain text file CommonProxyTest.php Class Class source
  Plain text file KProxyTest.php Class Class source
  Plain text file NoProxyTest.php Class Class source

 Version Control Unique User Downloads Download Rankings  
 100%
Total:397
This week:0
All time:6,593
This week:80Up