PHP Classes

Fast PHP CURL Multiple Requests: Retrieve the content of multiple URLs using CURL

Recommend this page to a friend!
  Info   View files Example   Videos Videos   View files View files (5)   DownloadInstall with Composer Download .zip   Reputation   Support forum (3)   Blog (1)    
Last Updated Ratings Unique User Downloads Download Rankings
2022-02-22 (4 days ago) RSS 2.0 feedStarStarStarStar 72%Total: 655 This week: 25All time: 4,863 This week: 10Up
Version License PHP version Categories
curl-multi 1.0.10GNU General Publi...5HTTP, PHP 5, Web services
Description Author

This class can retrieve the content of multiple URLs using CURL.

It can take an array of URLs and sends HTTP requests to all of them in parallel using the CURL extension multi-request support.

The class queues the URL of each page to be retrieved one at a time to make it more efficient.

Recommendations

Very Fast Curl class
The fastest Curl class

I want to get the all info related to particular product fromURL
Amazon url get data from it.

custom search engine using phpp
search engine using php

read html files
I need to read and publish html file content

Innovation Award
PHP Programming Innovation award winner
August 2018
Winner


Prize: One big elePHPant Plush Mascott
The PHP CURL extension allows retrieving data from Web servers of multiple URLs at the same time, however this may overload the server of a given site.

This class provides a solution that makes this task lighter by just sending the request to the first URL of a list and only after that it sends the requests to the remaining URLs of that list.

Manuel Lemos
Picture of riccardo castagna
  Performance   Level  
Name: riccardo castagna <contact>
Classes: 7 packages by
Country: Italy Italy
Innovation award
Innovation award
Nominee: 3x

Winner: 1x

Details
#Author Riccardo Castagna
#email: 3315954155@libero.it 
#cUrl Multi fast & furious and also very simple and stable.

The cUrl_extension is one of the most important extensions of the PHP especially when we have to interface with external APIs and we must communicate with other applications.

Well, as we know, many novels and many volumes have been written on this topic :))) :
something about to sleep while running and to stay calm and others nice and beautiful things! I joke ... 
 
During a project that I was carrying out, where I needed to make multiple and simultaneous asynchronous requests, I came across a problem with the cUrl_Multi, which, after some vain but useful research, some analysis and many tests I developed a solution through an insight.

The problem is when you use the cUrl Multi with arrays to add the handles because it can cause the loss of some requests.
 
Well, sometime the solutions are complex and sometime are simple.

In this case, the cUrl Multi, adding a simple solution during the execution of the first loop to add the handles, became stable.   

The problem has been in how the array with the cURL handles was executed, 
in fact if the loop of the array is executed normally all together, from the key number zero to the key number (n)keys, the cUrl multi, sometime, could returns errors or loses some hits.

The solution, I found, is to detach the first key adding the first handle to it without a loop, than executing the first loop, to add the handles, for the subsequent keys. 
All the requests will still remain anyway simultaneous because they are performed by the second loop with the curl_multi_exec.

In this way is very stable, light and fast. I have stressed very much this class with several tests, and it never lost a beat.

Since I had read somewhere, I do not remember where, that cUrl_Multi gave some problems when the number of requests were equal or greater than 10,
I did also a test with 13 simultaneous requests and it is ok, no problems.
 
I have also simulated an error inserting a wrong url inside the array and it go on to execute all the rest with no stop and no errors.    


About this topic I wrote also something here:
"3315954155 at libero dot it" 
http://php.net/manual/it/function.curl-multi-add-handle.php#122964

As default options inside the main class there are:

curl_setopt($x, CURLOPT_URL, $y); 
curl_setopt($x, CURLOPT_HEADER, 0); 
curl_setopt($x, CURLOPT_FOLLOWLOCATION, 1); 
curl_setopt($x, CURLOPT_RETURNTRANSFER, 1); 
curl_setopt($x, CURLOPT_TCP_FASTOPEN, 1); 
curl_setopt($x, CURLOPT_ENCODING, "gzip,deflate"); 
curl_setopt($x, CURLOPT_SSL_VERIFYPEER, 0); 
curl_setopt($x, CURLOPT_SSL_VERIFYHOST, 0);

obviously you can change this options, according to your needs and according
to your libcurl version (view: https://curl.haxx.se/libcurl/c/symbols-in-versions.html) 
editing the main class file: ./lib/class.curlmulti.php  
private function set_option($x, $y)  
--------------------------------------------------------------------------------------------
Usage:
 
include_once("./lib/class.curlmulti.php"); 
$ref= new cURmultiStable;

$urllinkarray = array('http://php.net/manual/it/function.curl-multi-add-handle.php', 
'http://php.net/manual/en/function.curl-multi-init.php', 
'http://php.net/manual/en/function.curl-multi-setopt.php'
);

$urls = $ref->runmulticurl($urllinkarray);

foreach ($urls as $value){
echo $value; 
}
----------------------------------------------------------------------------------------------

OR:

include_once("./lib/class.curlmulti.php"); 
$ref= new cURmultiStable;

$urllinkarray = array('http://php.net/manual/it/function.curl-multi-add-handle.php', 
'http://php.net/manual/en/function.curl-multi-init.php', 
'http://php.net/manual/en/function.curl-multi-setopt.php'
);

$urls = $ref->runmulticurl($urllinkarray);

echo $urls[0],$urls[1],$urls[2];  
---------------------------------------------------------------------------------------------- 
OR:

include_once("./lib/class.curlmulti.php"); 
$ref= new cURmultiStable;

$urls = $ref->runmulticurl(array('http://php.net/manual/it/function.curl-multi-add-handle.php', 
'http://php.net/manual/en/function.curl-multi-init.php', 
'http://php.net/manual/en/function.curl-multi-setopt.php'
));

echo $urls[0],$urls[1],$urls[2];
----------------------------------------------------------------------------------------------
OR for a single request:

include_once("./lib/class.curlmulti.php"); 
$ref= new cURmultiStable;

$urls = $ref->runmulticurl(array('http://php.net/manual/it/function.curl-multi-add-handle.php#122964'));

echo $urls[0]; 
//or 
foreach ($urls as $value){
echo $value; 
}
---------------------------------------------------------------------------------------------- 
THE END ... SIMPLE, FAST, LIGHT AND STABLE !!! 

Sincerely, good job to everybody.  
  Files folder image Files  
File Role Description
Files folder imagelib (1 file)
Accessible without login Plain text file index.php Example EXAMPLE FILE
Accessible without login Plain text file index_2.php Example EXAMPLE FILE
Accessible without login Plain text file index_3.php Example EXAMPLE FILE
Accessible without login Plain text file readme.txt Doc. README FILE

  Files folder image Files  /  lib  
File Role Description
  Plain text file class.curlmulti.php Class MAIN CLASS

 Version Control Unique User Downloads Download Rankings  
 0%
Total:655
This week:25
All time:4,863
This week:10Up
User Ratings User Comments (1)
 All time
Utility:91%StarStarStarStarStar
Consistency:91%StarStarStarStarStar
Documentation:83%StarStarStarStarStar
Examples:91%StarStarStarStarStar
Tests:-
Videos:-
Overall:72%StarStarStarStar
Rank:289
  
For more information send a message to info at phpclasses dot org.