Memory problem solved

Recommend this page to a friend!

Memory problem solved

Subject:	Memory problem solved
Summary:	Use Attila's rewrite of this class to solve memory problem
Messages:	1
Author:	Anon E Mouse
Date:	2009-11-12 01:13:55

1. Memory problem solved

Report abuse

Anon E Mouse - 2009-11-12 01:13:55

I already posted replying to another thread

But thought I would make a new thread to help others

Attila rewrote this class to add one zip at a time.

Memory used is now reduced to only 2*(filesize currently being zipped)

Works great!

I used the example script below with Attila's class to zip 30 .mov files which created a zip file of 330MB (impossible to do in memory with the original version of the class)

Here is my example file, Attila's class follows that.

EXAMPLE FILE:

<?php

set_time_limit(0);

include "conn_doc.php";

include_once("zipfile.inc.php");

while(list($key,$value)= each($_GET)){
$$key = $value; // this line does the actual creation and assignment of variables
}

echo("<html><head></head><body>");
echo(' Zipping your files now, please be patient.... ');
flush();

$directoryToZip="./"; // This will zip file(s) in this present working directory
//$outputDir="/"; // The desired output directory. NOTE: leaving it that way made some weird msg unzipping
$outputDir="/yourfiles/"; // The desired output directory.

$createZipFile=new CreateZipFile;

$rand=(rand()%9999);
$zipName=$JobID."-".$rand.".zip";
$fd=fopen($zipName, "wb");
$createZipFile->ZipFile($fd);

$createZipFile->addDirectory($outputDir);

$sql = "SELECT * from Job WHERE JobID=".$JobID." ORDER BY IncludeInList DESC, Priority";
$ri = mysql_query($sql);
for ($count = 1; $row = mysql_fetch_row ($ri); ++$count)
{
$Movie=$row[20];
If ($Movie){
// Code to Zip a single file
$fileToZip=$Movie;
$createZipFile->addFile($directoryToZip.$fileToZip, $outputDir.$fileToZip);
echo("adding ".$fileToZip." ");
flush();
} // end If Movie

} //finish the Job recordset For Loop

mysql_close ();

$createZipFile->close($fd);

echo("Successfully created file ready for downloading");

echo(' <A HREF="'.$zipName.'">Click here to Download the zip file</A>');

echo("</body></html>");

?>

ATTILA'S CLASS

Note, you can get it directly from here:

cygnussystems.hu/php/ZipClass.html

Here it is pasted below for completeness:

<?php
/**
* Original source was created by Rochak Chauhan, www.rochakchauhan.com.
* But it used a lot of memory so it was modified.
* The original code kept the continuesly growing .zip stream in memory and in addition
* adding a new file to the stream required filesize*4 bytes.
*
* Now the growing .zip stream is not kept in memory, it is written to the file continuesly.
* Adding a new file to the .zip requires filesize*2 bytes of memory.
* Plus some more memory required to store the .zip entries - this is written only at the end
* of the process.
*
* @author ironhawk, Rochak Chauhan www.rochakchauhan.com
* @package zip
*/
class ZipFile {

public $centralDirectory = array(); // central directory
public $endOfCentralDirectory = "\x50\x4b\x05\x06\x00\x00\x00\x00"; //end of Central directory record
public $oldOffset = 0;

protected $fileHandle;
protected $compressedDataLength = 0;

/**
* Creates a new ZipFile object
*
* @param resource $_fileHandle file resource opened using fopen() with "w+" mode
* @return ZipFile
*/
public function ZipFile($_fileHandle)
{
$this->fileHandle = $_fileHandle;
}

/**
* Creates a new folder in the zip
*
* @param string $directoryName folder full path to the .zip root, e.g. "/qqq/www"
* @return void
*/
public function addDirectory($directoryName) {
$directoryName = str_replace("\\", "/", $directoryName);

$feedArrayRow = "\x50\x4b\x03\x04";
$feedArrayRow .= "\x0a\x00";
$feedArrayRow .= "\x00\x00";
$feedArrayRow .= "\x00\x00";
$feedArrayRow .= "\x00\x00\x00\x00";
$feedArrayRow .= pack("V",0);
$feedArrayRow .= pack("V",0);
$feedArrayRow .= pack("V",0);
$feedArrayRow .= pack("v", strlen($directoryName));
$feedArrayRow .= pack("v", 0 );
$feedArrayRow .= $directoryName;
$feedArrayRow .= pack("V",0);
$feedArrayRow .= pack("V",0);
$feedArrayRow .= pack("V",0);

fwrite($this->fileHandle, $feedArrayRow);
$this->compressedDataLength += strlen($feedArrayRow);
$newOffset = $this->compressedDataLength;

$addCentralRecord = "\x50\x4b\x01\x02";
$addCentralRecord .="\x00\x00";
$addCentralRecord .="\x0a\x00";
$addCentralRecord .="\x00\x00";
$addCentralRecord .="\x00\x00";
$addCentralRecord .="\x00\x00\x00\x00";
$addCentralRecord .= pack("V",0);
$addCentralRecord .= pack("V",0);
$addCentralRecord .= pack("V",0);
$addCentralRecord .= pack("v", strlen($directoryName) );
$addCentralRecord .= pack("v", 0 );
$addCentralRecord .= pack("v", 0 );
$addCentralRecord .= pack("v", 0 );
$addCentralRecord .= pack("v", 0 );
$addCentralRecord .= pack("V", 16 );
$addCentralRecord .= pack("V", $this->oldOffset );
$this->oldOffset = $newOffset;
$addCentralRecord .= $directoryName;
$this->centralDirectory[] = $addCentralRecord;
}

/**
* Adds a new file to the .zip in the specified .zip folder - previously created using addDirectory()!
*
* @param string $directoryName full path of the previously created .zip folder the file is inserted into
* @param string $filePath full file path on the disk
* @return void
*/
public function addFile($filePath, $directoryName) {

// reading content into memory
$data = file_get_contents($filePath);

// create some descriptors
$directoryName = str_replace("\\", "/", $directoryName);
$feedArrayRow = "\x50\x4b\x03\x04";
$feedArrayRow .= "\x14\x00";
$feedArrayRow .= "\x00\x00";
$feedArrayRow .= "\x08\x00";
$feedArrayRow .= "\x00\x00\x00\x00";
$uncompressedLength = strlen($data);

// compression of the data
$compression = crc32($data);
// at this point filesize*2 memory is required for a moment but it will be released immediatelly
// once the compression itself done
$data = gzcompress($data);
// manipulations
$data = substr($data, 2, strlen($data) - 6);

// writing some info
$compressedLength = strlen($data);
$feedArrayRow .= pack("V",$compression);
$feedArrayRow .= pack("V",$compressedLength);
$feedArrayRow .= pack("V",$uncompressedLength);
$feedArrayRow .= pack("v", strlen($directoryName) );
$feedArrayRow .= pack("v", 0 );
$feedArrayRow .= $directoryName;
fwrite($this->fileHandle, $feedArrayRow);
$this->compressedDataLength += strlen($feedArrayRow);

// writing out the compressed content
fwrite($this->fileHandle, $data);
$this->compressedDataLength += $compressedLength;

// some more info...
$feedArrayRow = pack("V",$compression);
$feedArrayRow .= pack("V",$compressedLength);
$feedArrayRow .= pack("V",$uncompressedLength);
fwrite($this->fileHandle, $feedArrayRow);
$this->compressedDataLength += strlen($feedArrayRow);
$newOffset = $this->compressedDataLength;

// adding entry
$addCentralRecord = "\x50\x4b\x01\x02";
$addCentralRecord .="\x00\x00";
$addCentralRecord .="\x14\x00";
$addCentralRecord .="\x00\x00";
$addCentralRecord .="\x08\x00";
$addCentralRecord .="\x00\x00\x00\x00";
$addCentralRecord .= pack("V",$compression);
$addCentralRecord .= pack("V",$compressedLength);
$addCentralRecord .= pack("V",$uncompressedLength);
$addCentralRecord .= pack("v", strlen($directoryName) );
$addCentralRecord .= pack("v", 0 );
$addCentralRecord .= pack("v", 0 );
$addCentralRecord .= pack("v", 0 );
$addCentralRecord .= pack("v", 0 );
$addCentralRecord .= pack("V", 32 );
$addCentralRecord .= pack("V", $this->oldOffset );
$this->oldOffset = $newOffset;
$addCentralRecord .= $directoryName;
$this->centralDirectory[] = $addCentralRecord;

}

/**
* Close the .zip - we do not add more stuff
*
* @param boolean $closeFileHandle if true the file resource will be closed too
*/
public function close($closeFileHandle = true) {

$controlDirectory = implode("", $this->centralDirectory);

fwrite($this->fileHandle, $controlDirectory);
fwrite($this->fileHandle, $this->endOfCentralDirectory);
fwrite($this->fileHandle, pack("v", sizeof($this->centralDirectory)));
fwrite($this->fileHandle, pack("v", sizeof($this->centralDirectory)));
fwrite($this->fileHandle, pack("V", strlen($controlDirectory)));
fwrite($this->fileHandle, pack("V", $this->compressedDataLength));
fwrite($this->fileHandle, "\x00\x00");

if($closeFileHandle)
fclose($this->fileHandle);
}

}
?>
</textarea/>
</body>
</html>

OK that's all folks.

The other thing I found useful was comparing differences between Attila's class above, and Rochak's version.

I used the free program CS Diff to compare what code had changed.
Very informative.

Thanks again to everyone who has made this script possible, it saved my bacon :-)

About us

Advertise on this site

For more information send a message to info at phpclasses dot org.