How to Use Queue To Speedup PHP Processing Tasks Part 1: Queueing Slow Tasks

Contents

The Importance of Queues to Improve Performance

Implementing Queues in PHP

Conclusion

The Importance of Queues to Improve Performance

Queues are everywhere. When you delete a photo on your iPhone, it goes to the Bin and will be deleted from there after 30 days. That is a queue. Without it you will not be able to recover a picture removed by mistake.

When you work on a plane and are able to write 3 or 4 emails offline, have them saved in outbox and get them sent when you get to the WIFI connection in airport, that is a queue. Without it you will not be able to write several messages offline, the mailing program would just complain that connection is not available after you finished your very first email and would not allow you to create the second one.

Literally, a queue is also a long line of people in McDonalds. Several managers just take a next customer and serve them their order. If there was just a single manager, the queue processing would take ages and huge profit loss.

So a queue allows the work to be shared among multiple "processors" (also known as "agents" or "workers"). So, these 3 examples just illustrate the basic principles of queues:

1. Queues make the user experience better

2. Queues allow to postpone the heavy processing and thus make the interface more robust

3. Queues allow separation of work between many workers

No surprise that if you apply queues in your application, its development will become more joyful and the application will become more robust.

A quick example is sending a notification email after a successful purchase. If you make your user wait until the email gets sent in real time and your email server is slow, your user will get not as happy as he or she could be if the interface would quickly say "Hey, thanks for purchase".

To achieve that speed of interface response, you just need to save the notification into your notifications queue and process it later soon, but not immediately. Magic!

So this article is about how to work with queues (easy!), and it consists of 2 parts with PHP examples:

1. How to fill a queue

2. How to process the queue.

Implementing Queues in PHP

So, let's clarify what a queue is: technically it is a data structure where data comes from one end and leaves the queue from another end. An example is a car wash and a line of cars: the first car from the line gets into the car wash and then leaves it, then the second car gets in, and so on.

That is why the queue is also called a FIFO: "first in, first out" — in contrast to a stack, another data structure, one which "last in, first out" approach is used (your browser history for example).

Since queues are that simple by definition, you can use almost any kind of data storage. You can even use a usual PHP array:

<?php
class Queue {

 protected $_storage = array();

 public function addItem( $item ) {
  $this->_storage[] = $item;
 }

 function getItem() {
  return array_shift($this->_storage);
 }

};
?>

That would work if your queue has a very short "time to live",— within a single script call. The $queue variable will not exist if you refresh the page.

So you need a persistent data storage container and PHP adapter to access it. From my experience that could be:

• A plain text file, a log file, why not?

• Gmail inbox, say, if you need to process email messages that keep and keep coming, then you don't have to care about filling the queue though.

• Special queues solutions like Apache ActiveMQ or Apache Kafka

• A web-service based storage like iron.io

• A database table, I think, one of the easiest and available solutions

Step 1. Create a Queue

Let's see an example of a database table, it's very easy. You just need to create a table called queue, where every line is a 'job' to process, and the table has these fields:

• id: this is an unique ID of a task

• data:— the task data to process. Can be image path to process, email to sent or a domain name to register — what your business is about

• status: a list of possible states of a job: new, processing, done, failed. Oh believe me, you will use this field a lot.

• error_message: optional field for more info when task processing failed: out of space, credit or time will be visible here.

• created_at: timestamp when the item was queued, it set once and never changes
• updated_at: timestamp when the item was touched or processed, it will change at list couple of times
• agent_id: only if you have more then one script that processes the queue (covered in 2nd part of this article)

Step 2. Implement your Queue Adapter.

<?php
class Queue {

 public function addItem($item) {

  $query = 'INSERT INTO (status, data, created_at) my_queue VALUES ("new", "' . json_encode( $item ) . '", NOW())';
  sql_run_query($query);
 }
};

?>

Step 3. Find in Your Code where the Heavy Operation is (sending the email):

<?php
 //this line is the heavy operation and can take up to 10 seconds of waiting since our mail server is slow
 Mail::send(array(
  'to' => 'user@server.com',
  'subject' => 'Thanks for your purchase',
  'body' => '<b>Thank your very much, here are the details...</b>'
 )); 
 echo 'Thanks for your purchase!';
?>

Step 4. Queue the Heavy Operation

<?php
 //this is now a super-fast operation since it's just a database INSERT, no real email is sent now
 $QueueManager->addItem(array(
  'to' => 'user@server.com',
  'subject' => 'Thanks for your purchase',
  'body' => '<b>Thank your very much, here are the details...</b>'
 )); 
 echo 'Thanks for your purchase!';
?>

Conclusion

So, as you can see, working with queues if very simple and can make your life easier.

In next chapter I will show how to process items and how to divide work between many workers.

If you liked this article or you have question regarding creating queue or queuing tasks, post a comment here.