PHP Immutable State Status Tracker: Manage queue status using immutable structures

Recommend this page to a friend!
  Info   View files Documentation   View files View files (48)   DownloadInstall with Composer Download .zip   Reputation   Support forum   Blog    
Ratings Unique User Downloads Download Rankings
Not enough user ratingsTotal: 108 All time: 9,335 This week: 328Up
Version License PHP version Categories
immutable-state-stat 1.0MIT/X Consortium ...5.6PHP 5, Databases, Files and Folders, L..., D...
Description Author

This package can manage the status of queue using immutable data structures.

It is a module for the Zend Framework 2 to keep track of the status of queues and jobs.

It can use multiple backends for storing job queues. Currently it supports files, databases or other backends supported by Doctrine ORM.

Innovation Award
PHP Programming Innovation award nominee
December 2015
Number 4


Prize: One book of choice by Packt
Immutable objects are objects that cannot be changed after creation.

Immutability is useful when there are multiple threads or processes accessing the same copy of the objects. It prevents that an object be changed by code running in parallel when it is not intended to allow any changes.

This package implements immutable objects for queues of jobs thus avoiding the need to implement other types of inter-process information locking.

Manuel Lemos
Picture of Jack Peterson
  Performance   Level  
Name: Jack Peterson <contact>
Classes: 1 package by
Country: United States United States
Innovation award
Innovation award
Nominee: 1x

 

Details

Immutable State Status Tracker

Build Status

Description: A ZF2 Module for tracking status in a multi-threaded worker environment.

  • Problem description: When a PHP application needs to transition to handling background processing one may need to keep track of workloads that have multiple sub-steps. Implementing such a status tracking system can be complex and greatly increase code complexity. This module is intended to reduce the complexity in a predictable manner.
  • Overall approach to the problem: Eliminate the complexity of managing 'shared state' of where workers or threads to ned to provide status about their progress.

Why is immutability important?

  • The concept of _mutability_ (the ability to change variables after an object is instantiated) can lead to unpredictable results -- especially when shared storage systems are used but work is distributed across threads and/or systems. The question can become, "Which thread has the 'right version' of the status object"? The answer can quickly become, "None!"

Locking schemes have been around for a while. Isn't this just reinventing the wheel?

  • Underlying data systems may or may not support pessimistic write locking. This modules takes an approach to solving lack of locking by never needing a lock to begin with.
  • Pessimistic write locking can become horribly slow and requires management whenever a worker needs to update the overall status. Even if you have pessimistic write locks in place that doesn't necessarily solve the problem of who has the correct status object. We are once again Back to square one!
  • Event pushing easier to comprehend and easier to diagnose than tracing a single object manipulation bug through distributed workloads.

How does this module approach the problem of managing shared state?

  • A single 'job' is created where all 'components' are defined in advance. A 'component' can be thought of as a single task that needs to happen. This 'component' should be considered a 'Job' in the SlmQueue sense of the word if you are using that module.
  • Component tasks (jobs in SlmQueue) push status events about a particular component whenever state changes. 'component', and 'job_id' are inputs to a given workload.
  • Job Status is calculated when it is required. The current status is inferred from the aggregate of NEWEST status events for each component.

Installation

The best way to install this module is with composer. You could also just clone this down into your modules path if you so desire/require.

require: [
...
"jackdpeterson/immutable-state-status-tracker": "dev-master"
...
]

In config/application.config.php:

return array('modules' => 
    array(
        ...
        'ImmutableStateStatusTracker'
        ...
    )
);

A conceptual example to work with

  1. Assume we need to perform bulk downloading and processing of images. Only after theses tasks are completed should our broader process move on to the next phase (in this case, notify the user).

    processBigImageCollectionJob (which is comprised of download Images, and resize the collection). The second subtask is to notify the user after ALL images have been downloaded and processed.

    Task #1 - Download a collection of images (say 10,000 images divided into 100 image processing blocks [100 components related to the downloadImageCollection task])

    Task #2 - resize collection of 100 images at a time [100 components related to the processImage task].

  2. __Only after resizing completes__ send a notification e-mail.

To accomplish #1 is fairly straightforward ... we'll create one job with the 200 total components (100 for downloading 100 images per job, and 100 for processing 100 images in a single batch operation).

To accomplish #2 we would have some kind of process that scans the jobs submitted for their state. upon completion or failure an action the job from the watchlist could be popped off the list and some action taken.

Creating the status tracking job

protected $statusTracker;

public function __construct(StatusTrackerServiceInterface $statusTracker) {
    $this->statusTracker = $statusTracker;
}
...
public function execute() {
    $jobToTrack = $this->statusTracker->createJob(array(
            'download_images_[0-9]', // saving space here, but this would literally be 10 entries (one for each respective job)
            'resize_and_upload_images_[0-99]', // same story -- except we have 100 entries here (let's assume higher computational complexity)
            'notification_email'
        ));
        
    // divide the list out and submit the 100 downloadAndStoreImageCollection job and pass in the identifier for which task id this is
    
    // submit a status tracking job <-- recurring magic happens here ;-)
    $newJob->setContents(array(
        'status_job_id' => $job->getJobId(),
        'shard_number' => 2,
        'collection_of_images_pointer' => 'somethingUseful'
    ));
    
}
...

Adding first status to mark the job as in-progress

protected $statusTracker;

public function __construct(StatusTrackerServiceInterface $statusTracker) {
    $this->statusTracker = $statusTracker;
}
public function execute() {
    $expectedParams = array(
        'collection_of_images_list_pointer' => 'someObjectReference (e.g., REDIS key)',
        'shard_id' => 35, // this is the identifier that will be used (effectively the shard key).
        'status_job_id' => 'something provided by the previous step'
    );
 
 // ADD IN A STATUS EVENT!!
      $event = $this->statusTracker->addStatusEvent($job->getJobId(), $job->getComponents()[0], StatusEvent::STATUS_IN_PROGRESS, 'Started downloading collection:' . 
      $inputParams['shard_id']);
}

Checking the status of a job

This last step is of course up to the implementor; however, in general one would call calculateJob() and work with the last events and the overall status to decide on the most appropriate course of action.

public function execute() {
    // adding this because the queue may be very quick and we don't want to introduce lots of repeated jobs for no reason.
    sleep(120);
    $status = $this->statusTracker->calculateJob() 

    // Push a new status check event into the queue (recursively run this until it appropriately exits) DANGER!!!.
    if ($status->getOverallStatus() == CalculatedJobStatus::STATUS_IN_PROGRESS) {
     // maybe check that all downloads are done processing
     // --> Do something here (e.g., fire off the notification task)!
     
     
    }
    
    if ($status->getOverallStatus() == CalculatedJobStatus::STATUS_COMPLETED) {
     	// fire event and then remove the job?
    }
    
    
    
    
     if ($status->getOverallStatus() == CalculatedJobStatus::STATUS_FAILED) {
     	// maybe send this to a special logging facility to notify devs and collect as much data as possible?
    }
    
}

Calling statusTracker->calculateJob() returns an instance of Entity\Calculatedstatus or throws exceptions.

Calculated status contains a few variables: * (string) overallStatus, * (Entity\Job) job, * (array __componentName => Entity\StatusEvent__) This is determined by the value of the createdAt value (highest = last).

Job is the instance of Entity\Job

F.A.Q.

Does this support [MySQL, MongoDB, Disk, Network Storage, Redis, Etc?!]

  • Contributions are welcome! Now, that being said ... two adapters are included by default: - Disk -- This is designed for those using networked storage [NFS, GlusterFS, Ceph, etc.] and for local testing purposes. - DoctrineORM -- This is probably the most common use case today.
  • Using your own Storage adapter means that you just need to specify a different ClassName in your storage adapter configuration (see config/ISST_*.php examples). If you want to submit a new one PRs are welcome. Just make sure it has unit tests backing it!

What module do you recommend for working with said queues?

https://github.com/juriansluiman/SlmQueue - SlmQueue Module

My calculated job didn't end with the completed event even though it was the last event that I can see in the (database, filesystem, etc).

This is a known issue. The precision of status events are to the second level (1/60th of a minute). If this is a frequent occurance in your code then adding [code below] prior submitting the status event should resolve this problem out until a better solution thought up.

sleep(2);

I received a 500 internal error when calling $serviceManager->get('immutable-state-status-tracker'); ... but I can't figure out what's wrong!

  • Check that you have the dependency installed (php composer.phar update) with composer
  • Check that you have ImmutableStateStatusTracker defined in your config/application.config.php as a ZF2 dependency
  • Now, for the StorageAdapter specific stuff ... make sure that you have the configuration (vendor/jackdpeterson/config/ISST_[pick_one].config.php) copied into config/autoload/jobStatus.global.php
  • If you're still having issues, look at your syslog as the factory sends information to there including the stack trace for further diagnostic information.
    On Ubuntu run:
    $ sudo tail -f /var/log/syslog
    

I just ran the clean operation and the database table is still (xyz) Gigabytes in length but MySQL hasn't released the space. What gives?

  • This is a known issue with MySQL. The workaround to this is to pause your workers (stop injecting events) for a period of time and re-create or clean up the tables. rename the tables, and then re-create them from a clean state. An alternative approach would be to truncate both the isst_status_event table along with the isst_job table. Obvously the latter is a much more destructive approach; however, it can be a quick fix to release data if needed. In the context that this was designed for . . . jobs are fairly ephemeral and not needed beyond a few days.

Questions/Comments/Contributions?

Submit an issue and/or Pull request!

  Files folder image Files  
File Role Description
Files folder image.settings (7 files)
Files folder imageconfig (3 files)
Files folder imagesrc (1 directory)
Files folder imagetests (3 files, 1 directory)
Files folder imageview (1 directory)
Plain text file .buildpath Data Auxiliary data
Plain text file .project Data Auxiliary data
Plain text file .travis.yml Data Auxiliary data
Plain text file autoload_classmap.php Aux. Class source
Plain text file autoload_function.php Aux. Class source
Plain text file autoload_register.php Aux. Class source
Plain text file composer.json Data Auxiliary data
Plain text file LICENSE.txt Lic. License text
Plain text file Module.php Class Class source
Plain text file phpunit.xml Data Auxiliary data
Plain text file README.md Doc. Auxiliary data

  Files folder image Files  /  .settings  
File Role Description
  Plain text file .jsdtscope Data Auxiliary data
  Plain text file com.zend.php.remoteproject.core.prefs Data Auxiliary data
  Plain text file org.eclipse.php.core.prefs Data Auxiliary data
  Plain text file org.eclipse.php.formatter.core.prefs Data Auxiliary data
  Plain text file org.eclipse.php.formatter.ui.prefs Data Auxiliary data
  Plain text file org.eclipse.wst.js...superType.container Data Auxiliary data
  Plain text file org.eclipse.wst.jsdt.ui.superType.name Data Auxiliary data

  Files folder image Files  /  config  
File Role Description
  Plain text file ISST_StorageAdapter_Disk.config.php Conf. Configuration script
  Plain text file ISST_StorageAdapte...trineORM.config.php Conf. Configuration script
  Plain text file module.config.php Conf. Configuration script

  Files folder image Files  /  src  
File Role Description
Files folder imageImmutableStateStatusTracker (2 files, 7 directories)

  Files folder image Files  /  src  /  ImmutableStateStatusTracker  
File Role Description
Files folder imageController (1 file)
Files folder imageEntity (3 files)
Files folder imageException (4 files)
Files folder imageFactory (2 files)
Files folder imagePaginator (1 directory)
Files folder imageService (1 file)
Files folder imageStorageAdapter (3 files, 1 directory)
  Plain text file StatusTrackerServiceInterface.php Class Class source
  Plain text file StorageAdapterInterface.php Class Class source

  Files folder image Files  /  src  /  ImmutableStateStatusTracker  /  Controller  
File Role Description
  Plain text file ISSTController.php Class console input

  Files folder image Files  /  src  /  ImmutableStateStatusTracker  /  Entity  
File Role Description
  Plain text file CalculatedJobStatus.php Class Class source
  Plain text file Job.php Class Class source
  Plain text file StatusEvent.php Class Class source

  Files folder image Files  /  src  /  ImmutableStateStatusTracker  /  Exception  
File Role Description
  Plain text file CalculatingJobStatusException.php Class Class source
  Plain text file ConfigurationException.php Class Class source
  Plain text file InvalidArgumentException.php Class Class source
  Plain text file StorageAdapterException.php Class Class source

  Files folder image Files  /  src  /  ImmutableStateStatusTracker  /  Factory  
File Role Description
  Plain text file ImmutableStateStatusFactory.php Class Class source
  Plain text file ISSTControllerFactory.php Class console input factory

  Files folder image Files  /  src  /  ImmutableStateStatusTracker  /  Paginator  
File Role Description
Files folder imageAdapter (1 file)

  Files folder image Files  /  src  /  ImmutableStateStatusTracker  /  Paginator  /  Adapter  
File Role Description
  Plain text file DoctrinePaginator.php Class Class source

  Files folder image Files  /  src  /  ImmutableStateStatusTracker  /  Service  
File Role Description
  Plain text file StatusTracker.php Class Class source

  Files folder image Files  /  src  /  ImmutableStateStatusTracker  /  StorageAdapter  
File Role Description
Files folder imageDoctrine (2 files)
  Plain text file AbstractStorageAdapter.php Class Class source
  Plain text file Disk.php Class Class source
  Plain text file DoctrineORM.php Class Class source

  Files folder image Files  /  src  /  ImmutableStateStatusTracker  /  StorageAdapter  /  Doctrine  
File Role Description
  Plain text file Job.php Class Class source
  Plain text file StatusEvent.php Class Class source

  Files folder image Files  /  tests  
File Role Description
Files folder imageImmutableStateStatusTracker (2 directories)
  Plain text file bootstrap.php Class Class source
  Plain text file cli-config.php Example Example script
  Plain text file TestConfiguration.php.dist Conf. Configuration Script

  Files folder image Files  /  tests  /  ImmutableStateStatusTracker  
File Role Description
Files folder imageService (1 file)
Files folder imageStorageAdapter (2 files)

  Files folder image Files  /  tests  /  ImmutableStateStatusTracker  /  Service  
File Role Description
  Plain text file StatusTrackerTest.php Test Unit test script

  Files folder image Files  /  tests  /  ImmutableStateStatusTracker  /  StorageAdapter  
File Role Description
  Plain text file DiskTest.php Test Unit test script
  Plain text file DoctrineORMTest.php Test Unit test script

  Files folder image Files  /  view  
File Role Description
Files folder imageimmutable-state-status-tracker (1 directory)

  Files folder image Files  /  view  /  immutable-state-status-tracker  
File Role Description
Files folder imageskeleton (2 files)

  Files folder image Files  /  view  /  immutable-state-status-tracker  /  skeleton  
File Role Description
  Plain text file foo.phtml Data Auxiliary data
  Plain text file index.phtml Data Auxiliary data

 Version Control Unique User Downloads Download Rankings  
 100%
Total:108
This week:0
All time:9,335
This week:328Up

For more information send a message to info at phpclasses dot org.