Immutable State Status Tracker

Description: A ZF2 Module for tracking status in a multi-threaded worker environment.

• Problem description: When a PHP application needs to transition to handling background processing one may need to keep track of workloads that have multiple sub-steps. Implementing such a status tracking system can be complex and greatly increase code complexity. This module is intended to reduce the complexity in a predictable manner.
• Overall approach to the problem: Eliminate the complexity of managing 'shared state' of where workers or threads to ned to provide status about their progress.

Why is immutability important?

• The concept of _mutability_ (the ability to change variables after an object is instantiated) can lead to unpredictable results -- especially when shared storage systems are used but work is distributed across threads and/or systems. The question can become, "Which thread has the 'right version' of the status object"? The answer can quickly become, "None!"

Locking schemes have been around for a while. Isn't this just reinventing the wheel?

• Underlying data systems may or may not support pessimistic write locking. This modules takes an approach to solving lack of locking by never needing a lock to begin with.
• Pessimistic write locking can become horribly slow and requires management whenever a worker needs to update the overall status. Even if you have pessimistic write locks in place that doesn't necessarily solve the problem of who has the correct status object. We are once again Back to square one!
• Event pushing easier to comprehend and easier to diagnose than tracing a single object manipulation bug through distributed workloads.

How does this module approach the problem of managing shared state?

• A single 'job' is created where all 'components' are defined in advance. A 'component' can be thought of as a single task that needs to happen. This 'component' should be considered a 'Job' in the SlmQueue sense of the word if you are using that module.
• Component tasks (jobs in SlmQueue) push status events about a particular component whenever state changes. 'component', and 'job_id' are inputs to a given workload.
• Job Status is calculated when it is required. The current status is inferred from the aggregate of NEWEST status events for each component.

Installation

The best way to install this module is with composer. You could also just clone this down into your modules path if you so desire/require.

require: [
...
"jackdpeterson/immutable-state-status-tracker": "dev-master"
...
]

In config/application.config.php:

return array('modules' => 
    array(
        ...
        'ImmutableStateStatusTracker'
        ...
    )
);

A conceptual example to work with

1. Assume we need to perform bulk downloading and processing of images. Only after theses tasks are completed should our broader process move on to the next phase (in this case, notify the user).

   processBigImageCollectionJob (which is comprised of download Images, and resize the collection). The second subtask is to notify the user after ALL images have been downloaded and processed.

   Task #1 - Download a collection of images (say 10,000 images divided into 100 image processing blocks [100 components related to the downloadImageCollection task])

   Task #2 - resize collection of 100 images at a time [100 components related to the processImage task].

2. __Only after resizing completes__ send a notification e-mail.

To accomplish #1 is fairly straightforward ... we'll create one job with the 200 total components (100 for downloading 100 images per job, and 100 for processing 100 images in a single batch operation).

To accomplish #2 we would have some kind of process that scans the jobs submitted for their state. upon completion or failure an action the job from the watchlist could be popped off the list and some action taken.

Creating the status tracking job

protected $statusTracker;

public function __construct(StatusTrackerServiceInterface $statusTracker) {
    $this->statusTracker = $statusTracker;
}
...
public function execute() {
    $jobToTrack = $this->statusTracker->createJob(array(
            'download_images_[0-9]', // saving space here, but this would literally be 10 entries (one for each respective job)
            'resize_and_upload_images_[0-99]', // same story -- except we have 100 entries here (let's assume higher computational complexity)
            'notification_email'
        ));
        
    // divide the list out and submit the 100 downloadAndStoreImageCollection job and pass in the identifier for which task id this is
    
    // submit a status tracking job <-- recurring magic happens here ;-)
    $newJob->setContents(array(
        'status_job_id' => $job->getJobId(),
        'shard_number' => 2,
        'collection_of_images_pointer' => 'somethingUseful'
    ));
    
}
...

Adding first status to mark the job as in-progress

protected $statusTracker;

public function __construct(StatusTrackerServiceInterface $statusTracker) {
    $this->statusTracker = $statusTracker;
}
public function execute() {
    $expectedParams = array(
        'collection_of_images_list_pointer' => 'someObjectReference (e.g., REDIS key)',
        'shard_id' => 35, // this is the identifier that will be used (effectively the shard key).
        'status_job_id' => 'something provided by the previous step'
    );
 
 // ADD IN A STATUS EVENT!!
      $event = $this->statusTracker->addStatusEvent($job->getJobId(), $job->getComponents()[0], StatusEvent::STATUS_IN_PROGRESS, 'Started downloading collection:' . 
      $inputParams['shard_id']);
}

Checking the status of a job

This last step is of course up to the implementor; however, in general one would call calculateJob() and work with the last events and the overall status to decide on the most appropriate course of action.

public function execute() {
    // adding this because the queue may be very quick and we don't want to introduce lots of repeated jobs for no reason.
    sleep(120);
    $status = $this->statusTracker->calculateJob() 

    // Push a new status check event into the queue (recursively run this until it appropriately exits) DANGER!!!.
    if ($status->getOverallStatus() == CalculatedJobStatus::STATUS_IN_PROGRESS) {
     // maybe check that all downloads are done processing
     // --> Do something here (e.g., fire off the notification task)!
     
     
    }
    
    if ($status->getOverallStatus() == CalculatedJobStatus::STATUS_COMPLETED) {
     	// fire event and then remove the job?
    }
    
    
    
    
     if ($status->getOverallStatus() == CalculatedJobStatus::STATUS_FAILED) {
     	// maybe send this to a special logging facility to notify devs and collect as much data as possible?
    }
    
}

Calling statusTracker->calculateJob() returns an instance of Entity\Calculatedstatus or throws exceptions.

Calculated status contains a few variables: * (string) overallStatus, * (Entity\Job) job, * (array __componentName => Entity\StatusEvent__) This is determined by the value of the createdAt value (highest = last).

Job is the instance of Entity\Job

F.A.Q.

Does this support [MySQL, MongoDB, Disk, Network Storage, Redis, Etc?!]

• Contributions are welcome! Now, that being said ... two adapters are included by default: - Disk -- This is designed for those using networked storage [NFS, GlusterFS, Ceph, etc.] and for local testing purposes. - DoctrineORM -- This is probably the most common use case today.
• Using your own Storage adapter means that you just need to specify a different ClassName in your storage adapter configuration (see config/ISST_*.php examples). If you want to submit a new one PRs are welcome. Just make sure it has unit tests backing it!

What module do you recommend for working with said queues?

https://github.com/juriansluiman/SlmQueue - SlmQueue Module

My calculated job didn't end with the completed event even though it was the last event that I can see in the (database, filesystem, etc).

This is a known issue. The precision of status events are to the second level (1/60th of a minute). If this is a frequent occurance in your code then adding [code below] prior submitting the status event should resolve this problem out until a better solution thought up.

sleep(2);

I received a 500 internal error when calling $serviceManager->get('immutable-state-status-tracker'); ... but I can't figure out what's wrong!

• Check that you have the dependency installed (php composer.phar update) with composer
• Check that you have ImmutableStateStatusTracker defined in your config/application.config.php as a ZF2 dependency
• Now, for the StorageAdapter specific stuff ... make sure that you have the configuration (vendor/jackdpeterson/config/ISST_[pick_one].config.php) copied into config/autoload/jobStatus.global.php

• If you're still having issues, look at your syslog as the factory sends information to there including the stack trace for further diagnostic information.
  On Ubuntu run:
  $ sudo tail -f /var/log/syslog
  

I just ran the clean operation and the database table is still (xyz) Gigabytes in length but MySQL hasn't released the space. What gives?

• This is a known issue with MySQL. The workaround to this is to pause your workers (stop injecting events) for a period of time and re-create or clean up the tables. rename the tables, and then re-create them from a clean state. An alternative approach would be to truncate both the isst_status_event table along with the isst_job table. Obvously the latter is a much more destructive approach; however, it can be a quick fix to release data if needed. In the context that this was designed for . . . jobs are fairly ephemeral and not needed beyond a few days.

Questions/Comments/Contributions?

Submit an issue and/or Pull request!