Re-start for this after #1492188: Update module creates duplicate queue items drifted back to being a workaround fix for update.module.

Right now, update.module does it's own unique checks based on cache entries in cache_update. This should be a feature provided by the queue system itself, either by providing an actual unique queue or by providing wrapper functions to deal with this.

API suggestion by @beejebus:

function drupal_unique_queue_item_create(QueueInterface $queue, $data, $key) {
  $status = db_merge('unique_queue_item')
    ->key(array('key' => $key))
    ->insertFields(array(
      'created' => REQUEST_TIME,
    ))
    ->updateFields(array(
      'updated' => REQUEST_TIME,
    ))
    ->execute();
  if ($status === DrupalCoreDatabaseMerge::STATUS_INSERT) {
    $queue->createItem($data);
  }
}

This is also relevant for #1547008: Replace Update's cache system with the (expirable) key value store which aims to re-unite update.module with the cache API, which is not possible right now due to these fetch_task entries. Not sure if it's the right approach though, maybe yet another API for the project information data might make more sense for that. But that isn't possible either until we this API here.

Comments

gielfeldt’s picture

Would it be better to encapsulate this into the Queue interface (making it an abstract class instead), so that all queue backends might take advantage of their own unique-handling if applicable?

<?php
abstract class QueueInterface { // <= should probably be renamed ...
  abstract public function createItem($data);
  abstract public function numberOfItems();

  ...

  public function createUniqueItem($key, $data) {
    // Shouldn't the unique item be per queue?
    $status = db_merge('unique_queue_item')
      ->key(array('key' => $key))
      ->insertFields(array(
        'created' => REQUEST_TIME,
      ))
      ->updateFields(array(
        'updated' => REQUEST_TIME,
      ))
      ->execute();
    if ($status === DrupalCoreDatabaseMerge::STATUS_INSERT) {
      return $this->createItem($data);
    }
    return FALSE;
  }

  public function deleteItem($item) {
    // Should this remove the "lock" from "unique_queue_item"?
  }
}

?>
pounard’s picture

The code above is wrong in many ways the most visible one being that this stub implementation hardwires db_merge(): different queue backends are not supposed to be hardwired to any storage backend.

But there is a very revelant comment inside: Shouldn't the unique item be per queue?. I think for the same of API consitency there is two parameters here:
1. The item can be unique only on a per queue basis (I have the right the set the key "2" for example in two different queues even if it may remain unique in each one of those);
2. I think that the "unique" property should be determined at queue level (either by creating a new interface, either by adding it as isUnique() or whatever getters/setters) so that when creating a new queue, exceptions such as UnsupportedQueueCapaticityException("unique") can be thrown for backends that don't support it.

gielfeldt’s picture

Granted, of course the stub should NOT be storage backend aware.

edited :-)

catch’s picture

We can't force queue storage backends themselves to implement uniqueness, it won't work with a 'proper' queue runner like beanstalkd at all - this was discussed in the original update module issue. So keeping the queue API more or less at it is and adding a separate layer on top seemed like the only way to do this - then you can run any queue runner, and if you're using redis and want queues and queue tracking in the same storage you can.

Possible we could have this new API extend from the queue interface though, I don't think there's any particular reason it shouldn't.

ottawadeveloper’s picture

So, last year I needed a queue that could be optionally provide the functionality of a Unique and Priority queue, in addition to a standard queue - since this was all up in the air at the time, I decided to build one that would work for now and integrate with whatever was built later on.

Since that seems to have not gone anywhere, would it be worth contributing my code? I've sandboxed it for you here: http://drupal.org/sandbox/ottawadeveloper/1948226

It provides a new static method (UniqueQueue:get()) for building queues that support both unique items and prioritized items (with both being optional) and provides a default DB-based implementation of an abstract class that could be extended to fulfill any data storage method necessary. The interface is very similar to the current queue interface (there are a few things I haven't implemented yet, like releaseItem() or createQueue() both of which should be done), however it overloads the methods with additional parameters for the unique and priority functionalities.

I chose to write my own full implementation that relies on a new queue table with the proper values as the alternative seemed to be altering the core table (which I try to avoid) or doing a join with a table (more expensive) and it also seemed difficult to integrate with the generic framework. It would be possible to create a version likely that relies on a regular queue as well (either by doing the join or extending the queue table) as a wrapper around a DrupalQueueInterface object if people feel that's preferable, but it makes sense to me that the storage requirements are different enough to completely reimplement the storage methods in a unique framework for performance reasons.

As I'm going to need this for some other projects I have on the go, I'm perfectly happy to commit to maintaining it if there is interest :).

catch’s picture

Version: 8.0.x-dev » 8.1.x-dev
Issue summary: View changes

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.0-beta1 was released on March 2, 2016, which means new developments and disruptive changes should now be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.0-beta1 was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

akalata’s picture

For D8 there is also queue_unique

jonathanshaw’s picture

Advanced queue is also implementing an api for this: #2918866: Add support for unique jobs

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

solideogloria’s picture

namespace Drupal\my_class\Queue;

use Drupal\Core\Queue\DatabaseQueue;

/**
 * Custom queue implementation to merge on creation, preventing duplicates.
 *
 * @ingroup queue
 */
class MergeDatabaseQueue extends DatabaseQueue {

  /**
   * {@inheritdoc}
   */
  protected function doCreateItem($data): ?int {
    // This is copied and modified from DatabaseQueue::doCreateItem().
    // Merge instead of insert to prevent duplicates. An item is updated if it
    // already exists.
    $serialized = serialize($data);
    $query = $this->connection->merge(static::TABLE_NAME)
      ->keys([
        'name' => $this->name,
        'data' => $serialized,
      ])
      ->fields([
        'name' => $this->name,
        'data' => $serialized,
        // Note that this will update the 'created' field for an existing item.
        // We cannot rely on REQUEST_TIME because many items might be created
        // by a single request which takes longer than 1 second.
        'created' => \Drupal::time()->getCurrentTime(),
      ]);
    return $query->execute();
  }

}