The queue daemon keeps dying recently, on a squeeze system (php 5.3, drush 7.x-5.x)

Restarting queue daemon with /usr/bin/php /var/aegir/drush/drush.php --php='/usr/bin/php' --php-options=''\'''\''\'\''

Calling proc_open(''\''/usr/bin/php'\''' ''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\''\'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\''\'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\''\'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\''\'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\''\'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\''\'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\''\'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\''\'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\''\'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\''\'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\''\'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\' \'''\''\'\''\'\'''\'''\''\'\'''\''\'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\''\'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\''\'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\''\'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\''\'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\''\'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\''\'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'' -d magic_quotes_gpc=Off -d magic_quotes_runtime=Off -d magic_quotes_sybase=Off'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\''\'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\''\'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\''\'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\''\'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\''\'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\''\'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''

\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\''\'\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\''\'\''\'\'''\'''\''\'\'''\'''\''\'\'''\'''\'''
--verbose @hostmaster hosting-queued.
Array to string conversion hosting_queued.drush.inc:191                                                                                                                       [notice]
pcntl_exec(): Error has occured: (errno 7) Argument list too long hosting_queued.drush.inc:191                                                                             [warning]
WD hosting_queued: Could not restart the queue daemon, aborting.                                                                                                           [error]
Drush command terminated abnormally due to an unrecoverable error.

It seemd to be related to the patch that went into drush from https://drupal.org/node/1748228#comment-7659959
However reverting that did not help.

I also quickly tried the change below... no luck:

--- a/queued/hosting_queued.drush.inc
+++ b/queued/hosting_queued.drush.inc
@@ -179,7 +179,7 @@ function hosting_queued_restart($signal = NULL) {
     // extra leading spaces.
     foreach ($options as $key => $value) {
       if ($value !== TRUE) {
-        $args[] = "--$key=" . escapeshellarg($value);
+        $args[] = "--$key=" . drush_escapeshellarg($value);
       }
       else {
         $args[] = "--$key";
CommentFileSizeAuthor
#8 hosting.code_.2077793-8.patch1.57 KBhelmo
#5 hosting.code_.2077793-5.patch1.21 KBhelmo
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

ergonlogic’s picture

ergonlogic
he/him
Primary language English
Location Montréal, Québec 🇨🇦
CreditAttribution: ergonlogic commented

This could be related to #2050881: drush.php: Permission denied creating crontab, or at least share a common solution.

It looks like we're building a drush command over here too, in hosting_queued_drush_find_drush(). As with that other issue, we should probably just call drush_build_drush_command() in both cases.

helmo’s picture

helmo CreditAttribution: helmo commented

Not sure if drush_build_drush_command would help, we need an array of arguments and an executable path.

ergonlogic’s picture

ergonlogic
he/him
Primary language English
Location Montréal, Québec 🇨🇦
CreditAttribution: ergonlogic commented

Right, but Drush shouldn't be called directly (drush.php) nor via PHP (php drush.php), but rather through the appropriate shell script. This is what drush_build_drush_command() is supposed to do for us. As for an array of arguments, shouldn't we be using the Drush API? That is, drush_get_arguments() should probably be preferred over looking directly in $_SERVER['argv'], no?

Anyway, I don't know that any of that would actually solve the problem at hand. Maybe just removing the escapeshellarg() entirely would help. After all, it's intended to escape user input, whereas these options should only be coming from the system itself.

anarcat’s picture

anarcat CreditAttribution: anarcat commented
Priority: Normal » Critical

this seems like a critical problem!

helmo’s picture

helmo CreditAttribution: helmo commented
Status: Active » Needs review
FileSize
hosting.code_.2077793-5.patch1.21 KB

New log ... shortened for readability.

WD hosting_queued: Restarting queue daemon with /usr/bin/php /data/www/initfour/indrupal/src/drush/drush.php --php='/usr/bin/php'     [notice]
--php-options='-d magic_quotes_gpc="" -d magic_quotes_runtime="" -d magic_quotes_sybase="Off"  -d magic_quotes_gpc=Off -d


WD hosting_queued: Started Hosting queue daemon, waiting for new tasks                                                                [notice]
WD hosting_queued: Restarting queue daemon with /usr/bin/php /data/www/initfour/indrupal/src/drush/drush.php --php='/usr/bin/php'     [notice]
--php-options=''\''-d magic_quotes_gpc="" -d magic_quotes_runtime="" -d magic_quotes_sybase="Off"  -d magic_quotes_gpc=Off -d

I've patched up a fix for this... tested on Debian squeeze and Wheezy. It's dirty dough :(

Setting a short "Process lifetime timeout: " on /admin/hosting/queued really helps testing this.

I also found the reason for the following warning:
Array to string conversion hosting_queued.drush.inc:191

The $_ENV array contained another array in $_ENV['argv']. Thanks php for the clear warning ;)
I'm adding an unset for this as I don't think we need that part of $_ENV.

anarcat’s picture

anarcat CreditAttribution: anarcat commented

hum.. and what about the idea of using drush_build_drush_command()?

helmo’s picture

helmo CreditAttribution: helmo commented

@anarcat: If that leaves us with a cleaner solution... I'm in favor.

I would like to fix this for squeeze, but also realize that my last squeeze Aegir setup should be upgraded to wheezy fairly soon.

helmo’s picture

helmo CreditAttribution: helmo commented
FileSize
hosting.code_.2077793-8.patch1.57 KB

Darn... now I also see this in an aegir-up vagrant wheezy VM.

This new patch simplifies it a great deal... or would that create issues? portability?

anarcat’s picture

anarcat CreditAttribution: anarcat commented
Status: Needs review » Needs work

whaaat a mess! again - i really think we should use the drush_build_command() function to find out what drush is, and stop guessing around argv...

anarcat’s picture

anarcat CreditAttribution: anarcat commented

for the record, this is how hosting-dispatch gets created for the cronjob:

  $cmd = sprintf("/usr/bin/env php %s %s hosting-dispatch ", DRUSH_COMMAND, escapeshellarg(d()->name));

can't se simply use something similar?

anarcat’s picture

anarcat CreditAttribution: anarcat commented

haha this is hilarious. @helmo, your patch should just work and is awesomely simple.

we should make a function out of it and also call it in hosting-setup. i'll take a stab at it.

anarcat’s picture

anarcat CreditAttribution: anarcat commented
Status: Needs work » Fixed

i started making a function, and it turned out to be so ridiculous that I feel we don't need a function at all:

/**
 * Get the command that this process was called with
 *
 * This can be used to restart this process or be processed to find
 * the drush command.
 *
 * To properly call this again with pcntl_exec(), make sure you clear
 * $_ENV['argv'] before passing it to pcntl_exec().
 *
 * @return Array the exact set of parameters this process was called.
 */
function hosting_queued_drush_self_args() {
  return $_ENV['argv'];
}

when you got more comments than code, you know that (a) something is wrong and (b) it's probably the language's fault.

and for the record, I believe that if we have any further problems with calling drush, we should use this:

/**
 * Get the proper way to call drush again.
 *
 * Note that unlike drush_find_drush() we return an array of parts. We
 * basically return all elements of argv until we match the "drush"
 * string in one of the parameters.
 *
 * @see drush_find_drush()
 */
function hosting_queued_drush_find_drush() {
  $args = $_ENV['argv'];
  $drush = array();
  foreach ($args as $arg) {
    $drush[] = $arg;
    if (strstr($arg, 'drush')) {
      break;
    }
  }
  return $drush;
}

aaanyways. i committed and pushed your patch. yay.

marafa’s picture

marafa CreditAttribution: marafa commented

hi
does this mean the hosting queue demon is not working? the task queue just sits there doing nothing if i enable it

helmo’s picture

helmo CreditAttribution: helmo commented

Yes, eventually the amount of quotes results in an "Argument list too long" error.
restarting is a workaround.... but updating, or patching #8 is better :)

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

  • Commit aaefd60 on 6.x-2.x, 7.x-3.x, dev-sni, dev-helmo-3.x authored by helmo, committed by anarcat: 
    Issue #2077793 by helmo: Fixed Queue daemon collecting quotes.

  • Commit aaefd60 on 6.x-2.x, 7.x-3.x, dev-sni, dev-helmo-3.x authored by helmo, committed by anarcat: 
    Issue #2077793 by helmo: Fixed Queue daemon collecting quotes.