Current status : cron fails on 4 releases due to 404s:

Started by timer
Building remotely on www2 in workspace <http://util.drupal.org:8080/job/localize.d.o_cron/ws/>
[localize.d.o_cron] $ /bin/sh -xe /tmp/hudson8108107103690373216.sh
+ export PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/bin
+ PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/bin
+ export TERM=dumb
+ TERM=dumb
+ drush='drush -r /var/www/localize.drupal.org/htdocs -l localize.drupal.org'
+ drush -r /var/www/localize.drupal.org/htdocs -l localize.drupal.org core-cron -y --verbose
Initialized Drupal 6.23 root directory at                               [notice]
/var/www/localize.drupal.org/htdocs
Initialized Drupal site localize.drupal.org at sites/default            [notice]
WD l10n_drupal: Unable to download and save                              [error]
http://ftp.drupal.org/files/projects/bigdaddy-7.x-1.5.tar.gz file
(404 Not Found).
WD l10n_drupal: Unable to download and save                              [error]
http://ftp.drupal.org/files/projects/bigdaddy-6.x-1.7.tar.gz file
(404 Not Found).
WD l10n_drupal: Unable to download and save                              [error]
http://ftp.drupal.org/files/projects/ip_geoloc-7.x-1.2.tar.gz file
(404 Not Found).
WD l10n_drupal: Unable to download and save                              [error]
http://ftp.drupal.org/files/projects/busy-7.x-1.6.tar.gz file (404
Not Found).
PHP Fatal error:  Allowed memory size of 419430400 bytes exhausted (tried to allocate 198130752 bytes) in /var/www/localize.drupal.org/htdocs_bzr/includes/common.inc on line 550
Drush command terminated abnormally due to an unrecoverable error.       [error]
Error: Allowed memory size of 419430400 bytes exhausted (tried to
allocate 198130752 bytes) in
/var/www/localize.drupal.org/htdocs_bzr/includes/common.inc, line 550
WD cron: Cron run exceeded the time limit and was aborted.             [warning]
Build step 'Execute shell' marked build as failure

Here's a patch that will continue to parse even if errors occur. Though in the report PHP hits its time limit.

CommentFileSizeAuthor
#11 1863554-ldo_cron_failing_404-11.patch1.97 KBs_leu
#9 1863554-ldo_cron_failing_404-9.patch1.96 KBs_leu
#4 1863554-ldo_cron_failing_404-4.patch1.92 KBSebCorbin
#2 1863554-ldo_cron_failing_404.patch1.88 KBSebCorbin
ldo-cron-404.patch1.6 KBSebCorbin

Comments

gábor hojtsy’s picture

gábor hojtsy
he/him
Primary language Hungarian
Location Hungary
commented

Look at the code, if goal is 5 but 2 have errors, then it would go back and query yet another 5 releases, all of which it would parse, so it would parse 8 instead of 5, no?

Also I guess the real disturbing problem is that it takes 400M memory to look up 4 releases in the first place. Don't really know if this is due to them 404-ing but that seems unlikely.

SebCorbin’s picture

SebCorbin commented
StatusFileSize
new 1863554-ldo_cron_failing_404.patch1.88 KB

Yup, I didn't read the db_query_range doc, modified parameters and indentation.

Also looked up through the code, there's no sense in taking so much memory...

gábor hojtsy’s picture

gábor hojtsy
he/him
Primary language Hungarian
Location Hungary
commented

This looks like would go to an infinite loop at the end of the queue. It would try to query for new releases and be unsuccessful with it.

SebCorbin’s picture

SebCorbin commented
StatusFileSize
new 1863554-ldo_cron_failing_404-4.patch1.92 KB

It's really annoying not being able to test this locally :(

gábor hojtsy’s picture

gábor hojtsy
he/him
Primary language Hungarian
Location Hungary
commented
Status: Needs review » Reviewed & tested by the community

That looks good to me now.

SebCorbin’s picture

SebCorbin commented
Status: Reviewed & tested by the community » Fixed

Pushed in 6.x-3.x, build is launched to pull changes in ldo

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

s_leu’s picture

s_leu commented
Version: 6.x-3.x-dev » 7.x-1.x-dev
Status: Closed (fixed) » Patch (to be ported)

This hasn't been ported to 7.x-1.x yet.

s_leu’s picture

s_leu commented
Status: Patch (to be ported) » Needs review
StatusFileSize
new 1863554-ldo_cron_failing_404-9.patch1.96 KB

Reroll of the patch for 7.x-1.x

berdir’s picture

berdir
Primary language German
Location Switzerland
commented
Status: Needs review » Needs work 
+++ b/connectors/l10n_drupal/l10n_drupal.moduleundefined
@@ -32,23 +32,31 @@ function l10n_drupal_cron() {
+        if (empty($return['error'])) {
+          $success;
+        }
+        $offset++;

$success is missing the increment.

$offset doesn't exist in the patches above but has been introduced in a later commit:

commit 1d30cbbb6ff6e474d8119ad385c594b8bc18b18b
Author: Sebastien Corbin <seb.corbin@gmail.com>
Date:   Tue Dec 11 15:06:08 2012 +0100

    Previous commit needed an offset
s_leu’s picture

s_leu commented
Status: Needs work » Needs review
StatusFileSize
new 1863554-ldo_cron_failing_404-11.patch1.97 KB

fixed that missing incrementation for $success

SebCorbin’s picture

SebCorbin commented
Status: Needs review » Fixed

Committed, thanks!

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.