cron not running on *.drupal.org

not running for any domain in fact. Will be fixed tomorrow, I'll run some manually for now.

Is this still a problem for api.d.o?

this does not match what's in CVS: http://api.drupal.org/api/function/db_result/6

and the change was committed on Aug 11 (4 days ago)

Is this still a problem for api.d.o?

this does not match what's in CVS: http://api.drupal.org/api/function/db_result/6

and the change was committed on Aug 11 (4 days ago)''

No, the cron jobs appear to be working correctly. The correct, updated file is in place:

drupal2 HEAD # grep Id /var/www/api.drupal.org/src/HEAD/includes/database.mysql.inc 
// $Id: database.mysql.inc,v 1.78 2007/08/12 15:55:35 dries Exp $
drupal2 HEAD # grep -A7 'function db_result' /var/www/api.drupal.org/src/HEAD/includes/database.mysql.inc 
function db_result($result) {
  if ($result && mysql_num_rows($result) > 0) {
    // The mysql_fetch_row function has an optional second parameter $row
    // but that can't be used for compatibility with Oracle, DB2, etc.
    $array = mysql_fetch_row($result);
    return $array[0];
  }
  return FALSE;

And the cron.php job is running every half-hour (with no entries in either squid cache) with this output consistently:

Removing ...<br />Removing ...<br />Removing ...<br />

Well something is broken (either cron or the API module) since the wrong code is still showing:

http://api.drupal.org/api/function/db_result/6

Trying to resolve, see http://drupal.org/node/163972 and http://drupal.org/node/172490 for more comments.

Narayan spent some time looking into whether this was cron-related when following up to 172490. His response went only to the mailing list:

I looked into this a bit today because I thought it might be releated to
me moving the cvs up from drupal2 to drupal1. However, the parsing of
the files is being triggered from the updated mtimes (I looked in the
log to verify this and also touched one of the files to make sure that a
Parse was inserted into the watchdog tables). I also checked to make
sure that the permissions were correct to the file in question and that
also seems fine (and that shouldn't have changed since the api cron has
run on drupal2 for quite awhile now).

Basically I got nowhere :). Seems like it gets to api_parse_php_file,
and just doesn't find anything new...hell if I know why.

Oh and the really genius thing is that one run did work. At 070902
20:00:12 UTC comment_new_page_count was inserted into api_documentation.

So, that rules out cron.

Next, Squid might have been caching these pages. Here's the Squid cache entry:

EY 0DA5AEC1FAA49EEC28D9C76692371823
	GET http://api.drupal.org/api/function/db_result/6
	STORE_OK      IN_MEMORY     SWAPOUT_DONE PING_DONE   
	REVALIDATE,CACHABLE,DISPATCHED,VALIDATED
	LV:1188787876 LU:1188787876 LM:1186564452 EX:280299600
	0 locks, 0 clients, 1 refs
	Swap Dir 0, File 0X00345E
	inmem_lo: 0
	inmem_hi: 2702
	swapout: 2702 bytes queued

So, the last-viewed time is the same as the last-updated time, with an expire time in 1970. Which means Squid goes back to the source (Apache) each page-view. Furthermore, I've verified by watching the Apache logs that each page-view hits Apache.

So, it must be within the webapp. Neil Drumm points out that it works for him when logged in (http://drupal.org/node/172490#comment-298784). Narayan confirmed that the correct data is being inserted into the DB by the API cron.php run, so somehow the webapp must be pulling wrong data from somewhere. Could it be that memcache is storing this data?

I don't think memcache is running on api.drupal.org. I did notice that nothing in API module clears the Drupal cache. I will add something for that and update this issue once the site is running that code.

As another example, changes I committed to the developer docs 15 hours ago still do not show right now.

http://cvs.drupal.org/viewvc.py/drupal/contributions/docs/developer/hook...

It looks to me like cron is running fine, its just not completing. The watchdog tables for at least api and drupal.org show huge numbers of 'attempt to start cron while its still running' and 'cron has been running for six hours' statements, but no cron completes.

On the other hand, groups.drupal.org seems to be completing fine.

api and drupal.org should be updated now. I still have very little idea what is actually going on, but have found a couple interesting things.

A. cron_semaphore is not removed from the cache correctly when the cron php shutdown function is called. This prevents cron from running correctly the next time around and generally makes this whole thing quite difficult to debug. I don't know why this is happening, nothing looks obviously wrong with the code.

B. The cron is dieing in the update_index hook of the node module. It dies on the search_index function call. To make life better, while I was debugging this things suddenly started working. The drupal.org and api crons started completing. Unless someone else did something, I got no clue why this happened. I do have some extra watchdog lines in drupal.org's node update_index hook now so that maybe I can have some more information the next time this fails.

In case someone wants to see where cron died and how long it takes various modules to complete the cron hook, there is a patch in the queue about making cron more granular and informative.

Regardless of whether it goes in core or not, it can be a valuable troubleshooting aid.