When I try to rebuild the sitemap it starts with the process, goes through all the 14 thousand odd links and gets hung with 4/7 pending tasks. Even after waiting for a very long time it was still waiting at that position. I could see in firebug that there was a post to the url every second but there was no change in the progress bar. I finally went to the sitemap page and there was this warning about the sitemap not correctly built. I then went to the rebuild page again and rebuit the sitemap without xmlsitemap and xmlsitemap_node unselected. This time it started up with 3 tasks and finished creating the sitemap successfully.

Comments

anoopjohn’s picture

anoopjohn commented

I set up the complete site on my local machine today and the rebuild went through completely on the local machine. I still have the same issue on the webserver though. Would this problem have to have some issues with file permissions?

Anonymous’s picture

Anonymous (not verified) commented

Not file permissions, you're already creating files. The issue maybe more related to the number of modules you have installed. Or what those modules are doing. Do you find an error in your php log file? Do you find an error in your mysql log file? Do you find an entry in the watchdog log (admin/reports/dblog)?

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented
Status: Active » Postponed (maintainer needs more info)

Hmm, sounds like it caught on the xmlsitemap_node part. Anything in your watchdog or server error longs to help us out?

anoopjohn’s picture

anoopjohn commented

Here are some of the errors that I found.

1) 

Type	access denied
Date	Thursday, September 24, 2009 - 17:42
User	Anonymous
Location	http://www.benzinga.com/batch?op=start&id=24
Referrer	
Message	batch
Severity	warning
Hostname	66.249.71.208
Operations

2) A tonne of following menu router errors. 

Type	php
Date	Thursday, September 24, 2009 - 17:54
User	Anonymous
Location	http://www.benzinga.com/bzstocks/js/ig
Referrer	http://www.benzinga.com/16355/ace/top-traded-stocks-f-crbc-intc-amr-xlnx
Message	Duplicate entry 'newsletter/subscriptions' for key 1 query: INSERT INTO menu_router (path, load_functions, to_arg_functions, access_callback, access_arguments, page_callback, page_arguments, fit, number_parts, tab_parent, tab_root, title, title_callback, title_arguments, type, block_callback, description, position, weight, file) VALUES ('newsletter/subscriptions', '', '', 'user_access', 'a:1:{i:0;s:24:\"subscribe to newsletters\";}', 'drupal_get_form', 'a:1:{i:0;s:36:\"simplenews_subscription_manager_form\";}', 3, 2, '', 'newsletter/subscriptions', 'Manage newsletter subscriptions', 't', '', 4, '', '', '', 0, 'sites/all/modules/simplenews/simplenews.subscription.inc') in /home/benzinga/benzinga.com/includes/menu.inc on line 2423.
Severity	error
Hostname	75.208.248.251
Operations

I had not taken the site offline while rebuilding the sitemap. I also have cron running every 2 minutes.
There were no php errors in the apache error log.

I then rebuild the sitemap with the xmlsitemap and xmlsitemap_node unselected in the select box in the rebuild page. The following error was recorded twice.

Type	access denied
Date	Thursday, September 24, 2009 - 18:12
User	Anonymous
Location	http://www.benzinga.com/batch?op=start&id=25
Referrer	
Message	batch
Severity	warning
Hostname	66.249.71.208
Operations
anoopjohn’s picture

anoopjohn commented

@Dave - Any suggested watchdog calls to debug this issue?

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented

Other people have reported a very similar issue with getting an access denied when running a batch process. It usually means that your database went down or away for some reason. You might want to give #434032: Batch update fails with access denied message a read and check if tweaking your database's configuration file helps.

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented

Marked #599710: Rebuild Sitemap stuck at 66% Remaining 4 of 9 as a duplicate of this issue.

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented

Can anyone confirm you've still had this problem with the latest code at all? If not, feel free to mark the issue as fixed.

anoopjohn’s picture

anoopjohn commented

I have not rebuilt my sitemap on the live site since then. I had updated to version 6.x-2.0-unstable5. I have just tested the rebuilding on the development server with XML sitemap 6.x-2.0-unstable5 and with 50000 nodes and it completed without any errors. I however found that the watchdog call did not have the time output. Created a new issue for that.

So the issue does not exist for me anymore and this can be marked as fixed.

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented
Status: Postponed (maintainer needs more info) » Fixed

Thank for reporting back anoopjohn. Much appreciated.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

anoopjohn’s picture

anoopjohn commented

Ok I have tried to rebuild the sitemap and hit a similar issue again. Only that the processing gets stopped while still processing the nodes. However I have a feeling that the issue could be with mod_fcgid used on the server which is throwing internal server errors once in a while. I will post back if it was indeed an xmlsitemap issue or the server issue. BTW I was curious as to why the sitemap cannot be rebuilt partially over several crons?

Anoop

anoopjohn’s picture

anoopjohn commented

My new issue is more like #720878: Problems running cron to build the sitemap on large sites. I have posted more information about the issue at that thread