I'm having an issue but I am not yet sure of the cause. I'll describe the workflow of how it happens. I'm currently using 6.x-2.x-dev as of 8/29 + Pressflow 6.19.92. The other relevant modules in use are Feeds and Content Access.

I am specifically dealing with nodes. The content type is controlled in that only an admin can create nodes (no user-created nodes) and no comments are in the mix. This entire node type is available to the anonymous user both through standard Drupal permissions and Content Access.

1) Let's say I am starting with 100,000 total nodes in the system before we make changes. Of those, 99,500 are published and 500 are unpublished. If I look at the XMLsitemap settings I see 100,000 available, 100,000 indexed, 99,500 visible.

2) Sometime during the day, a process runs which uses the Feeds module to create and update nodes. No nodes are deleted during this process and none are marked as unpublished. Just new nodes and CCK/taxonomy changes. Let's say we created 500 new nodes and changed 500 other nodes. So now we have 100,500 nodes, where 100,000 should be visible.

3) XMLsitemap runs via Cron at midnight. After the cron run is completed, the number of visible nodes decreases by some amount, but I can't correlate the exact figure. But after Cron, we wind up with 100,500 available, 100,500 indexed, but somewhere around 95,000 visible. During that cron run something is determining that the nodes are no longer visible and is thus dropping them from the sitemap.

4) If I head over to "Rebuild Links", it says that the sitemap is up to date and does not need to be rebuilt. Ignoring that, I rebuild it anyway. At the conclusion of that process, I now have 100,500 nodes available, 100,500 indexed, and 100,000 visible. The rebuild of the sitemap works and sees all of the nodes.

Since all of these nodes are available to the anonymous user at all times, I am thinking that perhaps some caching is implemented during the cron run that is expired during the rebuild links process. Whatever that caching is, it's not picking up on changes to nodes that took place through the Feeds process.

I haven't dug into the code in detail yet, but I thought I'd put it out there. If one of the module authors comes across this and has an idea of what to look at, I'd appreciate a pointer. Otherwise I'll just start adding more debug statements to try to zero in on the issue.

Thanks.

CommentFileSizeAuthor
#19 content Visible xmlsitemap issue.JPG100.67 KBshyamprakash
#16 content Visible xmlsitemap issue.JPG100.67 KBshyamprakash

Comments

rjbrown99’s picture

rjbrown99 commented
Title: XMLSitemap » Cron process + Caching - Nodes not marked as visible during cron run

Fixing the title. Grr.

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented

That's very odd. There's not really any caching at work in the XML sitemap side.

rjbrown99’s picture

rjbrown99 commented

Interesting. Perhaps I need to dig in to exactly how it determines if a node should be updated or not. I'm not sure if there is a difference in how this works for a cron run/update vs. how it works for a rebuild. That seems to be the key difference.

I can't make heads or tails of it either - my temp workaround was to write a cron+drush script that queries links from the xmlsitemap_sitemap table and figures out if the sitemap got smaller (since it should theoretically never get smaller, since we're only adding new nodes.) If it does get smaller, we get an e-mail notice and then recreate the sitemap.

The bigger problem at one point was when a bunch of content dropped out and Google picked up on it and expired all of those nodes out of its index.

It's worth pointing out that this also happened with 6.x-2.0-beta1, which is why I first moved to a -dev release hoping it was fixed.

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented

Yeah the key would be to figure out why stuff is dropping out. There shouldn't be anything in XML sitemap that's doing that so I would suspect something else on hook_cron().

rjbrown99’s picture

rjbrown99 commented

Thanks Dave, I'll keep digging and post back my findings.

bethhauck’s picture

bethhauck commented

This process you mentioned that makes CCK/taxonomy changes when it imports feeds - do these changes cause pathauto to change the aliases associated with the non-visible pages?

I'm just wondering if visibility has some dependency on the alias. For some reason I've noticed XML sitemap doesn't remove a page after you unpublish it, but does remove it if you go in and delete the leftover alias, too.

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented

@bethhauck: There's no dependency on aliases at all.

bethhauck’s picture

bethhauck commented

My mistake - it wasn't removing the unaliased pages after all.

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented

@bethhauck: You'll have to run cron for the links to be removed from the sitemap files.

rjbrown99’s picture

rjbrown99 commented

#6 - paths and aliases for existing nodes would not change. New nodes would get a path however. Per #7 this probably isn't an issue.

What I am trying to do now is to nail down the numbers and try to correlate them. For example, let's say we do one Feeds import of 1,000 new items and also have 500 changes. How does that impact the XML sitemap count of available, indexed, and visible after a cron run? Do they increase/decrease and by how much?

rjbrown99’s picture

rjbrown99 commented

Just thinking out loud - similar issue with feeds, specifically see comment #1: #894724: Does not appear to work with Activity module

bethhauck’s picture

bethhauck commented

I know. What I described is a separate issue, but I thought it might be a clue. Nevermind :)

crash-land’s picture

crash-land commented

I'm running the latest 6.2.dev version and I'm seeing this exact same behavior. We are also using feeds to load and update thousands of nodes. At this point for nodes it says indexed 388904 visible 323509. This is also happening with taxonomy terms where not all of them are visible even though they are indexed. Cron runs cause more node to be dropped each time it is run.

Anonymous’s picture

Anonymous (not verified) commented

Is feeds module perhaps dropping the node and then adding a new one with the same title? Is autopath module dropping an alias and adding a new one or just adding a new one without dropping an existing one? Are you using the Global Redirect module? Are you using some other module that might change the URL?

rjbrown99’s picture

rjbrown99 commented

#13 I still haven't looked into this in detail, and my workaround continues to be rebuilding the XML sitemap. That's the only method I can see to reindex the stuff touched by the Feeds module. It's far from ideal.

@DaveReid - I see you are a co-maintainer of Feeds now. Do you have any new insights to share since you are now using/maintaining both modules?

shyamprakash’s picture

shyamprakash commented
Version: 6.x-2.x-dev » 6.x-2.0-beta3
StatusFileSize
new content Visible xmlsitemap issue.JPG100.67 KB

XMLsitemap runs via Cron . After the cron run is completed, the number of visible nodes decreases by some amount, but I can't correlate the exact figure. But after Cron, we wind up with 3279available, 3277 indexed, but somewhere around 3058 visible. During that cron run something is determining that the nodes are no longer visible and is thus dropping them from the sitemap. would you please tell me region find detail about attached image

Anonymous’s picture

Anonymous (not verified) commented
Status: Active » Postponed (maintainer needs more info)

#16: Are anonymous users allowed to see the content you say is missing? The sitemap.xml report should only show links that are available for anonymous users.

shyamprakash’s picture

shyamprakash commented

Yes Earnie i have allowed to anonymous users see the content. please visit my attached file

My concern is content is Available 3279, Indexed 3277 and Visible is 3058 why ? Visible should be 3279?

shyamprakash’s picture

shyamprakash commented
StatusFileSize
new content Visible xmlsitemap issue.JPG100.67 KB

I am describing Details

I'm having an issue but I am not getting solutions. I'm currently using 6.x-2.0-beta3 module.

I am working with nodes. The content type is controlled in that only an admin can create nodes (no user-created nodes) and no comments are in the mix. This entire node type is available to the anonymous user both through standard Drupal permissions and Content Access.

1) Let's say I am starting with 3158 total nodes in the system before we make changes. Of those, 3158 are published. If I look at the XMLsitemap settings I see 3158 available, 3158 indexed, 3158 visible.

2) After 1-2 days we created 120 new nodes. So now we have 3280 nodes, where 3280 should be visible.

3) XMLsitemap runs via Cron. After the cron run is completed, the number of visible nodes decreases by some amount, but I can't correlate the exact figure. But after Cron, we wind up with 3280 available, 3280 indexed, but somewhere around 3058 visible. During that cron run something is determining that the nodes are no longer visible and is thus dropping them from the sitemap.

Thanks.

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented

Not sure I can help without seeing your database directly. Your best bet is to start looking at the {xmlsitemap} table, checking which records have type = 'node' and access = 0 and checking those node IDs.

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented