Unpublished nodes appearing in sitemap.xml

After some coordinated debugging we have determined that something else is responsible and has been reported to the Drupal Security Team.

So what else was responsible - I'm still having this issue

This was happening to me with Drupal 7.8 and xml_sitemap 7.x-2.0-rc1. After we upgraded to the latest core (7.19 at the moment), I can confirm that un-published pages are no longer being added to the XML sitemap. Just FYI, it seems to be a problem with Drupal itself, so upgrade.

we have xml_sitemap 7.x-2.0-rc1 and drupal 7.23 core and it is still putting unpublished nodes in the xml site map. What can we do?

I have the same issue on my websites. Because of the mass of unpublished nodes it is not possible for me to exclude them manually. Is there something new about this issue? Thanks!

If you rebuild the Links, there are no unpublished nodes in the xml sitemap.
will the node be updated, for example the status is set to 0, the link will not be removed.
maybe the status in xmlsitemap table will not be changed.

in function xmlsitemap_node_create_link() I check for the current status of the node and pass it as $node->xmlsitemap['status'] so it will updated in table xmlsitemap.
if the status is set to 0 the node link will be removed from the sitemap after running the cronjob.

as i'm still not sure why the node link does not disappear from the sitemap if the node access status is set to 0 in xmlsitemap table, this was the only quick solution for me at the moment.

my patch is not working properly. i'll check for another solution.

So I can confirm that there is no problem with this module with unpublished nodes.
we have another issue that has nothing to do with this module.

if you update a node, the access value will be set correctly to 0 or 1 in the xmlsitemap table.
after running the xmlsitemap cronjob, the links will be updated correctly.

I'am using the current release version 7.x-2.0.

You need to run xmlsitemap_cron() in order to re-generate the sitemap files. xmlsitemap_node_cron() only nodes that haven't been indexed yet, which typically only happens if you install the module and you had nodes that already existed.

Sorry for reopen this issue.

Example:
I have 200 published nodes, i unpublish one of them, if i execute cron there are 200 nodes in the xmlsitemap, but if i go to admin/config/search/xmlsitemap/rebuild and select node and rebuild it, yes it will be 199 nodes in my xmlsitemap. But this is a problem, because I think that this should be automatic, not manually.

One solution could be add one option in select (xmlsitemap_get_status_options) called "Dependent of entity status" or something like that, this way allow exclude or include in the sitemap for specific nodes (althought I think that nobody wants a 403 in their sitemap).

Other solution could be in the xmlsitemap_node_create_link with something like that:

  if (!$node->status) {
    $settings['status'] = 0;
  } 

Implements else or simplify $settings['status'] = $node->status; the user shoulde be know that if he changes node status xmlsitemap it will be changed too. If it's published to unpublished everything is ok, but it's unpublished to published may be he don't want this change. This could be fix with another $setting:
$settings['status']
$settings['status_user']

  if (!$node->status) {
    $settings['status'] = 0;
  }
  else {
    $settings['status'] = $settings['status_user'];
  }

Other solution is implements drupal queue to clean it in each cron execution.

Other solution is don't show in the execution or in the final query.

Another day of sleuthing and I have found the cause of a whole section of my unpublished pages still appearing in sitemap.xml, which was a bit of a shock to discover:

* If you have a page connected via its URL in a D menu that is indexed by sitemap, and the menu item's XML SITEMAP setting is 'included', it will be included in the sitemap even if the menu item Enabled flag is OFF, and even if the referenced page is Unpublished.
* Under these conditions, the only time the page will not be included is if the page's own XML SITEMAP setting is Excluded.

Not good for either SEO or security. I would argue that the D menu Enabled flag is the same status as a page's Published flag. If the flag is OFF, the item should not be included.

I have a whole new site section under development, with several dozen new menu items that are currently Enabled=OFF. For now, I've had to edit every one of them manually to set XML SITEMAP to Excluded, and will have to edit every one of them manually again when the new section is ready.

@neilsky I'm not sure that opening this closed ticket is going to get anything resolved, especially as there are lots of 'unpublished nodes listed' type issues for this module. Normally making yet another ticket ensuring branch names etc are correct works better.

But, I can help with your last comment, I have recently had a similar task of changing a load of nodes to excluded and back again, I found this sub-module. It works great. https://www.drupal.org/node/2445217#comment-9690011

Going back to the conversation @jansete was having, I too am having issues with unpublished nodes tenaciously appearing in the sitemap.

Stepping through with a debugger if I take an unpublished page and edit it, xmlsitemap_node_node_update calls create link and save link. Both functions hardcode the status of the link to 1, despite having the entity's status available to it.

Line 1086 on xmlsitemap.module:

'status' => XMLSITEMAP_STATUS_DEFAULT,

Line 583 on xmlsitemap.module:

 'status' => 1,

And it seems to me that line 185 of xmlsitemap_node.module should be pulling the status of the node rather than the one from settings eg:

'status' => $settings['status'],

Maybe should be:

'status' => $node->status,

But perhaps we are not understanding what status should mean in the sitemap context vs the node context. In my mind the following query shouldn't return thousands of rows (as it does for our site):

SELECT n.nid, n.status, x.status FROM node n JOIN xmlsitemap x ON n.nid = x.id AND x.type = 'node' WHERE n.status != x.status;

It would be really helpful to fix, document, explain this issue for the community. If that has already been done somewhere it would be great to link to it here and close this issue.

@camerongreen, Just wanted to point out that the lines mentioned assigning values to 'status' are part of a += array addition which add keys with default values if they do not exist in the array. So if the value of 'status' is incorrect for you then it is either not in the array sent to these functions or already wrong before hand.

As for the meaning of status for xmlsitemap, I have no idea. I think the issue I am having has more to do with the 'access' key, so I am going to research that now.

@sgoodwin can you point out to me where xmlsitemap tries to get the status of the node, I couldn't find it, only the references above.

Also seeing this issue of unpublished nodes appearing in the sitemap.xml.

Issue from #27 https://www.drupal.org/project/xmlsitemap/issues/1371386#comment-12608996 looks like it was redirect from a published node, to an unpublished node. Looking at new issue: https://www.drupal.org/project/xmlsitemap/issues/2968055

Let's continue this in #2266933: Unpublished node still display in sitemap and leave the original issue closed.