Releases block broken on project pages - doesn't show newly published security release

#1077880-4: Block cache of issue counts on project pages is not invalidated when it should be

Sorry, I missed the 2nd part of this. Agreed that's not duplicate, and yes, we need an issue open about it.

However, I don't think this bug with the release table was actually anything to do with caching. The problem was that project_release_check_supported_versions() wasn't getting called when it should have been. That's the real bug. Once I forced that to happen via drush, it all worked as expected. So, I'm guessing this is a bug in project_release somewhere...

Beta releases I added yesterday for Migrate still aren't showing up on the project page - @dww, could you try that magic drush command?

I was concerned that I had screwed it up by initially adding tags 6.x-2.x-beta1 and 7.x-2.x-beta1 instead of 6.x-2.1-beta1 and 7.x-2.1-beta1 - if the drush command doesn't work, maybe there's something to that theory...

Thanks.

@mikeryan: Nope, that's not a bug at all. That's expected behavior. See #647428: Show latest prerelease (if different from recommended release) with dev releases for a feature request about enhancing the current behavior to handle the case you're in right now.

Marked #1179726: project_release_supported_versions not updated when release is manually published as duplicate. I'm marking this as critical, since it clearly means that people will be suggested to download insecure releases even though releases with security fixes are available.

The query for that part of the download table for profile2 is:

SELECT node.nid AS nid,
   project_release_nodes.version AS project_release_nodes_version,
   project_release_nodes.nid AS project_release_nodes_nid,
   node.uid AS node_uid,
   node.type AS node_type,
   node_revisions.format AS node_revisions_format,
   project_release_nodes.update_status AS project_release_nodes_update_status,
   term_data_project_release_nodes.weight AS term_data_project_release_nodes_weight,
   project_release_nodes.version_major AS project_release_nodes_version_major,
   project_release_nodes.version_minor AS project_release_nodes_version_minor,
   project_release_nodes.version_patch AS project_release_nodes_version_patch,
   project_release_nodes.version_extra_weight AS project_release_nodes_version_extra_weight,
   project_release_nodes.version_extra AS project_release_nodes_version_extra
 FROM node node 
 LEFT JOIN project_release_nodes project_release_nodes ON node.nid = project_release_nodes.nid
 INNER JOIN project_release_supported_versions project_release_supported_versions_project_release_nodes ON project_release_nodes.pid = project_release_supported_versions_project_release_nodes.nid AND (project_release_supported_versions_project_release_nodes.major = project_release_nodes.version_major AND project_release_supported_versions_project_release_nodes.tid = project_release_nodes.version_api_tid)
 INNER JOIN term_data term_data_project_release_nodes ON project_release_nodes.version_api_tid = term_data_project_release_nodes.tid
 LEFT JOIN node_revisions node_revisions ON node.vid = node_revisions.vid
 WHERE (node.status <> 0 OR (node.uid = ***CURRENT_USER*** AND ***CURRENT_USER*** <> 0) OR ***ADMINISTER_NODES*** = 1) AND (project_release_nodes.version_api_tid in ('103', '87', '78', '79')) AND (project_release_supported_versions_project_release_nodes.supported <> 0) AND (project_release_nodes.rebuild = 0) AND (project_release_supported_versions_project_release_nodes.recommended_release = node.nid) AND (project_release_supported_versions_project_release_nodes.recommended <> 0) AND (project_release_nodes.pid = 623208)
   ORDER BY term_data_project_release_nodes_weight ASC, project_release_nodes_version_major DESC, project_release_nodes_version_minor DESC, project_release_nodes_version_patch DESC, project_release_nodes_version_extra_weight DESC, project_release_nodes_version_extra DESC

I found that the where clause that causes the problem is:

(project_release_supported_versions_project_release_nodes.recommended_release = node.nid)

select * from project_release_supported_versions where nid = 623208 limit 5;
+--------+-----+-------+-----------+-------------+----------+---------------------+----------------+-------------------------+
| nid    | tid | major | supported | recommended | snapshot | recommended_release | latest_release | latest_security_release |
+--------+-----+-------+-----------+-------------+----------+---------------------+----------------+-------------------------+
| 623208 | 103 |     1 |         1 |           1 |        1 |             1100474 |        1100474 |                       0 | 
+--------+-----+-------+-----------+-------------+----------+---------------------+----------------+-------------------------+

It appears that the recommended_release column still has the old value instead of the new 1181018 value.

I was able to work-a-round the bug for profile2, by editing+saving the release-node of beta3 again. Now it appears in the block.

That did not work for me back with l10n_install. Damien ran drush commands locally to fix the recommended release info. :|

Before that, I tried changing the "Recommended major version" to none and setting it back afterwards, but that didn't help. Maybe it was a combination of both.

I did both (and other things like editing the project page) on the l10n_install release and nothing helped. Only Damien's intervention with drush helped.

I am having the same problem with the 6.x-1.1 security release for the spam module.

fixed it manually through drush

Thinking this is the same issue here: The release page for Installation Profile grandfoo_profile shows updates available for IMCE and Token modules, but they are already updated in the .make file and packaged with the newer releases. I examined the package today and all is in order there.

Yes, just eval:

http://drupal.org/node/1179726#comment-4569012

+1 http://drupal.org/project/commons_release (1.6 should be shown)

We just hit this with http://drupal.org/project/workbench_moderation with a manual security release - now we are linking to a still insecure release. :(

FYI http://drupal.org/project/filedepot also has this problem with a security release not showing on project page.

Common element here seems to be these are all security releases. This is a pretty critical bug if we are showing insecure releases to new users.

I'm finding that the latest Workbench Moderation release is not showing up in the Update module's "Available updates" report, either--in fact, Update module reports that there are no available releases. See the attached file, available_updates.png.

However, the latest release is showing up when I run drush pm-releases workbench_moderation; see drush_pm-releases.png.

I just had to do this for role_delegation.

The recipe is:

drush php-eval 'project_release_check_supported_versions($pid, $tid, $major, $delete);'

/**
 * Verifies the data for supported release versions, and updates if necessary.
 *
 * @param $pid
 *   The project ID.
 * @param $tid
 *   The API compatibility term ID.
 * @param $major
 *   The major version of the new/modified/deleted release.
 * @param $delete
 *   Boolean to indicate if we're deleting a release of this major or not.
 *
 * @return
 *   TRUE if we updated a record in {project_release_supported_versions},
 *   otherwise FALSE (e.g. if there were no published releases on the
 *   requested branch).

The pid is the project node id.
The tid is either 103 for 7x or 87 for 6x.
The major is an integer 7.x-MAJOR.x (just enter the integer for the major version, not the whole version string itself)
The delete should almost always be false.

For role delegation release 7.x-1.1 I used:
drush php-eval 'project_release_check_supported_versions(169314, 103, 1, FALSE);'

For anyone who cares to bughunt: http://drupalcode.org/project/project.git/blob/HEAD:/release/project_rel...

FWIW, I've seen this happen 3 times in the last week, once on Mail Logger and twice for Forward. In all three cases, editing and saving the release nodes without making any changes fixed the issue and made the releases show up on the project node.

subscribe

So, one theory I just had is that this works for mr.baileys because he is not a "project administrator" on d.o while many of the people who have this issue ARE project administrators.

Another data point: after I published it, mikesteff edited his own release node and it showed up on the project page.

Currently happening now with views 6.x-3.0-rc2 it seems.

Interesting theory about the "project administrator" permission. Hrm. I'll try to investigate a bit.

So I was debugging why security releases were acting so weird in the tables, and found this...

(my reimplementation may or may not be sane, but I think ya'll get the general idea here...)

bdragon shared his debugging notes with me in irc - "on initial save it looks like array(4 => array(array(54 => '54')) and on update it looks like array(4 => array(array(0 => '54'))"

There's also the trick that it looks different when the administrator edits (i.e. the security update tag is in the form) and when a regular user edits (the security update tag is somewher else.)

The values I mentioned correspond to my local test site. For drupal.org the tid is 100 and the vid is 7. For an admin it was consistently being keyed by the tid, but for normal developers it was being keyed by tid on initial creation and then unkeyed for updates.

This was causing {project_release_nodes}.security_update to be 0 for security releases depending on who edited the release node last, which then affected the calculations for {project_release_supported_versions}...

The test won't respond because the branch is currently broken. http://qa.drupal.org/pifr/test/48308

I don't get why it's broken with all passes and 3 exceptions, but pifr thinks it is.

I know why, I just need to get around to fixing the 6.x-1.x tests and rerun the branch test.

I will do so shortly.

There we go.

Wow, awesome work bdragon! Thanks!!!

Given how long this has languished, I lean towards just committing this as-is.

However, this seems like it's just masking a deeper bug. WTF is going on where the structure of $node->taxonomy on release nodes is so fragile and changing depending on who edits and when? I'd be more okay committing this as a stop-gap if we opened another bug report about this and added a code comment about it or something.

Also, although the fact that tests are passing is a good sign, it doesn't tell us anything about if this bug itself is fixed, since there are no tests for this case. So, we probably need more testing before we deploy this, and/or ideally, a unit test that covers this bug.

Thanks again!
-Derek

WTF is going on where the structure of $node->taxonomy on release nodes is so fragile and changing depending on who edits and when

In my experience this weirdness is more about taxonomy system than nodes.

This issue kills like 10 minutes of time for 2 people very Wednesday. I'd love a shiny unicorn patch with tests, but let's not postpone improvement in search of perfect for too long.

dww: WTF is going on is that the structure is subtly different when dealing with something coming from the form values vs the automatically reconstituted version that is coming from loading the preexisting node. (which is used instead if the multiselect is locked down...)

It's just one of those things that sucks about dealing with the taxonomy array during saving (on d6 at least..)

There are something around 330 release nodes on d.o that I measured were misfiled as non security due to this bug. (give or take)

And yeah, it does need tests.

Going back to active (needs tests, deployment)
I still fail to break it after hacking on simpletests for a while, so I am happy with it and will go forward with it now.
http://drupalcode.org/project/project.git/commit/b8ad71ab82c5ec707d7b14b...

The bit of code that was disabling the vocab in the first place was found in drupalorg_project, but I can imagine other sites having similar problems after doing a similar workflow, so I think we need to go with checking the values instead of the keys anyway... (in the future once #642106: Add native support in project_release for a "Release type" vocabulary and #642110: Add native support in project_release for a "Security update" term are completed this will all be a moot point, but for now let's just cope with the fact that we can't trust the array keys...)

Deployed.

Verified that it fixes {project_release_nodes}.security_update upon editing broken nodes.

I'm not seeing latest_security_release update however. I just spotted a possibility as to why it's not updating though. Investigating....

It's getting late.

As my normal drupal.org user, touching issues is fixing {project_release_nodes} but it is not propagating to {project_release_supported_versions}.

In the attached patch, I mark where I think things aren't working. It's just a guess because my local test site is not behaving the same and I'm having trouble reproducing.

It's possible this may not actually be a problem in practice due to the way release nodes get published when an SA is published. I do not know.

Spun up a test site today and it looks like my suspicions are correct.

What is happening is that it is not checking if $node->taxonomy[$vid] is a tid or an array of tids, and then attempting to call project_release_check_supported_versions() with an array for the core version tid. This obviously doesn't work and therefore the table doesn't get updated properly.

Given that I already tested this against a full drupal.org development environment on stagingvm, I am confident that it works.

I will plan on deploying it later today.

Deployed, verified that editing a broken release fixes both tables in the db.

http://drupalcode.org/project/project.git/commit/edc8023f3008cffad1de801...

Dropping down to normal priority, just coming up with some way to test this remaining. It may be necessary to do the tests in drupalorg_project, during my hacking around with tests yesterday I was finding it impossible to reproduce because part of the bug depends on how drupalorg_project was altering stuff, but I would have to modify the test to be drupalorg_project specific....