On a fresh install of HEAD with only one 6.14 platform, I've duplicate rows of packages in hosting_package. The most obvious case of this is in the site form, as there are two 'default' install profiles listed (and listed because > 1 profile means the profile form item isn't hidden). Closer inspection reveals 2 of every package in hosting_package.
mysql> SELECT * FROM hosting_package LIMIT 20;
+-----+-----+--------------+------------+-------------+
| vid | nid | package_type | short_name | description |
+-----+-----+--------------+------------+-------------+
| 4 | 4 | platform | drupal | |
| 10 | 8 | module | aggregator | |
| 11 | 9 | module | user | |
| 12 | 10 | module | user | |
| 13 | 11 | module | system | |
| 14 | 12 | module | system | |
| 15 | 13 | module | tracker | |
| 16 | 14 | module | tracker | |
| 17 | 15 | module | comment | |
| 18 | 16 | module | comment | |
| 19 | 17 | module | help | |
| 20 | 18 | module | help | |
| 21 | 19 | module | profile | |
| 22 | 20 | module | profile | |
| 23 | 21 | module | forum | |
| 24 | 22 | module | forum | |
| 25 | 23 | module | menu | |
| 26 | 24 | module | menu | |
| 27 | 25 | module | search | |
| 28 | 26 | module | search | |
+-----+-----+--------------+------------+-------------+
20 rows in set (0.01 sec)
This might not be in Hosting, but I have no idea where this is being caused and I haven't look too closely yet, just getting it down before I forget.
Comments
Comment #1
Anonymous (not verified) commentedIt looks like 2 Verify tasks are being created for the platform on fresh install!?
The second one actually throws error but the task is 'successful'
Comment #2
Anonymous (not verified) commentedFurther testing: it is definitely instigated on the initial 'hosting setup' command or something spawned from it. Getting closer
Comment #3
Anonymous (not verified) commentedStill haven't worked out why it looks like a verify is occuring twice. Here's an excerpt from a mysql log http://drupalbin.com/11477
I just installed an 0.4 alpha1 system and the problem exists there too, so it's not a recent HEAD commit
Comment #4
Anonymous (not verified) commentedComment #5
adrian commentedcommited the fix in both hostmaster and hosting
i don't know how to fix this for existing installs yet.
Comment #6
adrian commentedputting this to needs work now till we sort out the upgrade path
Comment #7
Anonymous (not verified) commentedSo I did an install of 0.3 which did not have this problem.
Then I patched the 0.3 install with the Ports patches from #515052: Customizable port field per site and did a re-install and the duplication occurs.
Adrian has a fix to prevent two node_save()'s by adding (I think) a no_verify flag or somesuch.
Meanwhile I have been doggedly trying to understand how this could happen and I think I might've found it:
This query from just after submitting the Web Server and File System tasks in the Hostmaster install:
Maybe somehow we are triggering that
hosting_add_task($nid->nid, 'verify');, it causes two Verify tasks as seen on a fresh install, and I assume what branches from this is the duplication of packages in the node and hosting_package tables. Strange because the hosting_web_server table shows 'ports' field as being 80 (I didn't add any others)Adrian notes that this could occur in the frontend as well, in that one can create a site and then immediately create a new verify task. He's committed a fix for the hostmaster profile and to Hosting to stop the duplicate verify task, along with the complete refactoring of the UI as well
Comment #8
Anonymous (not verified) commentedAdrian beat me to the comment submission and I wiped his metachanges to the ticket.
Comment #9
anarcat commentedI feel this is fixed: if adrian fixed the duplicate task stuff, then it's fixed. Otherwise I have worked on the task interface so that tasks are revisionned properly (#422970: have one task type per site node) and that should also fix the problem.
Comment #10
Anonymous (not verified) commented@anarcat: The issue is that though the bug is fixed, anyone who's installed 0.4 alpha1 or running off HEAD has duplicate rows for every package in the node and hosting_package table. We need to write a hook_update() for the next release that somehow fixes these tables.
Comment #11
jonhattanIn my fresh 0.4 alpha 1 install those packages are not duplicated:
Those one are the duplicates:
This is distinct from mig5's dump. At least `forum`is duplicate for mig5's and not for me.
Those are all tasks that have run out here:
Comment #12
jonhattanProbably I miss something but following query results in all packages that are alone in hosting_package and are the ones to be deleted:
if that is true, following update should do the work:
Anyway this is so simple that probably I'm missing some big thing :)
Comment #13
jonhattanIn addition to `hosting_package` duplication is also in `node` and `node_revisions`.
Comment #14
jonhattanPrevious patch was wrong.
I do use raw sql to delete the nodes in order to use update_sql() and show a pretty message but probably usage of node_delete() will be better option.
there's also duplication in hosting_package_instance:
I think `rid` is the platform nid. Nodes 48, 51 and 84 are packages, not platforms. All of those package instances are also to be removed, isn't it?
Comment #15
Anonymous (not verified) commentedQuick note
Just tested this on a alpha 0.1 release where I could reproduce the bug. I get sql errors on update.php
mysql> SELECT nid FROM hosting_package WHERE nid NOT IN (SELECT package_id FROM hosting_package_instance);
Empty set (0.01 sec)
Comment #16
jonhattanmig5: your failure is because there is no duplication... I've added a condition to avoid that.
Still pending code to remove those
SELECT iid FROM hosting_package_instance WHERE rid NOT IN (SELECT nid FROM hosting_platform);as I said in #14. I still dont understand the underlying logic on how did all of this happen.Comment #17
Anonymous (not verified) commentedThe problem being I *do* have duplication, I'm looking at it :)
Here's my hosting_package and node tables.
Yes I had the two verify tasks for the first platform etc too.
Comment #18
adrian commentedjust deleting the nodes should delete the instances.
the issue is the existing nodes with wrong references, which is basically all sites pointing to the wrong profile node.
Comment #19
jonhattancurrent patch does node_delete to remove all packages, truncate tables hosting_package_{instance,languages} and schedule a reverify for platforms.
After verification duplication still exists, but in a diferente flavour (note not all of us has the same duplication issue): now I have no duplicates in hosting_package but hosting_package_languages content is:
those iid are in fact duplicates in hosting_package_instance.
I updated to HEAD prior to running update.php and result is the same.
Comment #20
univate commentedHere is a db dumps of what I see with one package that is duplicated from apha1:
http://drupalbin.com/11650
The attached patch fixed the issue for me of duplicate packages appearing in site and platform lists on the site as well as duplicates in the node and hosting_package db tables although there appears to still be issues in the database records that don't make sense to me in hosting_package_instances:
Take for example package_id#35 which is the filter module
nid#5 - platform
nid#44 - Minnelli theme
nid#80 - site
I don't understand the Minnelli theme being in this list?
Comment #21
univate commentedFrom testing #19:
It removes the duplicates from the hosting frontend site/platform package tables.
After this is run there are a couple of extra records in hosting_package_instance (which means there are also these records in the language table)
You can see the records above that don't match up with any sites or platforms.
Comment #22
jonhattanI've realized that in alpha1 the content of hosting_package_languages is 16 rows instead of 8 after patching, updating and reverifying.
So I could spect 20 results insted of 10 in
no luck. If there are 13... where are the rest? Those ones seems orphaned rows in hosting_package_languages:
So so so, what I want to conclude is that this is a separate issue. I've just installed HEAD and it's confirmed that hosting_package_languages has 8 rows that match 7 packages:
and the 8th row, you guess, is an orphan:
Comment #23
univate commentedHere is a combined patch discussed in IRC
It finds all duplicates where there are mupltiple package_instance's for each platform or site (#20) and where there are is no package_instance for a package (#16).
If there are any nid that match these:
* call node_delete() on each nid
* re-run verify on each platform.
I wonder if we should also re-run verify on each site as well.
Comment #24
Anonymous (not verified) commentedI applied this patch (with some minor {} bracket corrections and some extra code documentation) and it applied fine.. It said that it deleted a whole bunch of nodes, that's great.
But both before and after the update, I ran this:
And got 193 rows returned. So even after the update, this happened.
When the re-verify went through of the platform, I ran the query again, and now i get 282 rows in set. Is this expected behaviour?
on the other hand, everything else seems normal (i.e during duplication, or at least in one of the cases, I see two 'default' profiles in the site node form when adding a site, and after this update, this has gone.)
Comment #25
anarcat commentedThere is a separate issue about orphaned items in #596714: orphaned entries in hosting_package_instance, please disregard that issue (which is also critical as it affects 0.3) when resolving the bug here.
Comment #26
Anonymous (not verified) commentedThanks anarcat, yes I confirm this update fixes the duplication issue (orphans are for that other ticket), so I've committed this to HEAD.
Thanks most of all to jonhattan and univate who knuckled down and sorted this one out for us :) mega gratitude to you both.