Just did the beta2 upgrade in prod - the migrate form never delivers. It doesn't seem to be eating ram or CPU on the server, I suspect this is an issue with DB indexes or at least within MySQL.

CommentFileSizeAuthor
#5 migratenewtab.png125.15 KBmig5
#4 migrate.png109.23 KBmig5
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

anarcat’s picture

To give you an idea: I started reporting this bug after i opened the "migrate" link in a new tab, and it's not yet rendered (still Loading).

This may be browser related - I'm using: Chromium 8.0.552.224 (68599) Ubuntu 10.04.

anarcat’s picture

For reference, I have 178150 rows in the hosting_package_instance table...

j0nathan’s picture

Same thing with Iceweasel 3.5.8 on Debian.
It doesn't load the site migrate form. But it loads the platform migrate form.

Anonymous’s picture

FileSize
109.23 KB

The migrate form Just Works for me on a fresh install of HEAD. screenshot attached

Chrome 7.0.517.44 under Ubuntu 10.04

Anonymous’s picture

FileSize
125.15 KB

Also works for me when opening in new tab (screenshot attached)

anarcat’s picture

I think we're suffering from the dreaded case of "zombie entries in the hosting_package_instance" table... Nevertheless, I think our current approach of creating tmp tables for comparison (see hosting_package.instance.inc) doesn't scale well with huge numbers of packages, and we need indexes. We've experimented with adding indexes to those tables with good results, and will patch soon.

bgm’s picture

While debugging with Anarcat we found that adding the following keys improves performance a lot:

alter table hosting_package_instance add key hosting_package_id_key (package_id);
alter table hosting_package add key hosting_package_nid_key (nid);
alter table hosting_package_instance add key hosting_package_rid_key (rid)

After this, a query that was taking 8 seconds went to 0.01 sec. The actual migrate page in the Aegir frontend went from over 2 minutes to about 15 seconds.

The bottleneck was found using Xdebug, which lead to _hosting_package_temporary_table(), called from hosting_package_comparison().

Here is more information on why those keys are necessary:

mysql> explain SELECT nid, short_name, old_short_name, version_code, schema_version, version, status from hosting_package p left join hosting_package_instance  i on p.nid = i.package_id where rid= 4603 ;
+----+-------------+-------+------+---------------+------+---------+------+--------+--------------------------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows   | Extra                          |
+----+-------------+-------+------+---------------+------+---------+------+--------+--------------------------------+
|  1 | SIMPLE      | p     | ALL  | NULL          | NULL | NULL    | NULL |   2070 |                                | 
|  1 | SIMPLE      | i     | ALL  | NULL          | NULL | NULL    | NULL | 182496 | Using where; Using join buffer | 
+----+-------------+-------+------+---------------+------+---------+------+--------+--------------------------------+

mysql> alter table hosting_package_instance add key hosting_package_id_key (package_id);
Query OK, 182508 rows affected (1.30 sec)
Records: 182508  Duplicates: 0  Warnings: 0

mysql> alter table hosting_package add key hosting_package_nid_key (nid);
Query OK, 2070 rows affected (0.41 sec)
Records: 2070  Duplicates: 0  Warnings: 0

mysql> explain SELECT nid, short_name, old_short_name, version_code, schema_version, version, status from hosting_package p left join hosting_package_instance  i on p.nid = i.package_id where rid= 4603 ;
+----+-------------+-------+------+-------------------------+-------------------------+---------+------------------------------+--------+-------------+
| id | select_type | table | type | possible_keys           | key                     | key_len | ref                          | rows   | Extra       |
+----+-------------+-------+------+-------------------------+-------------------------+---------+------------------------------+--------+-------------+
|  1 | SIMPLE      | i     | ALL  | hosting_package_id_key  | NULL                    | NULL    | NULL                         | 182508 | Using where | 
|  1 | SIMPLE      | p     | ref  | hosting_package_nid_key | hosting_package_nid_key | 4       | aegirkoumbitnet.i.package_id |      1 | Using where | 
+----+-------------+-------+------+-------------------------+-------------------------+---------+------------------------------+--------+-------------+
2 rows in set (0.00 sec)

mysql> alter table hosting_package_instance add key hosting_package_rid_key (rid);Query OK, 182508 rows affected (2.18 sec)
Records: 182508  Duplicates: 0  Warnings: 0

mysql> explain SELECT nid, short_name, old_short_name, version_code, schema_version, version, status from hosting_package p left join hosting_package_instance  i on p.nid = i.package_id where rid= 4603 ;
+----+-------------+-------+------+------------------------------------------------+-------------------------+---------+------------------------------+------+-------------+
| id | select_type | table | type | possible_keys                                  | key                     | key_len | ref                          | rows | Extra       |
+----+-------------+-------+------+------------------------------------------------+-------------------------+---------+------------------------------+------+-------------+
|  1 | SIMPLE      | i     | ref  | hosting_package_id_key,hosting_package_rid_key | hosting_package_rid_key | 4       | const                        |  490 | Using where | 
|  1 | SIMPLE      | p     | ref  | hosting_package_nid_key                        | hosting_package_nid_key | 4       | aegirkoumbitnet.i.package_id |    1 | Using where | 
+----+-------------+-------+------+------------------------------------------------+-------------------------+---------+------------------------------+------+-------------+
2 rows in set (0.00 sec)

matt / bgm

anarcat’s picture

Status: Active » Needs review
Issue tags: +optimization

I have a fix waiting in the prod-koumbit branch, which does the following:

ALTER TABLE {hosting_package_instance} ADD INDEX hosting_package_id_idx (package_id)
ALTER TABLE {hosting_package_instance} ADD INDEX hosting_package_rid_idx (rid)
ALTER TABLE {hosting_package} ADD INDEX hosting_package_nid_idx (nid)

... in hosting_package.install. I am now running this in production with good results.

http://git.aegirproject.org/?p=hostmaster.git;a=shortlog;h=refs/heads/pr...

Right now it's a mish-mash of three commits because I screwed up the update, but I'll squash this in a single commit in a merge, once I get a go here.

joestewart’s picture

I've had this running for quite awhile and it made a huge difference.

ALTER TABLE {hosting_package_instance} ADD INDEX hosting_package_rid_idx (rid)

Definitely welcome changes. Have not tried the other two yet though. thanks.

Anonymous’s picture

I haven't taken a close look at this, but I saw unconed in IRC whining about the same slowness / load deadlocks, and he pastebinned this suggestion

http://pastebin.com/cZ1yuc7i

Not sure if there's anything new here or not

Anonymous’s picture

Does this commit have a typo?

http://git.aegirproject.org/?p=hostmaster.git;a=commitdiff;h=7cccc62e924...

index a057722..4282556 100644 (file)
--- a/modules/hosting/package/hosting_package.install
+++ b/modules/hosting/package/hosting_package.install
@@ -43,6 +43,7 @@ function hosting_package_schema() {
       ),
     ),
     'primary key' => array('vid'),
+    'indexex' => array('hosting_package_nid_idx', array('nid')),
   );
anarcat’s picture

Indeed there is... Fixed!

Anonymous’s picture

Status: Needs review » Reviewed & tested by the community

I am seeing an improvement here even in my small (for some reason!) package tables., with those indexes above

Please do merge it in before rc, and maybe see if any of unconed's there are worthwhile?

anarcat’s picture

Status: Reviewed & tested by the community » Fixed

merged the commit on the master branch at last.

Status: Fixed » Closed (fixed)
Issue tags: -optimization

Automatically closed -- issue fixed for 2 weeks with no activity.

  • Commit 81be8c3 on 6.x-2.x, 7.x-3.x, dev-ssl-ip-allocation-refactor, dev-sni, dev-helmo-3.x by anarcat:
    #1033072 add index on hosting_package tables
    
  • Commit 5406d3d on 6.x-2.x, 7.x-3.x, dev-ssl-ip-allocation-refactor, dev-sni, dev-helmo-3.x by anarcat:
    #1033072 add index on hosting_package tables
    

  • Commit 81be8c3 on 6.x-2.x, 7.x-3.x, dev-ssl-ip-allocation-refactor, dev-sni, dev-helmo-3.x by anarcat:
    #1033072 add index on hosting_package tables
    
  • Commit 5406d3d on 6.x-2.x, 7.x-3.x, dev-ssl-ip-allocation-refactor, dev-sni, dev-helmo-3.x by anarcat:
    #1033072 add index on hosting_package tables