Create Server as cluster is broken after save [#1016890]

When saving ("Create Server") a cluster, the web response is database password is invalid, even though the database (mysql) option was not selected.

Steps to reproduce:
1) enable "Web Clusters" in /admin/hosting/features
2) create new server (type cluster) at /node/add/server
3) Fill out "Create server" page:
3a) Server hostname: webcluster.blah.local
3b) Ip addresses: 203.1.2.3
3c) Database: None
3d) Web: cluster
3e) Servers: web01.blah.local, web02.blah.local
4) Save

Response page:
1) error: "The specified passwords do not match."
2) database options is now shown (port, username, password) even though "None" is selected.
3) web options is now shown (port, restart command, port, restart command, ssl port), even though "cluster" is selected.

Not sure if this is related:
---
Log message
Task starts processing
Running: /usr/share/drush/drush.php provision-save '@server_webCLUSTERlocal' --backend
The command could not be executed successfully (returned: Segmentation fault , code: 139)
Running: /usr/share/drush/drush.php @server_webCLUSTERlocal provision-verify --backend
Drush bootstrap phase : _drush_bootstrap_drush()
Load alias @server_webCLUSTERlocal
Found command: provision-verify (commandfile=provision)
Initializing drush commandfile: drush_make
Initializing drush commandfile: drush_make_d_o
Initializing drush commandfile: provision
Load alias @server_master
Loading apache driver for the http service
Including /var/aegir/.drush/provision/dns/verify.provision.inc
Including /var/aegir/.drush/provision/platform/backupmigrate/verify.provision.inc
Including /var/aegir/.drush/provision/platform/verify.provision.inc
Provision configuration root path /var/aegir/config exists.
Provision configuration root ownership of /var/aegir/config has been changed to aegir.
Provision configuration root permissions of /var/aegir/config have been changed to 711.
Provision configuration root path /var/aegir/config is writable.
Provision configuration path /var/aegir/config/server_webCLUSTERlocal exists.
Provision configuration ownership of /var/aegir/config/server_webCLUSTERlocal has been changed to aegir.
Provision configuration permissions of /var/aegir/config/server_webCLUSTERlocal have been changed to 711.
Provision configuration path /var/aegir/config/server_webCLUSTERlocal is writable.
Command dispatch complete
Peak memory usage was 7.96 MB
An error occurred at function : drush_hosting_task
Command dispatch complete
Peak memory usage was 21.21 MB
---

Comment	File	Size	Author
#32	1016890-provision-cluster-save-32.patch	651 bytes	joestewart
#31	1016890-provision-cluster-save-31.patch	518 bytes	joestewart
#28	1016890-web_servers-28.patch	505 bytes	joestewart
#25	1016890_cluster_debug.patch	1.38 KB	anarcat
#24	1016890_cluster_debug.patch	1.09 KB	anarcat
#23	1016890_cluster_debug.patch	877 bytes	anarcat
#4	1016890.patch	1.82 KB	mig5

Comments

Comment #1

Anonymous (not verified) commented 6 January 2011 at 08:34

Status:

Needs review

» Postponed (maintainer needs more info)

I've seen this before and it's when your browser has saved your username/password and injected that into the Database user/password fields (which are hidden because you have not selected a DB service).

You get a 'passwords do not match' because it only injects it into the first password field and not the 'confirm password' field, typical browser stuff.

You can select MySQL in the form to erase the auto-injected credentials and then set it to None again (or just don't save passwords in the browser).

Sure that's not the case for you too?

Don't know about the task debug. Segmentation fault, possibly needs an strace or something to see what's going on. Please confirm whether you hit the above browser issue and whether fixing that fixes the segfault

Comment #2

dsobon commented 6 January 2011 at 09:49

Status:

Postponed (maintainer needs more info)

» Needs review

I can confirm that removing the "saved password" in browser solved the problem.

Now to figure out why drush then segfaults.

Comment #3

Anonymous (not verified) commented 7 January 2011 at 11:06

Category:	bug	» support
Status:	Needs review	» Active

Should only be marked 'for review' if there's a patch to review. Just for filters.

The segfault should really be a separate support request, but i can't be bothered.

Comment #4

Anonymous (not verified) commented 27 January 2011 at 05:30

Category:	support	» bug
Status:	Active	» Needs review

Status	File	Size
new	1016890.patch	1.82 KB

Can you test this patch and let me know if it makes clusters work?

Comment #5

Anonymous (not verified) commented 5 February 2011 at 04:49

Status:

Needs review

» Reviewed & tested by the community

Cafuego confirmed the fix in passing in #1047174: Reverse proxy support, which matches my experience. I'll commit this

Comment #6

Anonymous (not verified) commented 5 February 2011 at 04:51

Status:

Reviewed & tested by the community

» Fixed

Comment #7

omega8cc commented 5 February 2011 at 23:03

Priority:	Normal	» Major
Status:	Fixed	» Needs work

This commit makes Provision working only with Drush 3.3 and breaks everything for any attempt (both Aegir install and upgrade) with Drush 5/HEAD.

Comment #8

Anonymous (not verified) commented 5 February 2011 at 23:31

Ok, I'll revert.

Can we get a bit more info than 'it breaks everything' though?

Comment #9

omega8cc commented 5 February 2011 at 23:46

With this commit it is simply impossible to provision or upgrade the Hostmaster site (using Drush HEAD), its drupal root is simply not created. Here is an example upgrade log with debugging enabled: https://gist.github.com/812930

BTW: I discovered it trying to reproduce my previous issue with not rewritten paths in the files table.

Comment #10

Anonymous (not verified) commented 6 February 2011 at 00:46

OK, reverted.

Someone is going to have to think of a fix that works in Drush 4. (How I wish there was never a Drush 4)

Comment #11

Anonymous (not verified) commented 6 February 2011 at 12:00

Status:

Needs work

» Fixed

I actually re-applied this after I realised what the over-arching issue was in #1047922: Drush 4.x and 5.0-dev/HEAD breaks Aegir in some (hidden) ways - paths in files not rewritten on clone/migrate/rename

The patch above had not been made against HEAD, and it blew away some changes we had already made to incorporate the new 'cli' context in Drush 4 that replaces the 'options' context.

So that's why it would've broken for you under Drush 4 on install: it would have been checking for an 'options' context that no longer exists.

I merged in this patch with the cli changes so it should work now.

Comment #12

Anonymous (not verified) commented 10 February 2011 at 22:38

Status:

Fixed

» Needs work

I am reopening this after we discovered this patch breaks the upgrade path and Migrate, and possibly other stuff too. So I have reverted this patch as it clearly needs more work.

See #1056864: Current upgrade path from 0.4-beta2 is broken for some other discussion, but I've marked it as a duplicate of this.

Comment #13

anarcat commented 11 February 2011 at 16:11

Priority:

Major

» Critical

I think this should be marked critical because it breaks the upgrade path, migrate and so on... Is that right? Or this commit was reverted and the upgrade path is okay?

Comment #14

omega8cc commented 11 February 2011 at 16:17

Priority:

Critical

» Major

This commit has been reverted: http://git.aegirproject.org/?p=provision.git;a=commit;h=6fa9ea1f63af5be7... so now only the cluster stuff is broken again. Changing priority back to major (I hope that is correct).

Comment #15

Anonymous (not verified) commented 11 February 2011 at 23:40

Yep, I reverted it since it simply needs work before it can be reapplied.

The upgrade path is ok right now as a result (just tested).

Might switch HEAD to use Drush Make 2.0 too (I hope it doesn't break anything, I'll test it as best as I can)

Comment #16

Anonymous (not verified) commented 11 February 2011 at 23:53

We can't use Drush Make 2.0 yet, I've identified a bug: #1059238: No core project specified

Good thing I tested :)

Anyway, I'll stop hijacking this ticket.

Comment #17

jgabor commented 24 February 2011 at 09:01

Do I understand it correctly that the primary thing that's "probably broken" with clusters at the moment, is that the hidden fields get auto-filled by the web browser? So if I fix that with Firebug, everything should work as intended?

Comment #18

anarcat commented 15 March 2011 at 15:08

Project:	Provision	» Hostmaster (Aegir)
Status:	Needs work	» Active

Indeed, I am not sure how a patch to provision would fix issues in the frontend. I think this should have been filed in the frontend in the first place...

mig5 - what is the provision patch supposed to fix? Should we throw it at this issue instead? #946606: provision-save core dumps on loops

Comment #19

jgabor commented 31 March 2011 at 14:04

Ok, I've been trying to get clusters working for a couple of days... And even though the RC3 release notes stated nothing about any work had been done on the multi-server feature, I crossed my fingers and hoped for the best. :)

But unfortunately, I ran into the same segfault as dsobon ran into:

Log message
Task starts processing
Running: /var/aegir/drush/drush.php provision-save '@server_webclusteraegirinternalnet' --backend 2>&1
The command could not be executed successfully (returned: Segmentation fault , code: 139)
Running: /var/aegir/drush/drush.php @server_webclusteraegirinternalnet provision-verify --backend 2>&1
Bootstrap to phase 0.
Drush bootstrap phase : _drush_bootstrap_drush()
Load alias @server_webclusteraegirinternalnet
Bootstrap to phase 5.
Drush bootstrap phase : _drush_bootstrap_drupal_root()
Loading drushrc "/var/aegir/hostmaster-6.x-1.0-rc3/drushrc.php" into "drupal" scope.
Initialized Drupal 6.20 root directory at /var/aegir/hostmaster-6.x-1.0-rc3
The drush command '@server_webclusteraegirinternalnet provision-verify' could not be found.
Could not find a Drupal settings.php file at ./sites/default/settings.php.
Command dispatch complete
Peak memory usage was 23.76 MB

We're trying to get a "production-ready" environment up and running for some of our corporate sites, and I'd love to get this working. So if there's anything I can help out with (except coding), just let me know.

(Keep up the awesome work, guys!)

Comment #20

anarcat commented 31 March 2011 at 14:50

It looks like you're a victim of #946606: provision-save core dumps on loops. Can you rerun with the php5-xdebug package enabled?

Please do run the command manually with --debug also so we get more information about what it's trying to save in that context...

Comment #21

anarcat commented 31 March 2011 at 15:20

Joe pasted the goods here:

https://gist.github.com/58181fc3aaab74391fef

It looks like something is failing in the provision-save command:

/var/aegir/drush/drush.php  --context_type='server' --master_url='http://vs70.mcnutility.com/' --remote_host='c1.mcnutility.com' --ip_addresses='10.1.1.74' --db_service_type='0' --http_service_type='cluster' --cluster_web_servers='@server_vs72mcnutilitycom,@server_vs73mcnutilitycom' provision-save '@server_c1mcnutilitycom'

When provision-save fails, it's usually because a referenced alias doesn't exist. Can you check if those aliases exist: @server_vs72mcnutilitycom,@server_vs73mcnutilitycom ? And try again with only one alias, or no alias at all...

Comment #22

joestewart commented 31 March 2011 at 15:38

The both existed. Had the same error when removing one. Removing "--cluster_web_servers='@server_vs72mcnutilitycom,@server_vs73mcnutilitycom'" allowed the command to run . Output added as a comment to https://gist.github.com/58181fc3aaab74391fef

Comment #23

anarcat commented 31 March 2011 at 16:38

Status	File	Size
new	1016890_cluster_debug.patch	877 bytes

Please try the attached patch to enable more debugging.

Comment #24

anarcat commented 31 March 2011 at 16:54

Status	File	Size
new	1016890_cluster_debug.patch	1.09 KB

more debugging again.

Comment #25

anarcat commented 31 March 2011 at 17:07

Status	File	Size
new	1016890_cluster_debug.patch	1.38 KB

again.

Comment #26

joestewart commented 31 March 2011 at 18:58

after applying the patch in #25

aegir@vs70:~$ /var/aegir/drush/drush.php  --context_type='server' --master_url='http://vs70.mcnutility.com/' --remote_host='c1.mcnutility.com' --ip_addresses='10.1.1.74' --db_service_type='0' --http_service_type='cluster' --cluster_web_servers='@server_vs72mcnutilitycom,@server_vs73mcnutilitycom' provision-save '@server_c1mcnutilitycom' --debug
Bootstrap to phase 0. [0.04 sec, 3.49 MB]                            [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drush() [0.04 sec, 3.77 MB] [bootstrap]
Bootstrap to phase 0. [0.12 sec, 12.42 MB]                           [bootstrap]
Found command: provision-save (commandfile=provision) [0.12 sec,     [bootstrap]
12.42 MB]
Initializing drush commandfile: drush_make [0.12 sec, 12.43 MB]      [bootstrap]
Initializing drush commandfile: drush_make_d_o [0.12 sec, 12.43 MB]  [bootstrap]
Initializing drush commandfile: provision [0.12 sec, 12.43 MB]       [bootstrap]
Load alias @self [0.12 sec, 12.43 MB]                                   [notice]
Load alias @server_master [0.12 sec, 12.44 MB]                          [notice]
Loading mysql driver for the db service [0.12 sec, 12.47 MB]            [notice]
Loading cluster driver for the http service [0.12 sec, 12.48 MB]        [notice]
join(): Invalid arguments passed cluster_service.inc:20 [0.13 sec,   [warning]
12.49 MB]
processing web clusters  [0.13 sec, 12.49 MB]                            [debug]
Invalid argument supplied for foreach() cluster_service.inc:21 [0.13 [warning]
sec, 12.49 MB]
initialized web clusters to @server_vs72mcnutilitycom,                   [debug]
@server_vs73mcnutilitycom [0.13 sec, 12.49 MB]
Loading cluster driver for the http service [0.13 sec, 12.5 MB]         [notice]
join(): Invalid arguments passed cluster_service.inc:20 [0.13 sec,   [warning]
12.51 MB]
processing web clusters  [0.13 sec, 12.51 MB]                            [debug]
Invalid argument supplied for foreach() cluster_service.inc:21 [0.13 [warning]
sec, 12.51 MB]
initialized web clusters to @server_vs72mcnutilitycom,                   [debug]
@server_vs73mcnutilitycom [0.13 sec, 12.51 MB]
Initializing drush commandfile: user [0.13 sec, 12.51 MB]            [bootstrap]
Load alias @server_c1mcnutilitycom [0.13 sec, 12.54 MB]                 [notice]
Loading cluster driver for the http service [0.14 sec, 12.57 MB]        [notice]
join(): Invalid arguments passed cluster_service.inc:20 [0.14 sec,   [warning]
12.58 MB]
processing web clusters  [0.14 sec, 12.58 MB]                            [debug]
Invalid argument supplied for foreach() cluster_service.inc:21 [0.14 [warning]
sec, 12.58 MB]
initialized web clusters to @server_vs72mcnutilitycom,                   [debug]
@server_vs73mcnutilitycom [0.14 sec, 12.58 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.14 sec, 12.58 MB]
Load alias @server_vs72mcnutilitycom [0.14 sec, 12.59 MB]               [notice]
Loading cluster driver for the http service [0.14 sec, 12.62 MB]        [notice]
join(): Invalid arguments passed cluster_service.inc:20 [0.14 sec,   [warning]
12.62 MB]
processing web clusters  [0.14 sec, 12.62 MB]                            [debug]
Invalid argument supplied for foreach() cluster_service.inc:21 [0.15 [warning]
sec, 12.62 MB]
initialized web clusters to @server_vs72mcnutilitycom,                   [debug]
@server_vs73mcnutilitycom [0.15 sec, 12.63 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.15 sec, 12.62 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.15 sec, 12.63 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.15 sec, 12.63 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.15 sec, 12.63 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.15 sec, 12.63 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.15 sec, 12.64 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.15 sec, 12.64 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.15 sec, 12.64 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.15 sec, 12.64 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.15 sec, 12.65 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.15 sec, 12.65 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.15 sec, 12.65 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.15 sec, 12.65 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.15 sec, 12.66 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.15 sec, 12.66 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.15 sec, 12.66 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.15 sec, 12.66 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.15 sec, 12.67 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.15 sec, 12.67 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.15 sec, 12.67 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.15 sec, 12.67 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.15 sec, 12.68 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.16 sec, 12.68 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.16 sec, 12.68 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.16 sec, 12.68 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.16 sec, 12.69 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.16 sec, 12.69 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.16 sec, 12.69 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.16 sec, 12.69 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.16 sec, 12.7 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.16 sec, 12.7 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.16 sec, 12.7 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.16 sec, 12.71 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.16 sec, 12.71 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.16 sec, 12.71 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.16 sec, 12.71 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.16 sec, 12.72 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.16 sec, 12.72 MB]
processing web clusters @server_vs72mcnutilitycom,                       [debug]
@server_vs73mcnutilitycom [0.16 sec, 12.72 MB]
PHP Fatal error:  Maximum function nesting level of '100' reached, aborting! in /var/aegir/drush/includes/drush.inc on line 621
PHP Stack trace:
PHP   1. {main}() /var/aegir/drush/drush.php:0
PHP   2. drush_main() /var/aegir/drush/drush.php:41
PHP   3. drush_dispatch() /var/aegir/drush/drush.php:101
PHP   4. call_user_func_array() /var/aegir/drush/includes/command.inc:204
PHP   5. drush_command() /var/aegir/drush/includes/command.inc:0
PHP   6. call_user_func_array() /var/aegir/drush/includes/command.inc:721
PHP   7. drush_invoke() /var/aegir/drush/includes/command.inc:0
PHP   8. drush_invoke_args() /var/aegir/drush/includes/command.inc:58
PHP   9. call_user_func_array() /var/aegir/drush/includes/command.inc:120
PHP  10. drush_provision_save() /var/aegir/drush/includes/command.inc:0
PHP  11. provisionContext->type_invoke() /var/aegir/.drush/provision/provision.drush.inc:227
PHP  12. provisionContext->method_invoke() /var/aegir/.drush/provision/provision.context.inc:226
PHP  13. provisionContext->services_invoke() /var/aegir/.drush/provision/provision.context.inc:209
PHP  14. provisionContext_server->service() /var/aegir/.drush/provision/provision.context.inc:359
PHP  15. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  16. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  17. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  18. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  19. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  20. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  21. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  22. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  23. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  24. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  25. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  26. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  27. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  28. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  29. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  30. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  31. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  32. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  33. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  34. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  35. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  36. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  37. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  38. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  39. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  40. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  41. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  42. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  43. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  44. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  45. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  46. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  47. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  48. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  49. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  50. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  51. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  52. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  53. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  54. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  55. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  56. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  57. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  58. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  59. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  60. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  61. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  62. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  63. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  64. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  65. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  66. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  67. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  68. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  69. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  70. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  71. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  72. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  73. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  74. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  75. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  76. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  77. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  78. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  79. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  80. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  81. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  82. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  83. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  84. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  85. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  86. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  87. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  88. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  89. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  90. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  91. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  92. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  93. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  94. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
PHP  95. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
PHP  96. drush_log() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:20
PHP  97. _drush_print_log() /var/aegir/drush/includes/drush.inc:2193
PHP  98. drush_format_size() /var/aegir/drush/includes/drush.inc:2301
PHP  99. dt() /var/aegir/drush/includes/drush.inc:2408

Fatal error: Maximum function nesting level of '100' reached, aborting! in /var/aegir/drush/includes/drush.inc on line 621

Call Stack:
    0.0016     736232   1. {main}() /var/aegir/drush/drush.php:0
    0.0342    3641600   2. drush_main() /var/aegir/drush/drush.php:41
    0.1177   13025088   3. drush_dispatch() /var/aegir/drush/drush.php:101
    0.1177   13026008   4. call_user_func_array() /var/aegir/drush/includes/command.inc:204
    0.1177   13026496   5. drush_command() /var/aegir/drush/includes/command.inc:0
    0.1335   13119488   6. call_user_func_array() /var/aegir/drush/includes/command.inc:721
    0.1335   13119552   7. drush_invoke() /var/aegir/drush/includes/command.inc:0
    0.1337   13120136   8. drush_invoke_args() /var/aegir/drush/includes/command.inc:58
    0.1345   13146336   9. call_user_func_array() /var/aegir/drush/includes/command.inc:120
    0.1345   13146824  10. drush_provision_save() /var/aegir/drush/includes/command.inc:0
    0.1412   13187824  11. provisionContext->type_invoke() /var/aegir/.drush/provision/provision.drush.inc:227
    0.1412   13188280  12. provisionContext->method_invoke() /var/aegir/.drush/provision/provision.context.inc:226
    0.1412   13188672  13. provisionContext->services_invoke() /var/aegir/.drush/provision/provision.context.inc:209
    0.1413   13191848  14. provisionContext_server->service() /var/aegir/.drush/provision/provision.context.inc:359
    0.1413   13192080  15. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1480   13235424  16. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1480   13235664  17. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1484   13237544  18. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1484   13237784  19. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1488   13240216  20. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1488   13240456  21. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1492   13242888  22. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1492   13243128  23. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1496   13245560  24. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1496   13245800  25. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1500   13248232  26. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1501   13248472  27. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1505   13250904  28. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1505   13251144  29. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1509   13253576  30. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1509   13253816  31. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1512   13256248  32. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1512   13256488  33. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1517   13258920  34. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1517   13259160  35. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1521   13261592  36. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1521   13261832  37. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1525   13264264  38. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1525   13264504  39. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1529   13266936  40. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1529   13267176  41. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1533   13269608  42. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1533   13269848  43. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1536   13272280  44. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1536   13272520  45. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1541   13274952  46. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1541   13275192  47. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1545   13277624  48. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1545   13277864  49. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1549   13280296  50. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1549   13280536  51. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1553   13282968  52. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1553   13283208  53. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1557   13285640  54. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1557   13285880  55. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1561   13288312  56. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1561   13288552  57. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1565   13290984  58. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1565   13291224  59. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1568   13293656  60. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1568   13293896  61. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1571   13296328  62. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1571   13296568  63. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1574   13299000  64. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1574   13299240  65. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1577   13301672  66. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1577   13301912  67. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1580   13304344  68. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1580   13304584  69. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1583   13307016  70. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1584   13307256  71. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1587   13309688  72. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1587   13309928  73. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1590   13312360  74. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1590   13312600  75. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1593   13315032  76. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1593   13315272  77. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1597   13318216  78. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1597   13318456  79. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1600   13320888  80. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1600   13321128  81. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1603   13323560  82. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1603   13323800  83. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1606   13326232  84. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1606   13326472  85. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1609   13328904  86. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1609   13329144  87. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1612   13331576  88. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1612   13331816  89. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1615   13334248  90. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1615   13334488  91. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1618   13336920  92. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1618   13337160  93. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1621   13339592  94. provisionContext_server->service() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:22
    0.1621   13339832  95. provisionService_http_cluster->setContext() /var/aegir/.drush/provision/provision.context.server.inc:116
    0.1621   13340256  96. drush_log() /var/aegir/.drush/provision/http/cluster/cluster_service.inc:20
    0.1621   13341576  97. _drush_print_log() /var/aegir/drush/includes/drush.inc:2193
    0.1622   13343216  98. drush_format_size() /var/aegir/drush/includes/drush.inc:2301
    0.1622   13344272  99. dt() /var/aegir/drush/includes/drush.inc:2408

Drush command terminated abnormally due to an unrecoverable error.   [error]
Error: Maximum function nesting level of '100' reached, aborting! in
/var/aegir/drush/includes/drush.inc, line 621 [0.17 sec, 12.73 MB]

Comment #27

joestewart commented 31 March 2011 at 18:59

Success though.

Adding "$this->server->cluster_web_servers = explode(',', $this->server->cluster_web_servers);" makes it work. But the check whether it is an array fails.

patch coming up.

Comment #28

joestewart commented 4 April 2011 at 16:03

Status	File	Size
new	1016890-web_servers-28.patch	505 bytes

Still troubleshooting, but uploading a patch that shows what was required to avoid the recursion problem. This code really shouldn't always need to be run. There are still warnings from using $this->server->cluster_web_servers when it is either empty or not an array.

UPDATE: There is more going on. Still can't get a site to install properly. This patch may just highlight a symptom.

Comment #29

Anonymous (not verified) commented 31 March 2011 at 20:59

FWIW,

Unconed's originL explanation of the recursion issue http://community.aegirproject.org/node/267

The patch in #4 was his attempted fix (but it broke aegir installs)

Comment #30

joestewart commented 11 April 2011 at 14:18

FWIW this problem was first released with alpha15. It might be obvious to some but took me awhile to catch up. This is when the context changes were first released. So it seems that it wasn't entirely some bug introduced later but that the cluster feature wasn't completely working at that time.

Upgrades from alpha14 with the cluster already defined still work in in alpha15-beta2 and possibly later I didn't test further.

Comment #31

joestewart commented 13 April 2011 at 03:24

Status:

Active

» Needs review

Status	File	Size
new	1016890-provision-cluster-save-31.patch	518 bytes

Patch to provision attached for review. Worked for initial testing from cluster creation to site migrations. Will test on a different install next.

Comment #32

joestewart commented 13 April 2011 at 04:16

Status	File	Size
new	1016890-provision-cluster-save-32.patch	651 bytes

another patch for review.

Comment #33

anarcat commented 13 April 2011 at 18:08

You know, I thougth of doing exactly that too. Since cluster is mainly broken for everybody i hear trying to make it work, and that patch works for you, why not just get that in?

Can you confirm this works in your tests? If so please mark RTBC.

Comment #34

joestewart commented 13 April 2011 at 18:44

Status:

Needs review

» Reviewed & tested by the community

I tested creating a cluster, adding a platform and site on another test envirnoment. RTBC for me.

Comment #35

Anonymous (not verified) commented 14 April 2011 at 00:14

Status:

Reviewed & tested by the community

» Fixed

I pushed this fix and tested that it didn't break anything else obvious.

I didn't test the cluster feature itself yet.

Thanks for the patch!

Comment #36

joestewart commented 14 April 2011 at 14:43

Just a note for future discussion...

Further testing still hasn't revealed any errors. I do see some things that look more like missing features.

1. Clusters seem to be http only. Nothing stops SSL sites from being created, but their vhost and certificate aren't setup correctly.
2. Adding a server to a cluster - platform on the cluster are verified. But the existing sites need a verify as well to have the configs sent to the new server.
3. Removing a server from a cluster - The existing vhosts that were part of the cluster are not removed.

Comment #37

Anonymous (not verified) commented 15 April 2011 at 00:11

We probably would need individual tickets for those.

1. sounds like a feature request
2. sounds like an FAQ thing: there are other cases in Aegir where auto-verifying sites after verifying platforms would mitigate issues like that, but we decided not to go down that route
3. sounds like a bug report

Comment #38

29 April 2011 at 00:13

Status:

Fixed

» Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

Comment #39

mrfelton commented 12 January 2012 at 09:58

Sorry to post in this old issue, but were tickets ever opened for 1, 2 and 3 as per #36/7? I can't find anything relating the the issue of nginx ssl vhosts not being created for clusters.