Integrate Provision Sync/Hosting Sync modules into DevShop

I've started working on extracting the 'sync' task into external modules in these two sandbox projects:

• provision_sync
• hosting_sync

So far, the code doesn't actually work. ;-) But I'm going to be working on it some today and then probably slowly over the next days and weeks. If anyone wants to help out, let me know and I'll give you access to commit to the sandboxes!

Once we've got something working, I'll make some patches to DevShop to use these modules to implement it's 'sync' task, just to make sure it's possible (probably with some hooks so DevShop can do DevShop-specific things).

Then I'll start looking for a permanent home for this code. I'm not sure it really makes sense as independent modules - maybe I can get them merged into provision_tasks_extra/hosting_tasks_extra or into provision/hosting themselves.

I'll update the issue description to reflect these developments in a moment...

Updated description for change in direction from "requesting information" to actually "executing a plan"

Removed reference to "today" since that doesn't make sense in the issue description.

Made the plan less narrative and more bullet point.

FYI, the sandbox modules actually work now! There's still a bunch of things I want to do in them though:

• Optionally backup the destination site before overwritting it
• Copy private files as well as public files
• Allow optional copying of modules, themes and libraries as well
• Document hooks
• General clean-up!

Hi Jon,

I'm very glad to hear that you support this effort!

I am curious how you will handle a sync form for a very large number of sites.

In the very first version of "Sync" I had a textfield for the alias to sync from!

Yeah, that's a very good question. Right now, it's limiting you to sites on the same platform with the same profile. Although, it could definitely allow different platforms (and would need to for DevShop) but that also means we'd need to make sure all the same modules are available on the other platform (kind of like the "migrate" task does) and I'm punting on that for now. :-)

Maybe we could show an autocompleting textbox if the user has access to greater than 10 sites or something similar?

I've already got it limiting the clients to just the sites they have access to (via hosting_client) which is usually a small number. But admins would have access to all clients and so it could be LOTS of sites.

Anyway, I'll keep you posted on my progress!

Best regards,
David.

The sandbox modules are starting to become relatively robust. Earlier versions could wipe out your database if you gave an invalid source alias but now it will validate everything before beginning, perform a backup of the original site and rollback in the case of an error during sync.

However, they could use a TON more testing!

I haven't yet attempted to patch DevShop to use them, but I did create a bunch of hooks that should allow it. That's probably the next step after this gets some more testing.

Please let me know what you think if you get a chance to take a look!

@ergonlogic: Thanks!

Is this limitation still in place? I think we'd really need to be able to sync between servers for this to be complete. That would allow sync'ing a live DB to a remote dev or testing environment. You may want to look at remote_import for a model there.

Yes, the limitation is still in place. However, it's a "soft limitation" in that we are simply preventing those sites from being listed as available sources. I think it would actually work since underneath we're using the Drush sql-sync and rsync commands which work to remote servers (AFAIK).

I just don't want to give users the option to do something unsafe, ie. sync to a platform that's using a lower version of Drupal or missing key modules. So, it's a matter of coming up with a way to check that the sync is safe (similar to how the "Migrate" task does it) and then listing those as options.

Thanks for the suggestion about remote_import - I'll definitely take a look at that when I get a chance.

There is already an alter hook (hook_hosting_sync_sources_alter), which DevShop could use to add/remove sites to the source list. But there will probably be other things that DevShop needs to make this work, which I'm sure we'll run into when we start trying to patch DevShop to use this. :-) That's the next step after the current code gets a little more testing - so, please help me test it!

Yeah, a config option sounds like a good idea. That would allow people who don't want to sync between platforms to cut down the list of options.

However, there is definitely a way to check if a cross-platform sync is safe, so I see no reason NOT to allow it by default. We just have to get that safety check in there first! (And then test it like crazy...)

Sure, we can give them a "warning" like the migrate task does, but still allow them to do it. What I'm most worried about is something that is definitely not safe, for example:

• Syncing data to a Drupal site that has a lower version than the site the data comes from (Drupal 7.22 to 7.10)
• Syncing data to site with a module that has a lower schema version, than the same module on the site the data comes from

We can always sync to a site with higher schema versions, because updatedb will get it. But going backwards will definitely break!

Anyway, I'm envisioning that this will work just like the migrate task does. I might even be able to reuse that code, but I haven't looked at it yet.

After discussing this so much, I should just sit down and do it! :-) But I probably won't find time until the weekend atleast. If anyone else wants to take a stab at it, just let me know and I'll give you access to push to the sandbox.

Ok! I stole a whole bunch of code from the migrate task for checking compatibility between platforms. So, it now allows you to sync from any site, with the same profile, on a compatible platform (ie. not going backwards with schema versions). It will even show the same Updates/Warnings/Errors numbers and the comparison table.

Please give it a try if you have a chance!

@Jon Pugh: Thanks so much for testing and taking an interest in this! Unfortunately, I won't personally have any time to work on this until at least the weekend after next. :-/ If no one has started working on it by then, I'll take a shot at the changes to DevShop.

Updated issue summary for progress.

Ok! I was planning to work on this today, but all I've managed to do was get a development environment setup for it. :-) I'll try to dig in for real tomorrow.

Attached is an initial patch to devshop_provision to do the provision part of it. It can successfully sync the database, but not everything works because the hosting part needs some adjustment first.

And here is a patch against devshop_hosting which implements the hosting end! I've done a bunch of testing, but more is needed. So far, I can verify that the basics work: syncing database, files and making a backup.

There may also be more opportunities for code sharing, that I'm missing. I haven't tried to be too aggressive on that, this is just the smallest changes to get it to work.

Please let me know what you think!

I forgot to assign it to myself. :-)

The sync modules just got accepted into hosting_task_extra and provision_task_extra: #2096777: Include hosting_sync / provision_sync

Next it's just a matter of reviewing this patch and getting it accepted into DevShop!

Bumping. I want to get this in.

Still waiting on review of my patch in #19!

That's awesome, Jon! Sorry I haven't had a chance to roll a new patch since your comment in September, but I'm glad to hear that this is moving forward. :-)

Woohoo! Thanks. :-)