Some modules will store the path of a site in the 'variables' table. For example, xmlsitemap stores the xmlsitemap_cache_directory hard-coded in the variables directory.

The solution was to update the variable in the table manually. (I actually deleted it and cleared the cache since the module then generates the correct value).

Is this the responsibility of aegir or of the module developer, and can aegir handle this issue when cloning?

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Anonymous’s picture

Project: Hostmaster (Aegir) » Provision

Moving this ticket into Provision, which kind of answers your question: we *do* do some switches when the *new* database is bootstrapped for a site that's in the process of migration/clone.

How we could capture all possible cases is a bit tricky, happy to discuss it here.

FYI, have a look at /var/aegir/.drush/provision/platform/drupal/deploy.inc

The provision-deploy (which is the nuts and bolts of what makes migration/clone work) invokes this 'engine' once bootstrapped, allowing us to do a bunch of queries and variable_sets to account for this sort of thing.

See also this related ticket #791262: Color module & theme logo/favicon paths not updated during site deployment

dsobon’s picture

I'm working on this (and all site path) issues as we speak.

I will upload a patch once testing has nailed all the instances where it needs to be updated.

anarcat’s picture

Title: Cloning a site doesn't handle file path change » Cloning a site doesn't handle file path change in the variables table

Narrowing this down - we're doing our best to cover most cases here, as mig5 mentionned, but it's possible we missed the variables table or some parts of it. Please document other such instances here even if you don't have a fix for it.

Thank you for your work.

dsobon’s picture

OK. three instances that I cannot fix are:

1) *.tpl.php files that contain references to /sites/oldsite.com/files/. this must be UPDATED by hand.

2) custom modules written to attempt including files from /var/aegir/platforms/SOME_PLATFORM_NAME/sites/SOME_SITE_NAME/modules/

3) imagecache appears to have a regex that looks for ^/sites/oldsite.com/files/ and replaces it with /sites/newsite.com/files if it matches, otherwise it _appends_ to the original path, ending up with /sites/oldsite.com/files/imagecache/sites/newsite.com/files/imagecache/ or something very similar. So if you install the site with a different sitename, access it, then migrate it, this is the result.

#1 and #2 are web development practice issues. #3 is a module-specific issue. My patch will resolve issues that are database-related. I'll post it early in the week once I've isolated more problems other than the above listed.

Trying to implement a production, staging and development environment requires that path names and hostnames can be migrated with zero issues.

Is there a drupal / aegir best-practices web development guideline? This would reduce time and effort spent trying to undo mistakes, especially if dealing with 100+ sites in a production & development environment.

dsobon’s picture

FileSize
6.33 KB

OK. #3 is another tpl.php problem.

You may ask why it iterates through all the aliases... this resolves half-baked site-rename manual migrations from broken aegir installations.

Anyway... patch attached.

dsobon’s picture

Priority: Normal » Major
omega8cc’s picture

Status: Active » Needs review

The patch from #5 needs review - changing status.

omega8cc’s picture

Status: Needs review » Needs work

The patch from #5 doesn't apply to the master. Please reroll and change status to needs review.

omega8cc’s picture

Also, please don't use tabs in the code - replace them with spaces: http://drupal.org/coding-standards#indenting

omega8cc’s picture

A quick note: I think you shouldn't limit paths rewrites to sites/%s/files, as there can be other URLs used in the sites/domain path, for example paths to themes or modules or private files etc. so it should be always sites/%s.

Also, there should be no trailing slash in the pattern, to avoid missed replacements.

omega8cc’s picture

Version: 6.x-0.4-alpha3 »
Status: Needs work » Needs review

Updated and simplified patch for paths rewrite on clone/migrate:
http://drupalcode.org/sandbox/omega8cc/1074910.git/commit/38a8e47

anarcat’s picture

Status: Needs review » Needs work

1. there are whitespace/unrelated changes again (in nginx)
2. i don't think we should update the menu_links, menu_router and system tables - that's get dealt with by the cache rebuilds
3. i do not understand what this is for:

+db_query("UPDATE {node_revisions} SET body    = REPLACE(body,       '://%s',    '://%s')",    $old_url, $new_url);

... it seems too broad. The xml_sitemap stuff should be refactored as:

+  variable_set('xmlsitemap_base_url', str_replace($old_url, $new_url, $var));

... for simplicity.

omega8cc’s picture

Yep, I just noticed it got some unrelated bits, sorry.

The replace (3) is probably to replace absolute URLs in the content (it happens often when people are using wysiwyg editors to paste images).

I just rewrote a bit the patch from #5, but will reroll it.

omega8cc’s picture

Status: Needs work » Needs review
anarcat’s picture

Status: Needs review » Fixed

looking good, committed, thanks.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

crea’s picture

Version: » 6.x-1.1
Status: Closed (fixed) » Needs review
FileSize
850 bytes

Follow-up: update default user picture variable

anarcat’s picture

Status: Needs review » Needs work
+++ platform/drupal/deploy.incundefined
@@ -43,6 +43,14 @@ db_query("UPDATE {node_revisions} SET teaser  = REPLACE(teaser,     'sites/%s',
+  $pattern = 'sites/' . $old_url;
+  $pattern = '/^' . preg_quote($pattern, '/') . '/';

i don't think the $pattern variable is necessary here, just inject the pattern right in there.

+++ platform/drupal/deploy.incundefined
@@ -43,6 +43,14 @@ db_query("UPDATE {node_revisions} SET teaser  = REPLACE(teaser,     'sites/%s',
+  $picture = preg_replace($pattern, 'sites/' . $new_url, $picture);

in fact, we usually use str_replace() in the orther locations, why use a preg in this case?

this way this could be shortened to a single line...

Powered by Dreditor.

crea’s picture

preg_replace() is more reliable because you can specify beginning of the string. str_replace() is dangerous because you could mis-replace the pattern in some non-filesystem path.

I would change str_replace() to preg_replace() everywhere if I was you, but I think that deserves a different issue..

crea’s picture

Status: Needs work » Needs review
FileSize
1.21 KB

Added smileys module support

omega8cc’s picture

Status: Needs review » Needs work

I'm not sure we should add fixes for any possible/existing contrib modules, because it never ends then, so I vote "no" for adding it for smileys here, because it is a bug in the module if it stores path to the images in its own table instead of using standard system path to files.

crea’s picture

I agrree that making smileys use fid instead of path would be better in theory, however, it's much harder to convince the maintainer to rework the module and it takes time to release new version with upgrade path. Meanwhile we need a fix for existing installations, otherwise we are just forcing everyone either to fork provision (as I did already) or to fork smileys. I'm not convinced that forking is a good solution here.
I see no reason not to include fixes for modules, they are harmless.

anarcat’s picture

Agreed. No to smileys! :) Seriously, we can't cover all third party contrib modules like this, it would be insane. Feel free to make a custom module for that. I think we should support whatever is in core.

crea’s picture

When did I ask to cover ALL third party contrib modules ? There will be probably only several which have problems with filesystem path.
I thought Aegir was made to manage real installations, not some ideal-world-all-perfect-code-modules sites.
Get out of your developer sandbox. Welcome to real world, Neo

Reported installs: 4074 sites currently report using this module.

anarcat’s picture

Wow, what's with the attitude, Morpheus? Want sunglasses with that comment?

Seriously, you need to understand our position here. *You* didn't ask for *ALL* modules, but as a whole, the community is incrementally asking to support more and more of those. I was just suggesting that *maybe* some of those can go into contrib.

Besides, if you want to support the smileys module in Aegir, make an issue about it, do not hijack another unrelated issue.

I am still waiting for a patch that will answers the issues I have pinpointed in #18. I am ready to concede on your point about preg_replace vs str_replace, but you still have an extra variable there.

Thanks.

ergonlogic’s picture

Version: 6.x-1.1 » 6.x-2.x-dev
Status: Needs work » Needs review
FileSize
4.25 KB
810 bytes

Here's a re-roll of the patch from #17 removing the extra variable.

I also took the liberty of creating a second patch cleaning up the rest of the file. In this case, I used a variable for the regex pattern, since I replaced str_replace() with preg_replace() throughout, as per #19 and #25.

ergonlogic’s picture

Oops, I made a copy/paste error in that second one. re-rolled.

anarcat’s picture

+++ b/platform/drupal/deploy.incundefined
@@ -16,37 +7,38 @@ $old_url = drush_get_option('old_uri', $new_url, $context);
+$old_url = drush_get_option('old_uri', $new_url, 'cli');
+$old_url_pattern = '/^' . preg_quote('sites/' . $old_url, '/') . '/', 'sites/';
[...]
-  $var['logo_path'] = str_replace($old_url, $new_url, $var['logo_path']);
-  $var['favicon_path'] = str_replace($old_url, $new_url, $var['favicon_path']);
+  $var['logo_path'] = preg_replace($old_url_pattern, $new_url, $var['logo_path']);
+  $var['favicon_path'] = preg_replace($old_url_pattern, $new_url, $var['favicon_path']);
   variable_set('theme_settings', $var);
 }

strictly speaking, this regex is not the same as the str_replace: it will replace data only at the beginning of the string, but that may be harmless.

other than that, this looks okay.

ergonlogic’s picture

Status: Needs review » Fixed

That was the point in #19 to which you were agreeing in #25, no?

Fixed in 0b6bb1b.

anarcat’s picture

true, thanks.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

  • Commit 0b6bb1b on dev-drupal-8, 6.x-2.x, dev-ssl-ip-allocation-refactor, 7.x-3.x, dev-subdir-multiserver, 6.x-2.x-backports, dev-helmo-3.x by ergonlogic:
    Issue #905326 by ergonlogic, crea: Improve file path changes.