Hi,

I have to replace several text strings with >9,000 occurences in one field (node body) in one content type on one site (Example text strings to search for/to replace: ">First</a>, ">Previous Picture</a>, ">Next Picture</a>, ">Last</a>), so I think I can not filter down the operation to smaller "batches" (e.g. by processing sequentially different content types or smaller subsets of the strings to replace).

Is there a way to make 'Search & Replace Scanner' process this amount of data safely?

The current behaviour is:

Searches are processed in about 40 seconds; after that amount of time, I'm getting the following message at ./admin/content/scanner:

Scanning for: [">First</a>] ...
max_execution_time changed from 30 to 600.
Found matches in 9048 fields. See below for details.
Default max_execution_time too small and changed to 10 minutes.

Then it starts to load Links to all matching nodes, without giving a pager, into ./admin/content/scanner (a page with several megabytes).

Search & Replace operations seem to take significantly longer than the given 10 minutes; even after letting the module run for over 20 minutes, no replaces seem to have been made, at least I'm getting no result screen. There is no progress indicator during this time, but - and that's quite cool - 'm also not getting a timeout or a WSOD or something similar. I simply don't know what it's doing in the background.

Is there a trick or workaround to make 'Search & Replace Scanner' process this amount of data safely? Are there e.g. other known modules that slow search & replace operations significantly down? Or maybe there are settings (e.g. "ignore case") that are especially "costly" in regard to processor and/or memory ressources?

I'm marking this as a support request; if 'Search & Replace Scanner' currently can't do this at the moment, I'd like to switch this to a feature request. If 'Search & Replace Scanner' is an will not be capable to process this amaount of data (e.g. because of restrictions of Drupal core etc.), please feel free to set this to 'won't fix'.

Anyway, thank you very much for your incredibly useful and reliable module!

Greetings, -asb

Comments

asb’s picture

asb commented

Hi,

I just made a little experiment and

  • disabled some "costly" modules (pathauto, xml_sitemap, globalredirect, relevant_content),
  • limit the search & replace operation to "Case sensitive search", and
  • let the search and replace operation run once more for 25 minutes.

At ./admin/content/scanner/scan/confirm, after this time, it's still saying: Replacing items... please wait....

When going back to ./admin/content/scanner/scan after these 25 minutes and searching for the string to replace, I'm now getting:

Scanning for: [">First</a>] ...
max_execution_time changed from 30 to 600.
Found matches in 7036 fields. See below for details.
Default max_execution_time too small and changed to 10 minutes.

Before: 9048 matches, and after: 7036 matches, so 'Search & Replace Scanner' might have done 2012 replacements in 25 minutes; maybe it times out silently in the background after some time (10 minutes, if search & replace sets the same timeout like search?), I'm not sure about this.

Observation by the side: When monitoring MySQL during the search & replace operation with the mytop utility at the shell, I'm not seeing not much activity on the site's database. Strange, but might be an imperfection of mytop.

However, I'll continue this approach and hope that I'm not causing any damage to my site... ;)

Greetings, -asb

Updates:

After 3rd run: "Found matches in 4438 fields." (= 2,598 nodes/matches processed; running time: 67 min.)
After 4th run: "Found matches in 1492 fields." (= 2,946 nodes/matches processed; running time: 71 min.)
When running the 5th time, I'm getting:

Replacing [">First</a>] with [">Erstes</a>] ...
max_execution_time changed from 30 to 600.
Replaced items in 746 fields. See below for details.
Default max_execution_time too small and changed to 10 minutes.

Ok, task finished. Now I have to repeat this three times for the other strings and hopefully I'm done (ETA: 6 hrs)...

aasarava’s picture

aasarava commented

Thanks for posting the outcome of your experiment. It's odd that changing only 9,000 entries takes so long for you. But as you've noticed, the module tries to extend PHP's script run time to the max if your query is running out of time. Unfortunately, that's not always enough, especially if the db queries are taking a *very* long time.

As a workaround, doing your replacements in batches like you're doing should be fine -- except for when your replacement might end up being recursive. For example, if you replace T with TEST on 100 out of 200 records, and then try to do it again, you're going to get TESTESTEST on those first 100 records.

asb’s picture

asb commented
Status: Active » Closed (fixed)

So far I haven't run into any side effects of the approach described in #1. It's quite ugly, but if it works... ;)

Setting this to 'closed'.

Greetings, -asb