I'm attempting to migrate a (reasonably) large wordpress blog from an exported wxr file into Drupal 7.8.
In terms of scale there are 369 categories, 193 blog posts and 6050 comments.
I initially tried importing the entire thing (without specifying drush path) but this fell over pretty quickly (same errors as below).
Instead I tried to import the following separately, but in the order below.
Categories.
Posts.
Comments.
The general behaviour is as follows. From the Migrate UI I check the item I'm importing, select Import from the drop-down and press Import.
For categories and posts, roughly half way through I get the following error:
n AJAX HTTP error occurred. HTTP Result Code: 403 Debugging information follows. Path: /batch?id=196&op=do StatusText: Forbidden ResponseText: Access denied | www.mysite.org @import url("http://mysite.org/modules/system/system.base.css?lrgvmx"); @import url("http://mysite.org/modules/system/system.menus.css?lrgvmx"); @import url("http://mysite.org/modules/system/system.messages.css?lrgvmx"); @import url("http://mysite.org/modules/system/system.theme.css?lrgvmx"); @import url("http://mysite.org/modules/system/system.admin.css?lrgvmx"); @import url("http://mysite.org/modules/comment/comment.css?lrgvmx"); @import url("http://mysite.org/sites/all/modules/date/date_api/date.css?lrgvmx"); @import url("http://mysite.org/sites/all/modules/date/date_popup/themes/datepicker.1...."); @import url("http://mysite.org/modules/field/theme/field.css?lrgvmx"); @import url("http://mysite.org/sites/all/modules/google_analytics_reports/google_anal..."); @import url("http://mysite.org/sites/all/modules/mollom/mollom.css?lrgvmx"); @import url("http://mysite.org/modules/node/node.css?lrgvmx"); @import url("http://mysite.org/modules/search/search.css?lrgvmx"); @import url("http://mysite.org/modules/user/user.css?lrgvmx"); @import url("http://mysite.org/sites/all/modules/views/css/views.css?lrgvmx"); @import url("http://mysite.org/sites/all/modules/ctools/css/ctools.css?lrgvmx"); @import url("http://mysite.org/sites/all/modules/panels/css/panels.css?lrgvmx"); @import url("http://mysite.org/modules/toolbar/toolbar.css?lrgvmx"); @import url("http://mysite.org/modules/shortcut/shortcut.css?lrgvmx"); @import url("http://mysite.org/themes/seven/reset.css?lrgvmx"); @import url("http://mysite.org/themes/seven/style.css?lrgvmx"); Skip to main content Home Hello admin Log out Administrative toolbarDashboard (View and customize your dashboard.) Content (Administer content and comments.) Structure (Administer blocks, content types, menus, etc.) Appearance (Select and configure your themes.) People (Manage user accounts, roles, and permissions.) Modules (Extend site functionality.) Configuration (Administer settings.) Reports (View reports, updates, and errors.) Help (Reference for usage, configuration, and modules.) Hide shortcuts Add content Find content Edit shortcuts You are hereHome Access denied Add to Default shortcuts You are not authorized to access this page.
I then have to 'reset' the item I'm importing and try again. I've managed to import all categories and posts, but with the comments the furthest I've got is 11% through (i.e. roughly 1000 comments) before it falls over as above.
I've tried providing the path to the drush command but that makes no difference. The problem is that even trying to pick up the comment import where it finishes gets nowhere at this point. Presumably it is to do with the volume?
I'm relatively new to Drupal, so was wondering if the problem could be config related - missing something in my php.ini perhaps, but the fact that I'm getting 403 indicates that it could be as simple as permissions, but I've no idea where to start with that! Any ideas
Comments
Comment #1
drewish commentedIf you look in the PHP error log do you see anything? What about the recent log messages (admin/reports/dblog)?
Comment #2
mostoneskin commentedUnder reports->log messages I am getting
Type - access denied
User - admin
Message - batch
Which doesn't tell me much!
Not sure whether this is from the php error log - how do I view that?
Comment #3
mikeryanTry running it in drush with the debug flag and see if you get any useful messages that way:
Comment #4
mostoneskin commentedThanks mikeryan, will give it a try tomorrow
Comment #5
mostoneskin commentedRunning 'all' means the process falls over when it can't find an attachment (an image file that was stored on the old site).
How do I run the process for a specific category (e.g. comments only, as everything else has migrated ok)?
Comment #6
mikeryandrush ms will show you the name of the comment migration, then you can use that as the parameter to drush mi instead of --all.
Comment #7
mostoneskin commentedI did that, and it gave me the comment migration key: of X so I do
. drush mi --X -d | tee log4.txt
and all I get is a bit of bootstrap output and then:
You must specify exactly one of a migration name, --all, or --group [error]
I'm assuming I'm missing the syntax here!
Comment #8
mikeryandrush mi X, not drush mi --X (-- indicates an option - all is an option, as opposed to a specific migraiton name).
Comment #9
mostoneskin commentedSorry mate, was being thick.
"Importing 'X' migration [2.2 sec, 36.36 MB] [notice]
Yikes! One of your processes (php, pid 8464) was just killed for excessive resource usage.
Please contact DreamHost Support for details."
Um, is that a site issue, i.e. upping memory allocation, or could it be a leak?
Comment #10
mostoneskin commentedbtw, googling around indicates that dreamhost + drupal = memory issues, which might be the problem. Still, not sure why the issue was hit at 2.2 seconds in!
Comment #11
mikeryanOK, if Dreamhost is killing processes for exceeding their low memory_limit, not a lot we can do about it except reducing the memory used. Now, the reason it happens so quickly is that we're using SimpleXML to parse XML files like the Wordpress dump, and it reads the whole file in and builds a big in-memory structure before the actual migration starts. As it happens, I've just started an effort to build an alternative XML source plugin that won't have as big a memory footprint for large files, so closing in favor of that issue: #1336534: XML source plugin does not scale.
Comment #12
mostoneskin commentedthanks mate,
I'll have a quick look to see if I can up dreamhost's memory limit sufficiently, in the meantime will keep an eye out for your plugin
Comment #13
mostoneskin commentedRight.
I upped limits massively with a local php.ini -
max_execution_time = 500 ; Maximum execution time of each script, in seconds
max_input_time = 1000 ; Maximum amount of time each script may spend parsing request data
memory_limit = 500M ; Maximum amount of memory a script may consume (8MB)
No idea whether these are well above the max, but anyway...after several attempts I got all 6000+ comments migrated bar 2 (no idea what their problem is).
I've noticed though that special chars end up looking odd. For example a blog title with "I've" in it ends up containing "I've". Etc. Not sure if this is a known bug, or a fixed bug, or not. I know this module is still dev so I can correct these manually, but thought I'd mention it and if you like I can raise a new issue.