Posted by mikeryan on June 7, 2009 at 3:03pm
| Project: | Migrate |
| Version: | 6.x-1.0-beta3 |
| Component: | Code |
| Category: | feature request |
| Priority: | normal |
| Assigned: | mikeryan |
| Status: | closed (fixed) |
| Issue tags: | ctools exportables |
Issue Summary
Provide a feature to export content sets, in terms of migration API calls (i.e., migrate_save_content_set()). Then, once you've created and tweaked a configuration for a particular migration source, you can easily instantiate it in code, so it can be part of a module, be subject to revision control, etc.
Comments
#1
+1 for this! If I can help in any way with this, please let me know.
#2
Ok I thought this would be an awesome feature to have, especially since I've had to regenerate my content sets numerous times whenever I've needed to start over with a clean database. So I went ahead and created a new migrate_copy module that does this. I admit to borrowing heavily from CCK's content_copy module however :)
It works quite well to be honest and I've tested it with a few different types of content sets. Like the content copy module, it provides a form where you can choose from a list of existing content sets and then export. You can then copy and paste the generated macro into an import form. The content set will only be created if (a) there is a corresponding view with the same name and (b) if there is no content set already present for the view.
Ideally I'd liked to have created definitions of content sets in code, that could be detected automatically and loaded into the database - much like the way the views module handles default views. However, I didn't have the time to attempt that and it seems like a bit of overkill considering the content sets will only be used in the initial site setup / migration. Perhaps a simple wrapper util could be provided to do bulk imports or something similar instead.
Cheers,
Stella
#3
i haven't looked, but my understanding is that ctools module provides features for modules to do the export and import from code with overriding. worth a look.
#4
Ok here's an alternative version which uses the ctools module.
Brief summary:
<?phpfunction mymodule_ctools_plugin_api($owner, $api) {
return array(
'version' => 1,
'path' => drupal_get_path('module', 'mymodule') . '/includes',
);
}
?>
mymodule.migrate_default_sets.incfile and put it at the path defined above.hook_migrate_default_content_sets()- works in much the same way as the views default views hook. It needs to return an array of content sets, which are keyed on the view_name.admin/content/migrate/destinationspath. Ideally you wouldn't insert content sets into the table until they were overridden, but I didn't want to have to change the processing part of the code. So this hack will have to do until either you fix it up or I find more free time :(admin/content/migrate/destinations/importadmin/content/migrate/destinations/123/export#5
Thanks, that looks very promising! I'll try to give it a thorough review within the next few days...
#6
I applied the patch to the version of Migrate from 6.x-1.x-dev from July 25th and had one collision on the migrate.info file which was easy enough to clean up.
I was able to export some content sets I had created no problem, and then I ran Import and saw the data for the content set I had created. Then I saved the new content set, and Migrate said it saved ok, but the content set didn't show up on the content list.
I then manually created the content set no problem.
Any suggestions on where to start to investigate this bug ?
This is really great feature for migrate, I hope we can get it working.
Also I'm using the ctools that is most recent as of today.
#7
What list did you look at? When you export it to code, currently you need to visit admin/content/migrate/destinations for the code versions to be detected. It's one of the items I've noted above as an area needing work.
#8
I've got a number of content lists that I've created. I didn't export it to code, I just exported one of the lists and pasted the code import directly into the new site's import textarea. I'll try importing it from code to see if that works.
#9
Just a note that I haven't forgotten, I just haven't had time to give this patch a spin (just eyeballed the code a bit).
One thought - how about removing the explicit dependency on ctools and showing the menu items if module_exists('ctools')? One less module to install for people not importing/exporting content sets...
Thanks.
#10
Instead of the ctools dependency, perhaps add a module_exists('ctools') to the access callback of any menu items that depend on it.
I guess the logic to get the content set from code versus get it from DB can be lifted from Views.
#11
Yep, we just need to ensure that the processing code can handle having content sets in the database and fetching others from the code. I just didn't want to have to deal with that part, as well I didn't have the time.
#12
Unfortunately, this patch no longer applies cleanly. Would be great if someone could reroll.
#13
The attached patch based on stella's patch on #4 applies cleanly against 6.x-1.0-beta3 and seems to work. Needs reviewing and testing.
#14
Thanks for the re-roll - I should have time to review and commit it Friday.
#15
Starting to look at it... Logging comments as I go along:
On the modules page, ctools itself (fresh HEAD checkout) is not selectable even though it depends on nothing else. The .info looks fine to me... I enabled Bulk Export and then disabled it, and was left with ctools enabled. Quick scan doesn't turn up an issue in the queue, I'll add one...
I had imagined that a hard dependency on ctools could be avoided, just enabling the export/import links and menu items if it was present, but it seems to want to take over all object management. C'est la vie...
Sorting on the content set listing page (just recently fixed) is now broken. Of course - the content set list comes from ctools - no more tablesort_sql etc. This will require removing the sortable headers (not a huge loss) and sorting the array by weight.
I did export, delete, and import a content set successfully, so the basic functionality works.
I have to say, so far I'm not fond of the ctools approach. It requires people to install another module which they don't necessary need, seems heavier than it needs to be for this application (do we need defaults/overrides?), and doesn't touch the front half of setting up a migration - Table Wizard and Views.
Here's an example typical of what we do in our current project to set up a content set programmatically:
<?php
// Add source tables
tw_add_tables(array('cms_blog', 'cms_author', 'cms_entry'), TRUE);
// Add foreign keys
tw_add_fk('cms_entry', 'entry_blog_id');
tw_add_fk('cms_entry', 'entry_author_id');
// Add relationships
tw_add_relationship('cms_entry.entry_blog_id', 'cms_blog.blog_id');
tw_add_relationship('cms_entry.entry_author_id', 'cms_author.author_id');
// Add the content set
$content_set = new stdClass;
$content_set->view_name = 'blogs_content_set';
$content_set->sourcekey = 'entry_id';
$content_set->contenttype = 'node';
$content_set->desttype = 'ourblog';
$content_set->description = 'Blogs';
$content_set->weight = 15;
migrate_save_content_set($content_set, array('base_table' => 'cms_entry'));
$mcsid = $content_set->mcsid;
// Add the content mappings
$mapping = new stdClass;
$mapping->mcsid = $mcsid;
$mapping->default_value = 1;
$mapping->destfield = 'uid';
migrate_save_content_mapping($mapping);
$mapping = new stdClass;
$mapping->mcsid = $mcsid;
$mapping->srcfield = 'cms_entry_entry_title';
$mapping->destfield = 'title';
migrate_save_content_mapping($mapping);
$mapping = new stdClass;
$mapping->mcsid = $mcsid;
$mapping->srcfield = 'cms_author_author_nickname';
$mapping->destfield = 'field_mtblog_byline';
migrate_save_content_mapping($mapping);
$mapping = new stdClass;
$mapping->mcsid = $mcsid;
$mapping->srcfield = 'cms_entry_entry_text';
$mapping->destfield = 'teaser';
migrate_save_content_mapping($mapping);
?>
So, my ideal export function would poke into the view structure to see what tables are in the view, for any tw-managed tables work out the relationship settings, and maybe for the hell of it export the view too (or at least link to the view's export).
What the hell, I threw together a quickie export (just the migrate data), attached, to show how I was thinking of this.
So, what do people think? If there's a consensus that the ctools approach better meets people's needs, I'll work it in...
Thanks for the work put into this so far.
#16
Small fix to that last patch...
#17
Seems like table wizard mapping is a good use case for overriding whats in the DB. You want to be able to do that in order to fiddle, and then re-export when you are done.
I'm not married to the ctools approach. I just like that it automatically gets us Features compatibility. But we can just as easily write our own Features compatibility. See http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/features/AP.... There are some more examples in the 'includes' directory there.
Features is nice as it gives us a framework for sayingn that a content set depends on a view and on a content type and on certain modules and so on.
#18
Hi Mike,
I had this, I *think* it's because I had migrate already installed and due to the ctools dependency added to migrate.info by the patch, if you comment that line briefly you can install ctools in the normal way.
This obviously raises a question about how dependencies work and what a suitable workaround for this situation would be, expecting users to edit the info file is not great.
#19
well the solution would be to disable the migrate module, enable the dependency and then re-enable the migrate module. You could try enabling the dependency module in a hook_update() but that requires the site to already have that module installed. We could print a warning message in hook_update() if the module isn't available to enable I suppose.
I never quite finished off the ctools implementation in my patch. I haven't looked at the newer versions, but the original patch always stored the content sets in the database - which is not the "ctools" way of doing it. Really, unless the content sets have been overridden by the user, they should only be stored in the code, but I didn't want to have to modify the migrate processing bit, so didn't finish that bit of the patch.
With the ctools implementation, I really like the fact that I can automatically import my content sets - without having to do an extra step of copying and pasting into an import UI (which my very first (non-ctools) patch in comment #2 above did). I have a lot of content sets at this stage and the idea of copying and pasting in export code isn't a good solution IMO.
#20
OK, we'll proceed with the ctools implementation. It does need work to integrate more cleanly (and I need to get up-to-speed on the ctools API to understand what it's capable of).
#21
Maybe I missed it in the above replies, but why can't we make "Content set export" a separate module like CCK does (and as the original patch did)? Then it could use a CTools dependency to its heart's content. :)
#22
The original patch didn't use ctools. Think of the way views works - you can store your view in code or in the database. If stored in code, it never reaches the database until it is modified. That's the way the ctools module works too. The above patch doesn't do that exactly, but it should (one of the areas that needs work). That way you can store your content sets in code and run the migration off both the ones in the database and those that aren't. One of the consequences of that is the migrate module would have a ctools dependency.
It might be possible to split it into a separate module, but the core module probably wouldn't be devoid of ctools api calls, as it would have to handle the case where content sets are in code. It may be worth investigating, but I'm not sure how feasible it would be.
#23
Sorry, I searched for "clone", not "export" so I missed this and submitted #621422: Add ability to clone/export content set definitions ... Slightly better title for the search-impaired like myself. ;)
#24
bump
#25
tagging
#26
not sure how we are dealing with this, but how do we handle consistent relationships to mapping tables? Might it make sense to give content sets a unique id, like views does? then we could have "migrate_map_my_content_set" and the tw exports will always work. I imagine it is easier to do the other import and export actions too if we have a unique id. I skimmed the post, so If it's covered already, I apologise.
#27
Noticing this also effects drush, which is using an integer as an ID, instead of a unique string.
Examples:
migrate import 4 Import new items in the content set with mcsid 4
migrate import 4 --update Import new items, and also update previously-imported items
migrate import 4 --idlist=4,9 Import two items in mcsid 4. The ids refer to the value of the primary key in base table
migrate import Articles --itemlimit=50 Import up to 50 items from the content set named Articles
migrate import Users --feedback="60 Display a progress message every 60 seconds or less
seconds"
migrate import 2 --feedback="1000 items" Display a progress message every 1000 processed items or less
#28
I've committed the basic export functionality I suggested previously. I've gone back-and-forth on the CTools approach, but I feel because it's a full commitment - CTools would become inextricably entwined with content set management, and I think most users of Migrate would not have a need for the functionality it provides - I'm tabling that for now. We can revisit this in a future version.
@frankcarey: The unique alphanumeric ID thought is a good one, and may be helpful for #698180: Content sets based on non-views data. I've opened a new issue for this task: #698178: Unique alphanumeric IDs for content sets. Thanks!
#29
Automatically closed -- issue fixed for 2 weeks with no activity.