I need to import pictures with file names of the form 1234-1 into general nodes (not users). 1234 is an item ID that has a corresponding NID already in the DB, and -1 is the picture number. There will be several pictures imported per node. All of the pictures will reside in the same subdirectory, sites/default/files/images/items, for example.

It seems this can't be done in Migrate directly.

Will this ever be a standard feature, and if so, approximately when?

I came across this script (http://drupal.org/node/522916) for importing user pictures from Xoops to Drupal and probably could modify it, but I'm not a programmer so it would be a learning experience for me.

Any tips how to do this?

CommentFileSizeAuthor
#10 CCK_imagefield_table.png25.63 KBfrank ralf

Comments

mikeryan’s picture

It's not clear to me what you mean by importing pictures into nodes... If you've got node 1234 in the db, and picture 1234-1.jpg under sites/default/files, what exactly do you want to happen when you "import" the picture? Have it attached to the node? Have an img tag inserted into the node body for it?

Note that you can do pretty much anything you want with the migrate module by implementing hooks for custom situations. This module isn't really designed for turnkey use - it's meant for use by developers dealing with large, complex migrations.

mikeryan’s picture

Status: Active » Postponed (maintainer needs more info)
druplicate’s picture

I want the migration to do the same as if you manually associated an image(s) with a node by uploading via the CCK image field (filefield - image). So I think you're saying that requires the hook to run the function of the imagefield module for each image.

Not being a seasoned developer, how would I accomplish that in 25 words or less? Is the code complex? Are their similar examples I could modify?

I have about 2,000 nodes and 10,000 images to import. I guess I'd expect it to be as turnkey as any other imported field, just that it's a bit more complex because of the myriad ways to associate images with nodes and because they are files not data.

So by "standard" feature I mean you tell migrate the image directory, first having named the files to match the item ID that already has a corresponding NID in the DB (or the NID itself in some cases). Then tell migrate which module you want to use to import the pictures (image field in my case) and click "go".

If you are creating the node by import from migrate at the same time as importing images, then the migrate module would have to know the key data to associate the image file name with. I think it's asking too much to import any kind of image metadata as well, such as captions.

dman’s picture

Yes, it's quite tricky last time i wanted to do that.
Programmatically creating nodes with ImageFields may be a place to start.

I'm not keen on the idea of renaming images to match node ids (!) and think instead - if I were doing it - the per-node image file path would be data given to the migrate process.

mikeryan’s picture

I'm still a bit confused, but let me describe a scenario that I hope is close to yours:

  1. You have a source table of "stories" with ids like 2000, 2001, 2002, etc.
  2. You have a folder of images related to the above, with filenames like 2000-1.jpg, 2000-2.jpg, 2001-1.jpg, etc.
  3. Before migrating, you copy all the images into sites/default/files/images/items.
  4. So, you need to implement hook_migrate_destination_prepare_node(&$node, $tblinfo, $row)
  5. From $row (the source row for a given item), you get the ID. You scan "sites/default/files/images/items/{$row->id}-*.jpg".
  6. For each file found, you add the appropriate image field data to the $node object (I haven't dealt with image fields at the programmatic level, so I don't know precisely how they're structured).

Is this helpful?

druplicate’s picture

Yup, that sounds about right, but not being a developer, it'll take a bit to digest that completely.

So, "hook_migrate_destination_prepare_node(&$node, $tblinfo, $row)" is part of the migrate module I assume?

Not sure why Dman objected to naming the pics with the NID, but I actually have another item ID that the pic names are based on that is unique to each item and each item has its own node, so I'd have to programmatically associate the two.

In step 6 then, I'd be calling some hook function from the file fields module to insert the images into the appropriate node, right?

Sounds like you aren't planning on incorporating something like this into the migrate module any time soon, but I would think it would be a common requirement for migrating data.

I don't need to do this right away as I can always get someone to do this manually for cheap money, but for future reference it's definitely something I want to do. Plus I'm sure there are a lot of people with similar requests.

dman’s picture

My objection to naming pics by ID is mostly philosophical. :-)

As a metadata guy, I've found that descriptive image names are often the most portable identifiers of the image content.
I really hate systems that take well-named files then store them internally with a hash key or ID making them unidentifiable on the filesystem unless you go through their internal database to find them again.
I like systems that allow you to re-use the same image in different places, so associating them only with one node is a bit backwards to me. But sometimes handy, I guess.
It sometimes seems necessary (or easier) in huge abstract datastores, or cases where you can't trust the input to have any good metadata. I just like to have a bit more shape to my file storage approach, so it can be meaningful in later years.

If your source data has already done that to you, then so be it, but given the choice I'd prefer the database to refer to my resources with a pointer, rather than manipulate my resource naming scheme to fit a databases expectations. Databases are good at managing lookup tables.

As such, if I was building this utility (and I've thought about it a few times) I'd have my CSV with a column that named the associated image(s), and code that found that image and linked to it. That would be useful to this module.
The alternative - code that has to have specific knowledge about your naming scheme and made deductions about where to find things - is just less general-case to me.
Hence why you'd have to do your own custom hook_migrate_destination_prepare_node() code. Which is still an OK approach, just less re-usable. IMO.

The reason I haven't done that so far is because I was only able to trace the filefield insertion process as far as the point where it assumed all new attachments were PHP uploads. Then the cck code got mysterious.
In order to programatically insert filefields it looked like I'd have to do direct database insertions - so I decided to stop there rather than delve into the cck schema.

Like I say, it's mostly philosophical. Other folk find UIDs to be the vital tool for everything... Depends which direction you approach it from, and your approach to RDBMS.

For your use-case here and now, certainly just do custom code that will work for your migration as needed!

druplicate’s picture

Makes sense. I just wanted an easy way programmatically to get the pics where they belong. They were all originally named with proper semantic descriptions, but I thought I'd move that into the text fields that filefield image provides.

Just found this module that seems to do part of what I want. It creates one new node from a single image import. One feature request asked for multiple images per node, and I created one for importing multiple images into one existing node, with some kind of mapping strategy, like pics named for the nid, or building a mapping table.

http://drupalmodules.com/module/imagefield-import

dman’s picture

Cool. There is certainly code that can be stolen from that project!

frank ralf’s picture

Status: Fixed » Postponed (maintainer needs more info)
StatusFileSize
new25.63 KB

Just want to share my workaround for the same problem:

  1. I migrated the text content of the old site into a CCK content type using the Migrate module (which works like a charm and is really a great module!). My content type allows for a maximum of two uploaded photos using an Imagefield.
  2. Imagefield needs a proper file ID for each file and keeps track of the associated files for each node in its own table using node ID, file ID and delta (for more than one image) - see attached screenshot.
  3. For getting all the images into the Drupal system and creating proper entries in the files table I used the File Import module.
  4. I then exported the imagefield table (using phpMyAdmin), added the file IDs in a spreadsheet and re-imported it.

(There might be some cool SQL statement to automise the last step but that's beyond my capabilities...)

Here are some other resources I consulted seeking for a solution:

Migrate from another module to FileField
http://drupal.org/node/432852

File Uploading and Managing
http://groups.drupal.org/node/20291

Media Mover
http://drupal.org/project/media_mover
(Introductory video from Drupalcon: http://dc2009.drupalcon.org/session/media-mover-file-processing-and-stor... )

hth
Frank

EDIT
Also have a look at

Migrating the Drupal way. Part I: creating a node
http://acquia.com/blog/migrating-drupal-way-part-i-creating-node

and the whole four part series: http://acquia.com/category/tags/migration-tips

mikeryan’s picture

@druplicate - between the addition of filefield support (#430100: Help Migrating an Associated Image Set into ImageField/FileFields), and the suggestions above, are you satisfied your questions are answered? If so, please change the status of this issue to "fixed" - if not, please let us know if there's anything else we can help you with.

Thanks.

druplicate’s picture

Status: Postponed (maintainer needs more info) » Fixed

I haven't got the time to try it right now, so I'll mark it fixed. If I have issues I'll reopen it.

druplicate’s picture

Status: Closed (fixed) » Fixed

Just noticed this post on importing images to filefield:

http://technosophos.com/content/simple-pattern-importing-images

EDIT: Another article on importing images:

http://www.appnovation.com/creating-nodes-programmatically-have-cck-imag...

EDIT: And yet another (using Migrate):

http://civicactions.com/blog/2009/oct/27/migrate_module_file_handling

Status: Postponed (maintainer needs more info) » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

frank ralf’s picture

Status: Fixed » Closed (fixed)

Thanks for these most useful links!

druplicate’s picture

Status: Active » Closed (fixed)

I finally got around to testing this method but I'm stumped because Fileimport creates entries into the file table but there's no NID column.

This baffles me because the UI for Fileimport clearly indicates that it's associating the images with a particular node as selected in the auto-complete field.

How does Fileimport make the association between NID and FID? Is there another table I'm missing?

Assuming I can figure this out I would have two columns appropriately sorted by NID from the Fileimport table and the Imagefield table, and just paste in the NID sorted columns containing FID, etc, and then reimport the resulting spreadsheet into the Imagefield table.

UPDATE: Found it - it's the "upload" table in the DB. Now I'm thinking I can use the Table Wizard/Migrate modules to do the Imagefield association rather than manually with spreadsheets. I'll report back if this works. Barring that, you can do the spreadsheet trick but how to create the delta values for multiple images per node? Use an Excel macro? I have 2,000 nodes and 10,000 images to associate.

UPDATE2: Table Wizard didn't like that the upload table had two primary indexes - not sure what to do about that. Also not sure how to deal with multiple images --> one node. Perhaps someone more knowledgeable about the TW/Migrate modules could weigh in here. If that gets sorted out I think it's just a matter of establishing a relationship between the two tables using Views and mapping the fields.

UPDATE3: Just ended up doing this manually with a spreadsheet. Export the upload and imagefield tables, sort all data in both tables by NID/VID and then insert the FID column into the imagefield table. I then wrote a simple formula to create the delta values. Took a little more manipulation to get the CSV values correct (empty fields for the last data field, for example). This is how the final imported data looks in a CSV file:

[vid], [nid], [delta], [field_pictures_fid], [field_pictures_list], [field_pictures_data]

First few rows of the CSV (one record, 4 images):

15634,15634,0,344,1,
15634,15634,1,345,1,
15634,15634,2,346,1,
15634,15634,3,347,1,

The trailing comma ensures that the last of the six fields is entered in the DB as "NULL". After resaving each node, the image data field will be correctly filled out with any text (alt text for example) you may have added.

Tedious work to be sure - sure wish I had the ability to write a proper module. The irony is most Drupal developers doing this sort of migration likely do have this ability but so far no one's contributed anything back. Also, this can tend to be a very specific task that doesn't lend itself to being handled by a generic module. At the least it would require a regex parser to gather info from the image file name so the module knew where to put it.

...Jeff

druplicate’s picture

Status: Closed (fixed) » Active
mikeryan’s picture

Status: Closed (fixed) » Closed (won't fix)

Yes, multiple-field PKs are a problem with Table Wizard, because they're a problem with Views. Table Wizard and Migrate 1 (which depends on views) are not getting more than bug fixes - Migrate 2 (which is much more flexible) is where our effort is going.