I have a file field that can have multiple downloadable files for an image node type. These images will have several versions (psd, jpg, eps) that will be downloadable. I would like to be able to append multiple values to this field. How would I do that when they are coming from multiple source columns (i.e. eps_filename, jpg_filename, psd_filename)?
I found related documentation here: http://drupal.org/node/1012810
But that documentation is for taxonomy terms where the details of the value can simply be a string with a separator. For this I am using MigrateFileFieldHandler::arguments(), so I can't just give it a separator.
Further what I would like to do is also be able to come back later and append more files to the migrated nodes. This is because the larger files will take longer to get on the server, so their smaller counterparts (jpg, png, etc) will be loaded first and then adding the eps files later. Now I already figured out how to update the previously migrated nodes using systemOfRecord. But I am not sure how to append to a multi-value field.
Thanks!
Comments
Comment #1
mikeryanFor your first question, the last section of the referenced doc page is the relevant one - you would pull together the array in prepareRow() or prepare().
For the second question, where you're updating existing rows with more data and thus using systemOfRecord = DESTINATION... Well, that's tricky - in that case what Migrate will do is replace entirely any field that's mapped, and leave in its entirety any field that's unmapped, neither of which is what you want. The only thing I can see to do off the top of my head is to override MigrateDestinationNode::import() with your own version which is exactly the same, except that in the "foreach ($old_node as $field => $value)" loop for field_files you would merge the $node and $old_node arrays.
Anyone else have another idea?
Comment #2
patrick.thurmond@gmail.comHmm, I have been trying the prepareRow() option but on the updating class that adds the EPS files and getting nowhere. This is probably why.
I will look into doing the override. Actually right before reading this reply I had already come to the conclusion that I would have to either override something in DestinationNode or write my own version of it entirely. This just seals the deal. Its taking me awhile to figure this stuff out because I am trying to understand how the code is working before I start doing things.
Comment #3
patrick.thurmond@gmail.comActually in looking at it further, since the import function calls "$this->prepare($node, $row);" before saving the node I think I might be able to modify the node itself here if I invoke passing by reference in my function declaration.
But there is still another problem I see. Isn't the file field object (and subsequent file table population and file transfer) done before we get to this point? If it is then I will need a way to do that again from this point.
Comment #4
mikeryanThat's the tricky part - the prepare() call invokes the field handlers (which take raw values as set like $row->filelist[] = $row->eps_filename; and replaces them with field arrays) before calling the migration's prepare(). So, it's too late for you to set the raw values and have the proper handling done for those values. Hmmm, unless you explicitly called the field handler from your prepare($node, $row)... See MigrateFieldsEntityHanlder::prepare() on how to call migrate_field_handler_invoke_all, if you call that with the array of new files to add as the last parameter, then merge the results into the existing field in $node, that should work.
Comment #5
patrick.thurmond@gmail.comOk, so I have been looking this over and that invoke will change it at the entity level, not the node level. So my first question is, is that at a deeper level than it needs to be for this to work or can it be done at the node level? The next question is where would I declare my prepare to make this happen? It doesn't look like preparing it at my setup class would work for the entity level and I am having trouble determining at what does extend the correct class to handle that.
It looks like to use that prepare statement I would have to extend the MigrateFieldsEntityHandler class or create a duplicate sibling class that can handle multiple file inputs. A more permanent solution would be to get it to allow multiple mappings to the same field by modifying addFieldMappings() in migration.inc. That bit is rather simple and could even be done at the level of the developers code that extends the "Migration" class. Below is what I am thinking of for that:
Though in order for that to work any code that loads the results of those mappings would have to be able to handle the multiple mappings as well. I haven't traced any of those just yet. But that would be crucial for a permanent addition of this feature. After that it just needs to ensure that any entity or node modification calls can handle the multiple values as well when dealing with it prior to saving. I have been considering the need for a flag in the migration declaration that states that this migration will allow multiple values in fields.
I am still pondering over all of this.
Comment #6
patrick.thurmond@gmail.comAnother question that has crossed my mind is that in the class "MigrateFieldsEntityHandler" in fields.inc the prepare function receives the $row object/array but doesn't do anything with it. If your going to have hooks that you invoke for a field then doesn't it make sense to pass in the row data in case it is needed? Or am I missing something?
The values the prepares do receive seem to be the information about the structure of the node entity field and not anything from the actual source or destination. So I am having trouble following where the row data is coming into the prepare function that is being invoked.
By the way a lot of this is my way to of speaking out loud to myself. Sometimes I just find it easier to work through things this way.
Comment #7
patrick.thurmond@gmail.comOk, so I have given it some thought and here is an approach for appending files to a node on node update. We add a parameter to the migration called "multiValueFields" that will be an array of the fields in which you would like to allow multiple values for. Then, as long as the field is an array, you append the new values on it.
Essentially this only works for translatable fields such as files, images, the body, etc. But then it doesn't really make sense to append together ints or bools. I added in some code for strings, but it may need to be considered that we need to determine if we are appending or pre-pending to the string. Also we need to consider any separators (commas, spaces, pipes, poems, whatever). So the string thing needs a bit of hashing out.
So my idea for patching the migrate module would be two-fold. First for creating a new node (or whatever entity type we are using). This would require allowing for duplicate mappings and checking to make sure the field supports duplicate values.
The second would be to path updating nodes (and other types) and letting the developer specify which fields get updated. This would also have to allow for multiple mappings at some point. But if the first step is already working then you can just use that stuff.
In the mean time I have attached a file that contains the new node destination class. It extends upon the original instead of overwriting it. So it can allow for multiple ways of approaching this.
Comment #8
drewish commentedAny way you could post that as a patch? It'd be a lot easier to review. Give http://drupal.org/patch if you need help figuring out how to do that.
Comment #9
patrick.thurmond@gmail.comYeah, I probably should have done that instead. Here is that patch file.
Comment #10
patrick.thurmond@gmail.comThe variable "$migration->multiValueFields" isset check could be eliminated if we set the property by default in the class as an empty array. We could even eliminate the "is_array" check if we use getters and setters for it to control the data integrity.
Comment #11
mikeryanComment #12
mikeryanMissed this on my previous roundup to #1240928: META: Refactoring of file destination/field handlers - all file stuff will be dealt with for Migrate 2.4.
Comment #13
mikeryanComment #14
robeano commentedI applied the patch in comment #9 and got it to work with multi-value text fields. There was a bug in the patch which kept it from saving values to the text field if the multi-value field on the existing node was empty. This new patch fixes that issue and I renamed a couple of variables to make them less 'files' specific or more descriptive.
Comment #15
mikeryanRetitling to reflect what this issue is currently about.
I have to say I'm not fond of this approach at all. What this is trying to achieve is something that is somewhere between a system-of-record of SOURCE and DESTINATION, merging the two sides on a field-by-field basis - this seems like it should be left to the specific application extending the destination class, rather than trying to build this into Migrate itself.
Comment #16
mikeryanI'm going to let this go - I think it's enough for Migrate itself to support a system-of-record of SOURCE or DESTINATION, any hybrid approaches should be dealt with on a per-application basis.