By building upon the entity API we could add a generic entity processor. In particular that would be useful for entity types being created via the entity API itself, e.g. profiles, messages or field-collections. However, it could also serve as an alternative implementation for the core-entities.
Related to #1005128: Rules integration & enable modules to customize imports, one could use Rules for creating the entities instead of directly the entity API. So we could benefit from the Rules form elements it implements for the entity property info data types, thus we get proper form elements for inputting dates, formatted text, entity references, .. In addition we would could benefit from Rules' input evaluation system and get stuff like token replacements for free. For that we could just re-use the 'entity_create' action + the form Rules provides for it.
Anyway, here is a first patch attached which just uses the entity API and implements simple very simple form elements. It should basically work, I've not really tested it though.
This is related but not identical to #1257314: Ability to attach a feeds source to any entity, not only nodes.
Comments
Comment #1
fagoops, used the wrong creation function.
Comment #2
Grayside CreditAttribution: Grayside commentedI tried configuring a Message importer, but when I attempt to save the processor to use the Message Processor, I get a couple errors.
Three of the first, followed by confirmation that I changed the processor plugin. Then the actual error. On page refresh, no processor is actually selected.
Comment #3
dasjograyside, i ran into your problem once as well but using current versions it works.
also there was a problem with the feeds_ui distinguishing plugins by class, which is fixed in the attached patch
Comment #4
Steven Jones CreditAttribution: Steven Jones commentedSubscribe
Comment #5
Grayside CreditAttribution: Grayside commentedOkay, tweaked a comment and made entity input format configuration reference the current entity, instead of sticking with the default "nodes" from FeedsProcessor.inc.
Continuing to test with the Message module, when I reached the mapping phase, I discovered that this integration does not identify the difference between entity properties and entity property bundles (not up on the correct terminology yet, sorry!) That's to say, Message has an "Arguments" property which can be a fairly arbitrary set of values, but the Mapper had no insight into this. In Message's case, this might require some smarter mappers, I explain it here for completeness' sake.
I tried jury-rigging a mapping anyway, and attempted to import a feed. The error that happened shortly into the batch process was massive, including gigantic entity definition arrays of doom--the piece actually explaining what went wrong here:
Deciding to switch the experiment to a more stable use case, I tested my system with a standard Node Processor, then the Entity Node Processor. Standard Node Processor worked fine, the Entity Node Processor ran into some problems.
Indicated as successes:
Indicated as warnings:
<repeated what I assume is 31 times>
Indicated as errors:
@dasjo: What type of entity did you use?
Comment #6
dasjograyside, i will see if i can test into this upcoming weeks but can't promise.
generally, feeds seems to get a push again due to the OA/MN migration which sounds great!
Comment #7
lathanKeeping an eye on this
Comment #8
e2thex CreditAttribution: e2thex commentedComment #9
febbraro CreditAttribution: febbraro commentedsub
Comment #10
e2thex CreditAttribution: e2thex commentedFirst of all I really like this path for the processors, and if we can get this working I think the current processors might go away (are atleast go away on the back end).
I have jumped in testing with a basic cvs file that I want to move into nodes
I am getting the same error as in #5
Warning message
Unknown data property feeds_item.
SQLSTATE[21S01]: Insert value list does not match column list: 1136 Column count doesn't match value count at row 1
And I think the issue is that in FeedsProcessor newItemInfo there is the line
but entity is a EntityDrupalWrapper instance, and when calling a property, it is using its __get method, which is trying to build a new EntityDrupalWrapper. (__get is part of the parent EntityStructureWrapper)
Looks like there might be something in EntityStructureWrapper::__get that lets you by pass this for dpm to work, I wonder if that could be more general (maybe a way to create a property besides just declaring it?)
Comment #11
fago>First of all I really like this path for the processors, and if we can get this working I think the current processors might go away (are atleast go away on the back end).
Great, so let's make it work again. :)
>but entity is a EntityDrupalWrapper instance, and when calling a property, it is using its __get method, which is trying to build a new EntityDrupalWrapper. (__get is part of the parent EntityStructureWrapper)
hm, $entity shouldn't be a wrapper but a usual entity object. If it is, this certainly causes troubles. The wrapper allows you only to set properties which are declared accordingly in the entity's property info.
Comment #12
e2thex CreditAttribution: e2thex commentedHere is where we build the new entity (from the patch)
looks like entity_property_values_create_entity returns a EntityDrupalWrapper (from entity.property.inc)
Comment #13
fagooh, good catch. We should use
then.
Comment #14
kapayne CreditAttribution: kapayne commentedsub
Comment #15
Shadlington CreditAttribution: Shadlington commentedSubbing
Comment #16
7wonders CreditAttribution: 7wonders commentedsub
Comment #17
Peter Törnstrand CreditAttribution: Peter Törnstrand commentedSubscribing.
Comment #18
jamsilver CreditAttribution: jamsilver commentedOK - new patch incorporating the suggestion in #13 and adding two new lines
Adding 'plugin_key' entry - needed to make ctools plugin work where key is different to handler file.
There is an issue with finding a general way of filling all default properties of an entity when creating it. Not sure where the good solution is for this problem, but adding this language line was needed in my use case, and is probably a common one.
Powered by Dreditor.
Comment #19
jamsilver CreditAttribution: jamsilver commentedMore work on it. Added hook_entity_insert/update/delete so feeds keeps track of the entities its imported.
Comment #20
jamsilver CreditAttribution: jamsilver commentedAgain more work: Fixing bug with call to entity_delete_multiple.
Comment #21
Grayside CreditAttribution: Grayside commentedGenerally a good idea to double check with the assigned developer before doing work on the issue. It's probably safe to assign it to yourself if you want to keep pushing a little more, since @e2thex seems to be radio silent here.
Comment #22
jamsilver CreditAttribution: jamsilver commentedBit more work - getting updates to work on existing entities, adding GUID field.
I'm just doing this because I really need to get it working for my own needs asap, and I thought that others may find it useful to find a working patch in this queue. In the long run e2thex is welcome to use as much or as little as my work as he wants.
Comment #23
podaroksubscribe
Comment #24
muschpusch CreditAttribution: muschpusch commentedTried the patch from #22 and tried to import profile fields (profile2) from CSV:
PHP Fatal error: Call to undefined method FeedsMissingPlugin::getMappings() in [..]modules/feeds/feeds_ui/feeds_ui.admin.inc on line 488
I disabled profile2 and switched to profile and chose "Mapping for Entity processor User" but still no new fields in the mapping section :(
Comment #25
wolan CreditAttribution: wolan commentedAlso tried the patch #22 but am getting an error when importing from csv to profile2 entity:
Anyone else experiencing this behavior?
/wolan
Comment #26
inversed CreditAttribution: inversed commentedsubscribe
Comment #27
muschpusch CreditAttribution: muschpusch commentedIt actually works with d7 Core user account fields.
Comment #28
scotthooker CreditAttribution: scotthooker commentedI am getting the same error "PHP Fatal error: Call to undefined method FeedsMissingPlugin::getMappings() in [..]modules/feeds/feeds_ui/feeds_ui.admin.inc on line 488" when attempting to make a feed with an entity processor for node.
Comment #29
Niklas Fiekas CreditAttribution: Niklas Fiekas commentedIntresting. Subscribe.
Comment #30
Dave ReidSome of this has already been committed (the hook_entity_insert/update/delete implementations were committed in another issue).
Comment #31
Niklas Fiekas CreditAttribution: Niklas Fiekas commentedSeams to be #1066806: Use hook_entity_insert/update/delete rather than type-specific hooks. Thank you.
Comment #32
Niklas Fiekas CreditAttribution: Niklas Fiekas commentedThis could also solve #1041804: Feeds and Media Module: A Drupal 7 Media Utopia?, couldn't it? (Media now only requires files to be imported, no specific Media stuff.)
Comment #33
webankit CreditAttribution: webankit commented+1
Comment #34
elly CreditAttribution: elly commentedSubscribey!
Comment #35
BarisW CreditAttribution: BarisW commentedSubscribing.
Comment #36
Niklas Fiekas CreditAttribution: Niklas Fiekas commentedHow is it going? (Is there a patch of the current state against Feed's HEAD?) How can I help?
Comment #37
hellomobe CreditAttribution: hellomobe commentedApplied patch #22 - get the following error
EntityMetadataWrapperException: There is no property for this entity. in EntityStructureWrapper->get() (line 381 of ...sites\all\modules\entity\includes\entity.wrapper.inc).
To recreate: I went to feed-importers ->add new -> processor -> selected entity field-collection
Comment #38
torgosPizzaSub.
Comment #39
thepak CreditAttribution: thepak commentedsub
Comment #40
Bevan CreditAttribution: Bevan commentedThe patch needs re-rolling. The patch from #22 does not apply to 7.x-2.0-alpha4 (which is same as 7.x-2.x-dev at the time of writing):
Removing assignee (e2thex) since (s)he only touched this issue twice and that was four months ago, on March 15 and 17. And I may re-roll the patch in the coming days or weeks (but do not count on it).
Comment #41
Bevan CreditAttribution: Bevan commented#1066806: Use hook_entity_insert/update/delete rather than type-specific hooks added
hook_entity_insert/update/delete
since comment #22. (The patch-conflict was something else in the same file,feeds.module
.) #1066806 (da8d974) excluded this from each implementation:The attached patch is like the patch from comment #22 but with the following changes;
hook_entity_insert/update/delete
are not added tofeeds.module
, since they are already there// $Id$
is removed fromplugins/FeedsEntityProcessor.inc
, since CVS tags are deprecatedfeeds.1033202-20-no-prefix.patch
is excluded from the patch file, since it is not required and patch files in patch files is very confusing! :)It is a git patch and thus applies with
patch -p1
.I do not know how to test this, since my interest is in #1063434: Add Feeds integration to FieldCollection. And that appears to require further work beyond the scope of this issue and patch. (If someone could confirm, that would be great.)
Comment #42
philipz CreditAttribution: philipz commentedI'm trying patch from #41 on Profile2 profile entities and after some trials of mapping / settings configuration it finally starts importing profile's fields.
The problem is Entity processor settings form where required fields of Type and User show up.
In order to import any fields to a profile I had to set up the User field to uid of one of existing user accounts.
This of course doesn't make sense because ideally I would set that uid binding in Mapping form taking the uid's from my csv file.
Using uids here also isn't perfect solution as I would prefer to use email or username field to bind imported profiles to user accounts...
I'm going to try and use this general entity processor to write my own only for Profile2 profiles entities. Then hopefully I will understand it better and maybe figure out how this could be done with general approach (or that it's not possible).
Is someone trying simillar thing ?
Comment #43
Grayside CreditAttribution: Grayside commentedTried to apply the patch, needs a reroll.
Comment #44
jyee CreditAttribution: jyee commentedFWIW, the patch in #41 was generated from the drupal root (named "app") and not the feeds module root. This means that if you run
patch -p1
from the feed module root (as is the Drupal standard), it will install a plugin in your feeds directory at feeds/app/sites/all/modules/contrib/feeds/plugins. You will need to move the FeedsEntityProcessor.inc file from that directory back to your feeds/plugins directory, otherwise the include will not be found and you will receive errors such as: "Missing Feeds plugin FeedsEntityProcessorProfile2. See profile_importer. Check whether all required libraries and modules are installed properly."Comment #45
tim.plunkettComing from #1063434: Add Feeds integration to FieldCollection
Comment #46
Bevan CreditAttribution: Bevan commented@jyee: Ooops! My fault.
patch -p7 < 1033202-entity_processor.patch
from the feeds module directory should workaround that issue until someone rerolls the patch.Comment #47
Anonymous (not verified) CreditAttribution: Anonymous commentedsub
Comment #48
wesnick CreditAttribution: wesnick commentedI have rerolled this patch in the standard fashion, I have been using it and it seems to work fairly well. It is worth noting that you must define your entities' metadata properties and their setters to have any success with this.
Comment #49
dasjothanks wesnick
Comment #50
wesnick CreditAttribution: wesnick commentedI have been using this patch for a few weeks and it works well. Marking "needs review"
Comment #51
webadpro CreditAttribution: webadpro commentedwesnick, Do all I need is your patch or is there another patch I must also apply to get this working, because after applying your patch I get all kinds of errors.
eg.
Missing Feeds plugin FeedsEntityProcessorUser.
Thanks
Comment #52
Mohammed J. Razemsubscribing
Comment #53
wusel CreditAttribution: wusel commentedI have patched Feeds from #48.
Then I created a new Feeds-importer with 'Entity processor Profil' to import the Profile2-fields ("complete_name","birthday","tel_1") from the CSV-file:
What shall I put in the field 'User' on the page 'Settings for Entity processor Profil' to connect the new Profille-fields to the existing users?
I don't know, what "The owner of the profile. Expected data type: user. Just enter the identifier of the entity, i.e. User ID" means in this case. I assume, that the CSV-file contains no field "UID"! The field "member_nr" is a historical member-ID and not the UID.
Please help me.
Thanks in advance
Comment #54
wesnick CreditAttribution: wesnick commented@webadpro: Make sure to clear your caches; since the classes are generated dynamically, you need to make sure that entity info is recached and all class definitions are added to the registry. Sometimes it requires two cache clears to fully refresh cached data correctly.
@wusel: The settings page is for you to add default values for required entity properties. It doesn't matter what you put here, just as long as it is the correct datatype. If your CSV does not have the values that correspond to these required fields, then the created/updated entity will get these default values. Best case, just ensure that your CSV has the appropriate values. If you want to match up existing users with values on a CSV you need to have a uid field that maps to the User's uid, and be sure to add it to the mappers section. If they are new users, then you do not need this. Once feed creates entities from a feed, this particular entity will exist in the feeds_item table so any future imports/updates will update the previously created entity with the fields from the CSV.
Comment #55
TimelessDomain CreditAttribution: TimelessDomain commentedThe generic entity processor sure is pulling in a lot more targets. Very nice. I haven't been able to test all of them, and am unsure of the format for some targets. I will post back my results. We need to compile a list of what generic entities this opens up for feeds, so that we can keep on top of each one. I started a list of them below, but did not add the targets within the entities yet. We need a documentation page for this.
Would it be possible to have a generic sub-entity processor (creating new entities from a field that is embedded within another entity)? At the beginning we could just do 1 level (which is what media feeds is doing http://drupal.org/project/media_feeds ) then eventually we could go deeper.
I don't think that Media Feeds allows the ability to fill in additional fields of the sub/child-entity, but this function would be ideal.
This is definitely where feed processing is going, so why not leave the design open to be adapted to this or get it done now to give our community a headstart on the rest. thanks
Comment #56
Grayside CreditAttribution: Grayside commentedHi Timeless,
I don't think Feeds should keep track of individual entity types, except perhaps ones in Core. Feeds should concentrate on providing generic support for Entities, and other modules should do any further finessing of the integration.
Recursing entity processing seems reasonable, but I think it should be addressed in a followon issue dependent on this one.
Comment #57
andreij CreditAttribution: andreij commentedSame error of #37 EntityMetadataWrapperException: There is no property for this entity. in EntityStructureWrapper->get() (line 408 di /modules/entity/includes/entity.wrapper.inc)
Debugging the code the get function in entity.wrapper.inc is called with a null parameter. Where should I look to try correct this bug?
Comment #58
andreij CreditAttribution: andreij commentedThe problem arises from this line:
$id_info = $wrapper->$name->get($info['entity keys']['id'])->info();
the get function throws:
EntityMetadataWrapperException: There is no property for this entity. in EntityStructureWrapper->get() (line 402 of /opt/local/apache2/htdocs/soraimar/sites/all/modules/entity/includes/entity.wrapper.inc).
$info['entity keys']['id'] is in fact null, what should it be?
Comment #59
patcon CreditAttribution: patcon commentedWas looking into what had happened to the FeedsDataProcessor in plugin in D7, and stumbled across this train of threads -- REALLY exciting stuff guys :)
Comment #60
patcon CreditAttribution: patcon commentedIn case anyone finds it helpful, here's th patch from 48 rerolled for alpha4 which will work with drush_make
Comment #61
mukesh.agarwal17 CreditAttribution: mukesh.agarwal17 commentedGreat work already. I went through the patches and I note a common thing, i.e. we are trying to use a single processor class for all our entities. The only issue with that is the feeds_ui form for processors. Using any of the above patches, create your custom import and go to the list of processors, select your "Entity processor MY_ENTITY" and save, it will not show the same as default value after save. The reason is the following line (in function feeds_ui_plugin_form in feeds_ui_admin.inc):
The default value sets the value from handler's class which is the same FeedsEntityProcessor for all the entities. Attached is the patch which works for me.
Also, I think entity import should be a separate module altogether. It is not a generic requirement and should stay out of feeds.
Comment #62
mukesh.agarwal17 CreditAttribution: mukesh.agarwal17 commentedAlso, not every entity needs properties and fields, they could exist as data (which might get used in some computations not necessarily related to display). All of the above patches assume that only the properties need to be imported. And therefore the line:
in getMappingTargets function. I'm currently working on making this more generic to allow all types of data with the entities importable (including fields). If anyone has already worked on it, please share the details.
Comment #63
mrfelton CreditAttribution: mrfelton commentedAfter applying the path from #60, and setting my processor to the Entity Field Collection processor, and then going to the settings page for the processor (/admin/structure/feeds/tc_feeds_rankings/settings/FeedsEntityProcessorField_collection_item) I get the following warning repeated 3 times:
As well as the following errors:
Note that the patch in #60 did not apply cleanly. It wanted to make some changes to feeds.info, but they were completely out of sync with the .info file in the latest -dev code. But, I don't think that is related to the problem here.
Comment #64
Anonymous (not verified) CreditAttribution: Anonymous commentedI can confirm the exact same issue as #63. I tried applying it to 7.x-2.0-alpha4. From what I can gather it's supposed to create a file "plugins/FeedsEntityProcessor.inc" but that's not happening.
I'm patching via OSX Terminal - what could we be doing wrong here?
Comment #65
colanAdding tag.
Comment #66
patcon CreditAttribution: patcon commented@mrfelton sorry, #60 is against alpha4, not dev. it was just a reroll of a previous, so that I could put it in a makefile
@afestein #60 was rolled for drushmake, which might be applying the patch differently than you are manually. look into the patch command's "p" flag
"patch -p0 < test.patch" should work from the module's root dir
Comment #67
bibo CreditAttribution: bibo commentedPatch from #60 + additional comments from #66 works for me.
Comment #68
cthiebault CreditAttribution: cthiebault commentedPatch #60 on alpha4 with field_collection dev generates the same error as #37:
Comment #69
elliotttf CreditAttribution: elliotttf commentedRe-rolled the patch against 7.x-2.x.
Code functionality was not changed; however, the .info file no longer includes plugins in its files array.
Comment #70
adam_b CreditAttribution: adam_b commentedI've been following this thread with interest, because I have a lot of data which I need to import into Field-Collection entities. The structure isn't complex -- each row has one date (year-only) field and three text fields -- but obviously none of them are presently visible in the field-mapping interface.
I'm a product manager rather than a developer, so I haven't tried any of the patches yet. Would anyone be able to give me an idea of how close we are to being able to do what I need? Should I just roll a patch and try it? Would a financial contribution speed up the process?
Many thanks.
Comment #71
bibo CreditAttribution: bibo commented@adam_b: If you can apply the latest patch in #69, go for it. elliotttf might also like a contribution?
Our contribution will be testing/reviewing this patch as soon as we can in the next few days. Entity support is really crucial for feeds. Hoping to get this RTBC asap.
Comment #72
bibo CreditAttribution: bibo commentedUnassigning e2thex , who apparently has not been around for 10 months. This was already suggested.
There have been a bunch of patches by different people, is there someone who this assign should belong to?
Comment #73
cthiebault CreditAttribution: cthiebault commentedPatch #69 still does not work with field collection... but that's normal as code functionality was not changed.
Comment #74
staniel25 CreditAttribution: staniel25 commentedI almost have it. Added patch #69. Now, the Entity tables now show up and are selectable.
However, the table fields are not showing up in the 'Target' selection list, as I would expect.
Any suggestions?
Tx Nube
Comment #75
killtheliterate CreditAttribution: killtheliterate commented@jdschwam is that list being built with a different query?
Comment #76
staniel25 CreditAttribution: staniel25 commentedNo special queries from my side. I think the query for Target values comes from the #69 patch.
I am wondering if my db table has been set up correctly an an Entity. Anybody have a suggestion on how to verify this?
Thanks in advance for any help.
Comment #77
Anonymous (not verified) CreditAttribution: Anonymous commentedI was following the great article by Yuriy Gerasimov about Entities on Trellon's website (here is the link http://www.trellon.com/content/blog/creating-own-entities-entity-api ). It gives you a great explanation about how to create your own entities.
I have applied the #69 and now I can see my entities on the processor selector. But none of my fields are available on the Mapping section. Note that the fields on my entity are declared in code so far.
I wonder why is this issue. I hope someone can explain.
Thanks in advance.
Comment #78
kreynen CreditAttribution: kreynen commentedI had to move the FeedsEntityProcessor.inc into the plugins dir after applying the patch, but that may have just been the way I applied the patch. I also extended FeedsEntityProcessor with class FeedsEntityProcessor[ENTITY] in the module I defined the entity. After that, I could map any fields added to the entity, but not fields defined by the entity. The only field showing up in the mapper was the [ENTITY]_id, but mapping that returns...
It also returns that error when trying to map an id to any of the entities already defined (comments, og, etc). Worth noting that Media does not show up in the list a targetable entities, but anything defined by extending the EntityAPI does. Not sure what would be required to add to Media to make it targetable, but that would helpful to at least define what's required to target the entity with the FeedsEntityProcessor.
Comment #79
etiennechataignier CreditAttribution: etiennechataignier commentedSame error as #73 here when I select Entity Field Collection Processor after applying patch #69.
However, I see that #74 partially succeded.
Is there a chance that versions of modules I use could lead to this error ?
I'm using Feeds 7.x-2.0-alpha4+40-dev.
Comment #80
bradjones1Reclassifying as this isn't a support thread; it has a pending patch needing review.
Comment #81
likewhoa CreditAttribution: likewhoa commentedThis is missing two plugins in order to work with profile2, they are:
Also getting
Fatal error: Call to undefined method FeedsMissingPlugin::getMappings() in sites/all/modules/feeds/feeds_ui/feeds_ui.admin.inc on line 502
when clicking on mappings. This is with latest -dev and patch from #69 applied.
Comment #82
geek-merlincrosslinking:
This is related but not identical to #1257314: Ability to attach a feeds source to any entity, not only nodes.
Comment #83
johnv@likewhoa,
Feeds mappers are better off in their dedicated module, so each module maintains its own Feeds support.
Therefor, shouldn't the Profile-Feeds support be in the Profile2-module? And then the status back to Needs Review?
(BTW, I see support-requests for Profile2 popping up in other modules, too, so perhaps Profile2 doesn't set some variables.. )
Comment #84
franz@johnv, I think this is not the case. This patch generates those classes for all entity types, so that's why it tries to find those classes. Maybe this is a case of clearing all caches? Either way, the fatal error is just a derivative of the first, since the proper class is not found, FeedsMissingPlugin is used instead.
Comment #85
likewhoa CreditAttribution: likewhoa commented@johnv yes i agree each contrib module should support their own feeds mapper because feeds module itself shouldn't have to cater to every module which wants a mapper but just provide the necessary api for them to use and implement.
@franz I can rule out cache which doesn't remedy the problem.
Comment #86
js CreditAttribution: js commentedThank you for this work.
I have installed patch in #69 and creating a feed into a "data table". The table shows up nicely in "select a processor", but I don't have the option to select certain fields as a "unique target".
I have used the data patch
http://drupal.org/node/1551430 #4.
I am also getting the following error, and don't yet know if it is related:
Notice: Undefined property: stdClass::$headers in http_request_get() (line 98 of ../sites/all/modules/feeds/libraries/http_request.inc).
Warning: array_change_key_case() expects parameter 1 to be array, null given in http_request_get() (line 98 of /var/www/nolet/html/sites/all/modules/feeds/libraries/http_request.inc).
Download of failed with code -1002.
Comment #87
js CreditAttribution: js commentedI haven't sorted out the logic yet, but the problem is that I am not setting up fields correctly when I create the data table structure.
Comment #88
js CreditAttribution: js commentedDoes someone have feeds working with /project/data?
I have used the patches for /project/data from here:
http://drupal.org/node/1551430
What seems to be the problem is that guid and url are not set for the feeds_item object.
I am very confused by where this should be mapped. The FeedsEntityProcessor.inc appears to me to work differently than the FeedsNodeProcessor.
I would appreciate help.
Comment #89
js CreditAttribution: js commentedI appreciate all the work and help with feeds and data. I have it working and for me for scale, this is substantially better then creating nodes.
Should the this be camelcase like the parent?
+ public function entitysave($entity) {
I think the parent is: entitySave
Comment #90
mikemadison CreditAttribution: mikemadison commentedThe entity processor seems to be working fine for me, but Data Tables aren't showing up as options. I have tried several different patches listed on http://drupal.org/node/1551430 without any success. I have created a couple of different data tables, and I still don't see anything.
Am I missing something?
Thanks for all the work!
Comment #91
emilyf CreditAttribution: emilyf commentedi patched with #69 against feeds dev using -p1 flag and it works correctly. then extended FeedsEntityProcessor with FeedsEntityProcessor[ENTITY].inc in my module but the only field that is mappable is the id.
i then integrated this patch: http://drupal.org/node/1063434#comment-5217400
as the other fields in my module are field collection fields.
now my field collections are also mappable, and data is coming in.
Comment #92
liquidcms CreditAttribution: liquidcms commentedjust weighing in with my findings so far:
i added the patch for field collections linked in #91
- this now lets me see the fields within my first collection: question and tip
- but i don't see how i would select delta to map to which specific on of these i am writing to (??)
- i also don't see the fields within the collection field (so i guess no support for recursively descending through the entity?) - granted this might be hard to map out from the other end (CSV file) anyway
other:
- on manage fields pages for collections i get numerous "Missing Feeds plugin FeedsEntityProcessor." errors
- i have not tried to actually import anything yet (no delta control so wouldn't make sense to import)
i added patch from #69 above and adds nothing to UI; but possibly would make things work (??)
i don't understand emily's comment above "then extended FeedsEntityProcessor with FeedsEntityProcessor[ENTITY].inc in my module" - perhaps this helps with something (??)
i could also throw some funding $$ at this issue if it would help (or coding effort)
Comment #93
wiherek CreditAttribution: wiherek commentedI applied patch from #69 to the latest dev version. I am getting two errors now, one of which is critical.
The minor:
drupal (and drush upon enabling the module) displays this warning:
Missing Feeds plugin FeedsEntityProcessor. Please contact your site administrator.
The critical error:
After importing some rows from a csv file, I get:
EntityMalformedException: Missing bundle property on entity of type node
I checked in the database and both 'bundle' and 'type' cells have false as value.
Comment #94
wiherek CreditAttribution: wiherek commentedOk, the problem was related to custom entities created by modifying the Model Entities module. We tried using Feeds with entities created with Entity Creation Kit module, and that works without errors.
Comment #95
gregglesAfter reading the comments in ~81-92 I'm not sure if this is "needs review" or "needs work" but it's definitely not just active :)
Comment #96
scottrigby@Greggles Agreed!
I should mention I've been using #69 with success (along with #661606: Support unique targets in mappers).
Though I continue to get this warning about a dozen times on every cache clear:
If someone else can confirm this, we should probably change to 'needs work'. Otherwise I may be missing something minor in my configs or setup (because it does seem to work perfectly otherwise).
Comment #97
imclean CreditAttribution: imclean commentedI'm also seeing the error "Missing feeds plugin FeedsEntityProcessor" when clearing the cache, as is at least one other.
"Needs work" seems to make sense.
Comment #98
gilgabar CreditAttribution: gilgabar commentedThe 'Missing feeds plugin FeedsEntityProcessor" errors look like they are a result of a bug in feeds_entity_property_info_alter() introduced in #1424992: Add implementation of hook_entity_info_alter() so that $node->feed_nid is available to Rules. The call to feeds_plugin() is passing in the class name as the plugin name. This works in most cases because the plugin name and class name are usually the same. It breaks in this case because the class 'FeedsEntityProcessor' is used for all of the feeds entity plugins, none of which use that exact name.
Changing that to use the plugin name fixes the missing plugin errors, but results in another set of errors. "Notice: Undefined index: type in FeedsEntityProcessor->entityType() (line 18 of /sites/all/modules/contrib/feeds/plugins/FeedsEntityProcessor.inc)." One of those for each entity plugin. This also appears to be due to the hook_entity_property_info_alter() implementation. It creates an instance of a processor with a fake importer for each plugin, so that it can check for the existence of the 'entityType' method and then add a feed_nid property to each of the entities supported by a plugin. The problem with this is that the fake importer appears to think it is configured to use a node processor, so when it gets to calling the entity class, which expects the plugin to have a type property, it doesn't find a type because the node plugin does not have one.
I would suggest removing the checking of processor plugins in feeds_entity_property_info_alter(). The addition of support for all entities with this patch should make that redundant.
I haven't tested extensively, but a revised patch is attached.
Comment #99
gilgabar CreditAttribution: gilgabar commentedAnd of course I forgot to set it to 'needs review'.
Comment #100
geek-merlin@gilgabar: i thank you wholeheartedly for that good explanaition.
I support this and it would be in line with #1739842: Deprecate some processors in favor of generic entity processor.
So we still need this reviewed and tested!
Comment #101
ndf CreditAttribution: ndf commentedFollowed #77 Turoibz instructions and patched against Feeds 7.x-2.0-alpha5.
Tests:
- Well Known Entities 'Node', 'User' and 'Comment'.
- Entity from Code.
- Entity from Code with Fields attached via UI.
- ECK (Entity Construction Kit) Entity with Fields.
My findings:
- For well known Entities 'user', 'comment', 'node' all properies and fields are working.
- For Entities build with ECK properies and fields are working.
- For coded Entities attached fields are not working / recognized automaticly. Properties do work.
NB:
- For ECK Entities getting recognized you need to flush cache.
Conclusion for sitebuilder and coders: use ECK if you want an extensible and 'feedable' site.
I am very interested in #62 Mukesh.agarwal17 not only have a generic entity processor, but also a generic attached field processor. My expectation (or wish) is that when we create a new entity with fields it is 'magicly' processable by Feeds.
Comment #102
ndf CreditAttribution: ndf commentedComment #103
geek-merlinthank you niels for your tests. this really brings us much closer to takeofff!
any ideas from folks here why this is the case and how difficult this ist to implement?
Comment #104
gilgabar CreditAttribution: gilgabar commented@nielsdefeyter could you clarify at what point the fields are breaking for the case of coded entities?
Re: #62, fields are added as targets via hook_feeds_processor_targets_alter().
I haven't had a chance to test a full import yet, but the field targets are showing up for my custom coded entities, as well as a contrib coded entity (profile2), so I'm curious if you are seeing the fields listed as available targets at all, or if the import is failing at some point after the feed is configured, or something else is going on.
Comment #105
miiimoooI've just tested this as well thanks to tests from #101
Using ECK, git 7.x-2.x version of feeds and feeds_sql the initial import works a charm for text and integer properties and fields.
I'm a bit confused about the update feature. Does this require the GUID or the Id to be set as unique target? Using the Id I get "SQL violation for the primary key" warnings. It looks like it's trying to create new items instead of updating the existing ones.
Comment #106
bpmbriguy CreditAttribution: bpmbriguy commentedI was able to successfully use an ECK entity and the feeds entity processor. I am importing a CSV file with a primary key, text, and integer fields. In this case my primary key is "ProviderID"
While using the update feature, I mapped ProviderID to GUID and set it as unique. Then I also mapped ProviderID to my entity ID.
Comment #107
jlyon CreditAttribution: jlyon commentedHere is an addition to the patch in #98 to always include a select box to choose the bundle on the config form. This was all I needed to be able to map to the attached fields for an entity in code that was generated from the Model module, as well as for the redhen_contact entity that is a part of the Redhen module.
Comment #108
scottrigby@jlyon that should probably be a separate issue. Cross-referencing #661606: Support unique targets in mappers & #1711648: Abstract bundle handling.
btw, interdiffs are really helpful on large patches like this, so people can see at a glance what you're proposing ;)
Comment #109
twistor CreditAttribution: twistor commentedOk, so I finally got around to re-reviewing this after initially reviewing it some time ago. I am sold, however, there's no way that this is going to get into the 2.x branch. I poked around the patch to get it to work on current dev, and for multi-columned fields like the Link field. It's awesome, but I think it's time for 3.x.
Actually, there is a way to get this into 2.x. The plugin would have to only declare itself for entities that are supported already, so it would have to be last and do some checks. The field handling that it does itself would have to be removed because it is very broken and doesn't allow for a bunch of fun, new things that are in Feeds. To do that we would need to extend Feeds api to be able to determine the plugin type and key externally to a plugin. Also, it would have to handle inheritance correctly, which, currently it basically ignores.
Attached is my *very* quick hacking to prove to myself that this is the way forward.
Comment #110
twistor CreditAttribution: twistor commentedErrr, this one.
Comment #111
pcambraLooks great, but there are some extra spaces and a dpm left behind in getMappingTargets.
This processor will kill all custom processors (such as commerce_feeds ones) so it'd be great to have it in feeds itself.
Comment #112
Cebra CreditAttribution: Cebra commentedI have applied the patch on the current dev-version. Do I have to do anything else to use feeds with a custom entity? Because I don't find a custom entity processor in the importer settings of feeds. Thanks
Comment #113
patcon CreditAttribution: patcon commented@cebra, if it's like the old one, you might need to create a bundle, if i remember correctly. Might be wrong, but just skim the comments above and they should recap the expectation :)
Comment #114
Cebra CreditAttribution: Cebra commentedThx. I created a bundles but I don't see any custom entity processor in the feeds importer settings.
I used the patch111 on the dev-version from september, 8th. But I received the messages:
Could it be that the patch hasn't been applied?
Comment #115
Cebra CreditAttribution: Cebra commentedOk.... I've found my mistake. ;-) Now the patch works fine.
But when I select an entity processor and click on the mapping-link afterwards there is a failure, so that my screen keeps blank. I thought it is an url problem, but I'm not sure.
Comment #116
twistor CreditAttribution: twistor commentedThis should get rid of the failures, but it's still rough.
Comment #117
Cebra CreditAttribution: Cebra commentedWorks fine! ;-) Thank you!
Comment #118
spotzero CreditAttribution: spotzero commentedAttached is a patch, based heavily on patch 116, that provides a working generic entity processor (at least everything I've tested so far is working), without requiring any other changes than adding the FeedsEntityProcessor.inc file (other than having to mention it in "feeds.plugins.inc" of course).
Also, this FeedsEntityProcessor has the entity type selection as a settings option, so it doesn't clutter the "processors" tab with a entity processor for each entity type.
I have to admit though, that I'm a little confused at what a lot of the changes to feeds in patch 116 do, since, as demonstrated by this patch, they don't appear to be necessary for a generic entity processor.
I haven't been following this issue for very long, so please let me know if I'm way off in my understanding of how things should work.
Comment #119
yang_yi_cn CreditAttribution: yang_yi_cn commentedHi spotzero,
Do you know why that in the file FeedsEntityProcessor.inc, near the end, this line was commented out?
I'm using the patch and it works fine with my ECK created entities until I try to map fields from the source to some location fields, which location_feeds module supports by calling the feeds_processor_targets hook. I copied the module for my own customization and changed the supported type from "node" and "user" to my own ECK types. Originally it didn't work until I find out that the patch commented out the hook calling code.
Just FYI, after I uncomment the code, location mapping is working for me. (location CCK field, attached to an ECK created entity)
Comment #120
yang_yi_cn CreditAttribution: yang_yi_cn commentedalso a minor error, I think might because the Feeds API changed? Getting a lot of warning using the patch 118:
Warning: Missing argument 5 for FeedsEntityProcessor::setTargetElement(), called in trunk/htdocs/sites/all/modules/contrib/feeds/plugins/FeedsProcessor.inc on line 410 and defined in FeedsEntityProcessor->setTargetElement() (line 137 of trunk/htdocs/sites/all/modules/contrib/feeds/plugins/FeedsEntityProcessor.inc).
The original interface in FeedsProcessor.inc is like this:
but in the FeedsEntityProcessor.inc provided by the patch, it has an extra "mapping" argument which is generating a lot of errors.
The "$mapping" is not used in the function anyway, so it should be removed.
Comment #121
Cebra CreditAttribution: Cebra commentedI received an error when importing an XML file where one node of the mapping is missing. It don't know if this problems occurs due to problem in the XPath Parser, the feeds module or this patch here.
Could anybody help me?
EDIT: Seems to be an entity problem. Replacing the entity with a node (custom content type) everythings works fine. But I need entities.
Comment #122
brayo4 CreditAttribution: brayo4 commentedi get this error with patch from #118:
Parse error: syntax error, unexpected T_STRING in \sites\all\modules\feeds\plugins\FeedsEntityProcessor.inc on line 69
Comment #123
yang_yi_cn CreditAttribution: yang_yi_cn commentedBTW all my test are only with CSV imports, and they are going well so far.
Comment #124
imclean CreditAttribution: imclean commentedA combination of #116 and #118 as the entity type selection settings option is quite handy.
I'm trying to get it to work with the Data module for scheduled and manual imports. See #1551430: implement hook_entity_property_info for Feeds integration
Comment #125
imclean CreditAttribution: imclean commentedI'm back to #116 again for testing but the question in #119 is a good one: why is drupal_alter() commented out? This changes the structure of the entity.
Comment #126
imclean CreditAttribution: imclean commentedThe drupal_alter() line is required to enable other modules to modify entity target mappings.
Comment #127
g089h515r806 CreditAttribution: g089h515r806 commentedI have test with the entity processer patch, it does not works for field collection.
So I create a module http://drupal.org/project/field_collection_feeds
Comment #128
rickmanelius CreditAttribution: rickmanelius commentedFYI #124 no longer applied cleanly for 7.x-2.0-alpha7. Working on a re-roll...
Comment #129
rickmanelius CreditAttribution: rickmanelius commentedThere were just some minor cleanup issues to get #124 to apply cleanly to the latest... setting to needs review (particularly with respect #116, which appears to be working for at least one other person).
Comment #130
mzwyssig CreditAttribution: mzwyssig commentedI have this error when I apply the patch:
I applied it on the latest dev version.
Comment #131
kreynen CreditAttribution: kreynen commentedI was able to apply the patch cleanly to both 7.x-2.0-alpha7 and 7.x-2.x-dev using...
The only issue I had after the patch with the base functionality was the same issue I pointed out in #78 where FeedsEntityProcessor.inc is created in the root module dir instead of in the plugins dir with the other processors.
Once installed, I can map to all the fields defined using the Field API. I do NOT see fields I've defined within the entity like title and description. With an entity like file, I can only map to File type and fid. With Comment, I get a much more complete list to fields to map to.
What controls which fields are mappable by default.
Also, when selecting an Entity type in Entity processor Settings, the fields for that entity's form are loaded after the select list option is selected. Values are required for any form fields that are required which seems odd. What's even more confusing is that fields with formatters like date or term reference show up as text fields.
Comment #132
imclean CreditAttribution: imclean commentedThis one of the reasons I went back to the patch in #116. Perhaps give that one a go, using git apply to apply the patch.
Comment #133
kreynen CreditAttribution: kreynen commented#1431952: Date field will only set in Rules if it already has a value
I think this may be an issues for mapping dates on entities in Feeds as well. Text fields work fine. Adding a date to the mapping returns...
Unable to set the data property value as the parent data structure is not set.
Comment #134
twistor CreditAttribution: twistor commentedAlright, I have started a branch, 1033202-entity-processor. That way we don't have to keep re-rolling this.
The current branch is where I think this should be going.
The dropdown for selecting entity types is clever, but is absolutely not going to happen. In fact, I am going to add a feature at some point that doesn't let you change the processor at all if it has imported entities. There's a lot of work around this that needs to be done, that's not directly related to this patch. Please create new issues based on this branch, and we'll reference them here.
I consider this to be the only blocker to a Beta, since we will end up deprecating all other processors, and hopefully removing them before launch.
Repeated: Please create new issues based on this branch, and we'll reference them here.
#1739842: Deprecate some processors in favor of generic entity processor
Comment #135
spotzero CreditAttribution: spotzero commentedTwistor,
Can you clairify:
Does it mean:
Or:
Comment #136
imclean CreditAttribution: imclean commented"Absolutely not going to happen" indicates to me it won't be included. There is a UI issue with it, as mentioned previously. It also makes no sense if other processors are going to be deprecated in favour of the entity processor. In that case you'd have only one processor in the initial list and a drop down consisting of everything else.
Comment #137
spotzero CreditAttribution: spotzero commentedFair enough, I agree that there really isn't a reason to include it if all processor are replaced with this one.
Comment #138
gmario CreditAttribution: gmario commentedhello everybody,
it is from some days that I'm trying to make to work this patch, but apart from of an initial success there is not way :)
I describe what i do:
1) install drupal 7.17
2) downloaded following modules:
job_scheduler (7.x-2.0-alpha3)
schema (7.x-1.0-beta4)
ctools (7.x-1.2)
entity (7.x-1.0-rc3)
views views_ui (7.x-3.5)
data data_ui (7.x-1.0-alpha4)
feeds feeds_ui (7.x-2.0-alpha7)
3) apply patch #129
4) enabled modules in this order:
job_scheduler
schema
ctools
entity
views views_u
data data_ui
feeds feeds_ui
5) from data section create a table (with primary key)
6) from feeds section created a feeds importer with "import from file", "cvs parser" and "Entity processor" but in "entity processor" settings there is not track of table created in 5)
I have done the same try for patch #116
Am I making some mistake? Should I to apply some other patch?
Thanks in advance for any help.
Comment #139
gmario CreditAttribution: gmario commentedan update related to the previous post that may be useful to someone else.
I've taken entity, feeds, data end views modules from their respective repository ( maybe an operation unneeded for all modules) then I've applied to feeds patch in #129. that has solved the problem.
That patch, anyway, doesn't can be applied as is to feeds but it needs to be correct. Here the version that worked with me: patch
Comment #140
imclean CreditAttribution: imclean commentedgmario, your patch doesn't include the .inc file. Also, please see comments from #134 and later. Based on this info, the most recent valid patch is #116. Please ignore mine in #124 and later ones which aren't based on #116.
Comment #141
gmario CreditAttribution: gmario commentedHi imclean,
it is more than a week that i'm trying to make to work data and feeds modules together. I'm read this post, this and this too.
There is a bit of confusion in me now :)
Here what i have done:
installed a fresh drupal 7.17 site
installed ctools, schema, job_scheduler, entity modules in a standard way by using drush, while data and feeds modules from git.
Then accordingly to this comment (posted by you :) ) I've applied patch #116 to feeds and patch #48 (following what that has been said in posts) to data in data I have also applied patch for labels.
enabled ctools, schema, job_scheduler, entity, views, views_ui
enabled data, data_ui, data_entity and feeds, feeds_ui
Now I've created a table from data_ui (doing so an error is appeared in reports - file attached)
Then I have created a feeds importer in which there is not listed "data entity processor". (image attached)
What I doing wrong? :)
thanks a lot for your help.
a si biri
gmario
Comment #142
js CreditAttribution: js commentedI had this working months ago, but because the patch didn't make it into an update, my feeds broke.
For me, the work around was to use
feeds_processor_targets_alter
to add the data fields back into $entity and they are currently saving into the Data table fine.
Comment #143
imclean CreditAttribution: imclean commentedHi gmario,
- Use the Feeds patch from #116 above.
- For the Data module, the patch in #1551430: implement hook_entity_property_info for Feeds integration has already been committed and nothing else required for basic feeds integration. Use the latest dev to be sure.
- Create your Data table and specify a Primary Key within the settings. The PK field must be an integer (or Serial)
- Clear your cache
After the above steps, your data table should appear as a processor ("Entity TableName Processor").
Check Data's issue queue, there are some other useful patches but nothing specifically required for a straight feeds import.
Comment #144
gmario CreditAttribution: gmario commentedThanks imclean, now it works.
your help it was really appreciated.
a presto
gmario
Comment #145
mErilainen CreditAttribution: mErilainen commentedThe patch in 116 worked for me also, but I had to do the same and enable the drupal_alter() in FeedsEntityProcessor.inc to make it work with another patch for file_entity module (See issue #1430190: Add feeds mappers for files for more information).
EDIT: I think there is some issue when trying to update the imported files. I get a database error for each existing image, so the GUID mapping is not working properly. It should update the existing file, not to add a new one.
Error message:
SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'public://path_to_file/12867588.jpg' for key 'uri'
Comment #146
mErilainen CreditAttribution: mErilainen commentedHere is the same patch from #116, but with the drupal_alter() included, because several comments mentioned that it should be included and it works in my case also. Just something that can be referenced.
Comment #147
alibama CreditAttribution: alibama commentedjust to be clear - this patch http://drupal.org/node/1616680 got me through the organic groups referencing when patching my entityreference module (drushed it in today + latest feeds) whereas this patch did not work - just an fyi for anyone else who may follow in the near future
Comment #148
coreycondardo CreditAttribution: coreycondardo commentedTrying to use this patch to import multiple profile entities to attach to multiple users.
Putting a specific UID into the User field of Entity Parser profile settings (see screen shot) gets me one user.
Is there any way that I can put a variable into that field to allow the import of multiple profiles and tie them to the associated users?
The use case being that we have thousands of profiles to import and tie to thousands of users.
Please and thank you.
Comment #149
alibama CreditAttribution: alibama commentedyes - it works fine, however that's a feeds tamper question - use the explode tamper and you should be fine
oh yeah - btw I've never gotten this patch to do exactly what i wanted, so refer to the link three up... that worked like a charm (please excuse me - if someone else has this all dialed in then that's awesome, just haven't gotten lucky myself at least with the organic groups stuff)
Comment #150
imclean CreditAttribution: imclean commentedNo need to keep rerolling the patch, see #134.
Check out the branch 1033202-entity-processor:
http://drupal.org/project/feeds/git-instructions
Comment #151
imclean CreditAttribution: imclean commentedI've had some problems with that branch so am still using #116.
So much has changed with Feeds since that patch was created. I've tried updating it to the current 7.x-2x version on git but am getting memory errors relating to _ctools_export_unpack_object().
Comment #152
geek-merlinCrosslinking #1911158: Fieldcollection Wrappers should autocreate ("EntityMetadataWrapperException - Parent data structure is not set") which should make field_collections work finally.
Comment #153
Todd Young CreditAttribution: Todd Young commentedI have applied the patch in #116 against latest 2.x and it seems to work, but I'm ignoring a fail while patching the .info file.
Is this going to make it into a dev version, or a 3.x, etc?
Comment #154
lolcode CreditAttribution: lolcode commentedSubscribing. Edited: I had posted a comment about an issue with the entity processor branch but after further checking my issue belonged to location feeds.
Comment #155
j0rd CreditAttribution: j0rd commentedSame problem as #120
Comment #156
webadpro CreditAttribution: webadpro commentedWhich patch should we be using on which branch ?
I've tried patch #116 on 2.3 and 1033202-entity-processor but its not applying correctly.
Comment #157
j0rd CreditAttribution: j0rd commented@webadpro, I'm currently using the branch from GIT specific for this patch.
I also tried to merge it with HEAD, but that caused me some problems, so currently I'm just using the branch.
I did a large import (~50,000 or so nodes) and it worked as expected.
Comment #158
webadpro CreditAttribution: webadpro commentedJ0rd, Thats exactly what I have done also. I've used the branch in place.
Thanks for the confirmation also. :)
Comment #159
vinoth.3v CreditAttribution: vinoth.3v commentedApplied the patch #129 against feeds_entity_processor branch.
Fatal error: Call to undefined method EntityValueWrapper::getPropertyInfo() in feeds/plugins/FeedsEntityProcessor.inc on line 152 (If I select EntityProcessor)
with another entity, I was unable to select any fields to match.
Comment #160
kreynen CreditAttribution: kreynen commentedI've been using the git branch without any patches with both http://drupal.org/project/cm_airing and http://drupal.org/project/media_event. It would be great to have a 3.x tag to file specific issues against, but since that doesn't exist I'm adding them here.
- Fields defined in the entity aren't mappable. The only solutions I could find were to write a custom processor or remove the fields from the entity's schema and redefine them using field API.
- Taxonomy's assigned to entity's aren't filterable. This has nothing to do with Feeds, but is something some people working with entities will run into. #1826068: Create an entity generic version of views_handler_field_term_node_tid solved this issue for me. It would be great to see more testing on that so the patch gets into Views.
Other than that, everything is working well with both new entity feeds as well as existing node feeds.
Comment #161
agn507 CreditAttribution: agn507 commentedNoticed a small error on the entity branch. An error in FeedsEntityProcessor.inc on line 76
if ($this->config['authorize'] && !empty($entity->uid)) {
about authorize not being defined. Making sure that authorize was set in the defaults fixed the problem.
Comment #162
jshimota01 CreditAttribution: jshimota01 commentedsubscribing d7 +1
I'm interested in testing this as it moves further along. I have a devel machine setup for it now using php5.29, sql and apache (wamp 2h)
Comment #163
j0rd CreditAttribution: j0rd commentedI think the branch has a bug where you're the appropriate defaulted "Entity Processor" is never set via the feeds_ui.
I've created a patch which resolves the issue for me.
I think it may be the same problem as experienced by comment #3
There appears to be another problem where you can not map fields on the entity.
This patch is created against the entity-generic branch.
PS. I'm using messages as my entities.
Comment #164
vordude CreditAttribution: vordude commentedOn the Access check before entity save (the method entitySaveAccess()) if $this->config['authorize'] isn't set, then you get a warning.
Here's a patch to make the warning go away.
I think this doesn't create an issue by bypassing a check if it does, slap my hand and I'll fix it.
Comment #165
franzComment #167
MegaChriz CreditAttribution: MegaChriz commentedI think the patches are made against the 1033202-entity-processor branch, therefore they fail to apply on the 7.x-2.x branch. Setting issue status back to "needs review".
Comment #168
j0rd CreditAttribution: j0rd commentedMy patches are against the 1033202-entity-processor branch, which is the one I think most people in here are using.
I personally can't really patch against -dev....as it's moved on since this generic entity branch was created and work when merged with -dev.
Comment #169
franzWell, can't you try to merge with -dev, fix it and post a patch? We'll have to do it sometime anyways. Or you can rebase from -dev.
Comment #170
MegaChriz CreditAttribution: MegaChriz commentedIn #167 I didn't meant the patches should be made against 7.x-2.x-dev (at least, not right now), I meant that the tests were failing because the testbot was testing the patch against the wrong branch, resulting in the (in this case) wrong issue status "needs work", so I had set the issue status back to "needs review".
I agree with franz that at some point the patch should be made against the 7.x-2.x-dev, but I think it should be done when the feature is considered complete. I have not read the whole issue and therefore I'm not sure in what state the feature currently is. Hm, sounds like this issue could use an issue summary. I add the tag "needs issue summary" to this issue.
Comment #171
imclean CreditAttribution: imclean commentedReading the comments by twistor in #134:
He even has an example. The only problem is the new branch can't yet be selected in the version dropdown for a new issue.
Comment #172
MegaChriz CreditAttribution: MegaChriz commented@imclean
Alright, that means that this issue should be used as a sort of "index" issue which links to several subissues. This is something that should be noted in the issue summary (I can not create one right now, cause I have not read the whole issue yet). A workaround for the problem that the branch can't be selected is to create a link to this issue in each subissue (on the top of their issue summary), for example:
Comment #173
franzI added a component, hope this helps. Then new issues can go in there.
Comment #174
franzJust to clarify, the "Version" drop-down can only select branches/tags that follow the naming conventions for releases.
Comment #175
imclean CreditAttribution: imclean commentedI have to admit, I still don't understand the future of the new branch. Feeds has changed so much since it was created.
How does or will the entity branch relate to the main Feeds branch?
Comment #176
j0rd CreditAttribution: j0rd commented@imclean, correct me if I'm wrong, but I still don't believe you can import generic entities with the main branch?
I could be wrong, but I tried a massive import a couple weeks ago where I needed it, and the main branch didn't have this feature.
Maybe I didn't know where to look, but that seemed to be the case....and I went back to using this branch.
Comment #177
imclean CreditAttribution: imclean commentedYou're right, it doesn't support generic entities yet. But I suspect any code developed in the new branch would need a bit of a rewrite before it could be included in the main branch. That was my main concern, I'm not really sure how these things work.
Comment #178
rooby CreditAttribution: rooby commentedI am using the special branch and the latest version of the data module and I get the option to use my data entity however not all the fields are available as targets.
I have a data table with 2 columns. An int field (PK) and a varchar field.
Only the int field is available as a feeds target.
Can anyone shed some light on why?
Is it safe to assume that this new branch will eventually be rolled into the main one?
Comment #179
rooby CreditAttribution: rooby commentedSeems the reason is that in FeedsEntityProcessor::getMappingTargets() it does:
It never passes
empty($property['field'])
because $property['field'] is always set to TRUE for my data table columns.Surely other people are seeing this, or is it just me?
Comment #180
rooby CreditAttribution: rooby commentedRelated - I have made a patch to address the issue in #179 for the data module #1957022: Entity property info declares the table columns as fields - This feeds entity support doesn't work properly for me with the data module without it.
Comment #181
juampynr CreditAttribution: juampynr commentedWe had to restore Feeds plugin info files as per #1201638: Plugins should be listed in info file, but the branch works well so far.
Are we far from merging this branch on top of 7.x-2.x? People using it (including me) will miss fixes/new features from the stable branch and the more time we spend the harder it will be to merge.
Comment #182
Steven Jones CreditAttribution: Steven Jones commentedSorry to add to the noise, but here's a patch that fixes the strict warnings about the method declarations being inconsistent.
Comment #183
xjmTag renaming. :)
Comment #184
xjmComment #185
kyleoliveira CreditAttribution: kyleoliveira commentedThis looks like it would be super handy, but I'm kind of lost in all the comments. How close is this to being put into the main branch?
Also: Has anyone figured out how to map multiple subentities in the same import? (e.g. a single importer that would import Users and their associated Profile2 profile(s)). Yes, I've seen the Migrate-based solution, but I'm trying to do this all with Feeds.
Comment #186
j0rd CreditAttribution: j0rd commented@kyleoliveria If someone was merging this into mainline, we'd be seeing patches.
So far it looks like most people in this issue are simply happy to continue to use the branch.
Eventually I assume someone will step up and merge it (like i've tried), but so far this hasn't happened.
I personally need some fixes on this branch, so that I can get my RSS feeds importing properly into message entities (used to work, but broke)...so I'm waiting on a fix or a merge. Eventually I'll get around to doing it myself if no one else steps up, but that won't happen until I finish a bunch of other work.
Comment #187
stopshinal CreditAttribution: stopshinal commentedI'm struggling to use this branch successfully. What is the structure of the data Feeds expects for an entity reference?
I have an entity reference field that links to a taxonomy. I've tried providing feeds with the vocabulary name as well as the vocabulary ID for this field. It does not seem to import.
Is there an entity ID I should be using instead?
Ideally I would like to have the feed creates new taxonomy if it is not already present. I'm not sure how I am going to get over the hurdle of having all of my data (taxonomy vocabulary) in plain text - and not in ID's.
Comment #188
rerooting CreditAttribution: rerooting commentedQuick thought - have you looked at http://drupal.org/project/feeds_entityreference by chance?
Comment #189
rerooting CreditAttribution: rerooting commentedAnd http://drupal.org/project/feeds_tamper_string2id per some discussion in the issues.
@Stopshinal I think your question pertains more to mapping to entityreference field types, whereas this issue thread is a discussion and development of a generic entity processor for feeds, which is a somewhat related but fundamentally different topic. I hope these links are helpful for you (I'm about to tackle the same issue today myself!)
Comment #190
stopshinal CreditAttribution: stopshinal commented@rerooting - thanks so very much.
I suppose I'm curious what this new branch solves for me (in the context of my destined goal) Otherwise I should probably just move to the regular dev branch, correct?
I'm excited by these two other thinks you provided. I was aware of the feeds_entityreference, but though this branch solved those issues. The feeds_tamper_string2id itringues me!
Which combination will you be using? (Which feeds branch, etc)
Comment #191
stopshinal CreditAttribution: stopshinal commentedUsing the entity branch of feeds, as well as the string2id has worked PERFECTLY for me. Not sure about any other branches.
Comment #192
james.williams CreditAttribution: james.williams commentedImporting feeds using this entity processor fails when there are already existing entities that are not in the feeds_item table (e.g. when they existed before feeds was being used to import that type of entity).
This is because the FeedsEntityProcessor does not implement the existingEntityId() method. Note that the FeedsNodeProcessor does do this, and checks the node table for any matching node IDs or titles.
I've produced a rough solution, based on the FeedsNodeProcessor that checks for the ID, but given that every type of entity has potential to have quite different data structures, it won't always work.
Comment #194
james.williams CreditAttribution: james.williams commentedIn comment #182, there's a patch to fix the problem raised in comments #120 & #155 about missing arguments. That one missed a couple of calls to setTargetElement() that also missed the new argument. Here's a patch that includes those calls.
Comment #195
j0rd CreditAttribution: j0rd commentedfor testbot.
Comment #196
guillaumev CreditAttribution: guillaumev commentedHere is an attempt to merge the entity_processor branch back into the 7.x-2.x branch.
Comment #197
j0rd CreditAttribution: j0rd commentedMuch appreciated @guilaumev. I'll try this out next week and let you know how it goes. I recommend others do the same.
Comment #198
guillaumev CreditAttribution: guillaumev commentedJust pushing this up as it would be great if this patch could be committed before it can not be applied anymore because of other changes committed...
Comment #199
j0rd CreditAttribution: j0rd commented@guillaumev could you post the commit in 2.x-dev you patched against for others future reference. I don't believe testbot has this information.
Comment #200
j0rd CreditAttribution: j0rd commentedAlright, I've tested @guillaumev's patch for the generic entity processing. While it applies cleanly, it has a minor issue with the bundle handling. I've resolved that. I'm attaching two files to this post.
feeds_entity_processor-1033202-196-200.diff.patch
This is the differences I've added to guillaumev's patch.
feeds_entity_processor-1033202-200.patch
All the changes from 196 with my minor fixes.
With my patch, I can now import RSS feeds into "Message" entities provided by the message module (aka a generic entity).
So for me, this works. Can we get others to test. There's many other people in here which need this patch. We need to get this in soon before this patch and -dev diverge again.
Comment #201
stewart.adam CreditAttribution: stewart.adam commentedLooks pretty good, I had a small issue with it though: when selecting the 'Entity processor User', I was prompted to select a bundle when editing the mapping. When clicking the link to bring me to the processor's setting page, there is no option to select a bundle (probably because only one exists). When selecting 'Entity processor node', I do see a bundle selector with each of my content types listed.
Comment #202
aaronpinero CreditAttribution: aaronpinero commentedI am working in Drupal 7.22. I have created user profiles using the core Profile interface. I would like to be able to create and update users by importing a CSV file using Feeds that will populate the user profile fields. I applied the patch in #200 to the feeds module but I am still not seeing data from an imported CSV file applied to the correct profile fields. There are no errors reported by Drupal (in fact, the Feeds importer tells me that my users were successfully updated) but a review of the individual user accounts shows that this is incorrect.
Comment #203
kreynen CreditAttribution: kreynen commentedOverall, the patched version of the Feeds 2.x branch works well for importing file entities directly using MediaRSS. I made one minor change to expose the Feeds views handlers to File views so that we could filter files by the source feed. The only other thing I noticed was in the UI. When using the Entity File Parser, the Bundle option is displayed twice.
This was added to the patch...
Comment #204
j0rd CreditAttribution: j0rd commentedgetting testbot in here.
Need more people to install these patches and test. Let's get this in quick before branches diverge again.
Most importantly, since this is adding new functionality, is making sure it doesn't regress previous behaviour. Does everything else not using this new processor continue to work for you guys?
Comment #205
kyleoliveira CreditAttribution: kyleoliveira commentedI'm getting "Invalid data value given. Be sure it matches the required data type and format." errors when I try to import Users with the Entity processor User processor and the last patch. It looks like it doesn't like email addresses despite the fact that I'm using normal email addresses and I've set the mappings (seemingly) correctly. I'm using the Feeds XPath module to read in an XML file, but it seems to be working correctly as well. Has anyone else successfully imported users' email addresses with the last patch? I'm doing this on simplytest.me, so I may just have set something up incorrectly.
Comment #206
kreynen CreditAttribution: kreynen commented@kyleoliveira
Does importing users work with the XML and the "old" User import? If you export your Feed Importer and attached that along with a sample of the XML with just demo data, I'd be happy to test.
Comment #207
kyleoliveira CreditAttribution: kyleoliveira commentedIt does work with the "old" User import. Also of note, the Entity-based importer does not allow you to set the Email or Username as Unique whereas the old (classic?) version does and the Entity-based importer lets you map a target for uid (which I haven't tested yet).
I've attached an example xml file. The importer is as follows:
Comment #208
that0n3guy CreditAttribution: that0n3guy commented#203 is forcing me to specify an entity ID (which is bad). See the attached screenshot.
The entity is created via ECK.
Comment #209
kreynen CreditAttribution: kreynen commentedObviously there are additional issues that need to be resolved to get the entity processor merged with the 2.x branch, but trying to track progress on resolving these as comments in a thread with 200+ comments in an issue started 2+ years ago is going to be difficult. So far we've identified 3 "merge blocking" issues...
None of these seem like they are going to be very difficult to resolve, but I'm guessing new issues will also be found as more testing is done. The 2.x branch of Feeds is still in alpha, but stable. An unpublishable branch for the entity work already exists, but new issues can't use that as their version nor can non-git savvy users help test that branch. While the generic entity processor is available as a Component for issues, breaking these issues up and working on them independently based on a patch this large is going to create a lot of work merging changes back into a single patch... and the 7.x-2.x branch can continue changing.
Adding entity support seems like an improvement worthy of a dev-only, 3.x branch. Once the entity work is completed, those changes could be back ported to the 2.x release.
While we've had 5 or 6 developers contributing to this patch in the last 6 months, without version control, packaging, and issue tagging it seems unlikely that we'll be able to get the momentum necessary to finish this. I created #2054709: Create 3.x branch for Entity Processor requesting a 3.x branch be created and someone w/ maintainer access commit to reviewing patches related to the entity processor.
Comment #210
MegaChriz CreditAttribution: MegaChriz commented@kreynen
Another possibility is to move the work over to a sandbox project, so a 3.x branch can be reserved for, well a complete new version.
Comment #211
kreynen CreditAttribution: kreynen commentedSandboxes don't have packaged downloads or version stamped releases which limits testing to git-savvy users. If we can't manage this within the Feeds project, GitHub is a better alternative to Drupal.org sandboxes.
Comment #212
guillaumev CreditAttribution: guillaumev commentedThe issue I see with creating a 3.x branch of Feeds to add the Entity processor is that, given the number of comments in this issue, it seems like many people need this sooner rather than later, and I doubt this 3.x branch will be released soon.
I might be missing something here, but what about doing the following:
My point here is that:
We could eventually add an "(EXPERIMENTAL)" in the labels of the entity feeds importers, so that we make sure users know the risks, but I think creating a 3.x branch for this is being a bit too careful maybe.
Comment #213
j0rd CreditAttribution: j0rd commentedI'm with Guillaumev on this. Providing this new processor does not break any of the old ones, it should be merged in to 2.x . Then additional bugs could be found and fixed as needed.
I'm personally running the patch on production right now, and it's working fine for my needs.
Comment #214
guillaumev CreditAttribution: guillaumev commentedAlso running the patch in production, and working fine...
Comment #215
kyleoliveira CreditAttribution: kyleoliveira commentedSounds good to me (that's what dev is for anyhow, right?). Who needs to do what to make this happen?
Comment #216
dsnopekWell, we can mark this patch as RTBC. :-) Then we need to get the maintainer to take a look at it and hopefully commit it.
Comment #217
kreynen CreditAttribution: kreynen commentedGod I love community driven decision making :)
I wasn't suggesting pushing the 3.x branch to a stable release, but using the branch as... oh, I don't know... a branch to work on this feature and then merge the 3.x work with any changes made to the 2.x branch while the remaining bugs are resolved. Drupal's move to git was a big improvement over CVS, but the link between packaging dev snapshots, issue tagging, and git tags is still really limiting. I'm suggesting options that work with Drupal.org's limitations rather than push changes with known bugs into the current, relatively stable branch.
I think everyone following this issue wants to see entity support in Feeds, but I know I have no idea what issues are preventing a beta release of the 2.x branch. I didn't want to make that work more difficult and don't have time to contribute to it so I was suggesting an option that got me what I wanted without causing more problems for someone else. I don't care if this gets committed to 2.x or 3.x, but that it gets committed or a list of what needs to be corrected before it will be committed is created.
We've been using the entity patch, then branch, and now patch again with media driven sites in production environments since March. The entity processor has issues, but it's better than alternatives for directly importing media/file entities...
https://vimeo.com/67261530
We just launched http://okv.se/play using File Entity processor to directly import 1,500+ YouTube videos to file/media entities.
I think the suggestion to label the entity processors as EXPERIMENTAL is great. I've updated the patch to include that in the title and reiterate it in the processor's description.
If the maintainers won't commit this to either the 2.x or a new 3.x branch, can we at least get a list of what would need to be resolved before this can be committed so we can work through that list?
Comment #218
inventlogic CreditAttribution: inventlogic commentedJust checking regarding patching of the git version of feeds 1033202-entity-processor that includes this patch.
Is that a feature frozen version that will receive no further updates for future bug fixes or is it an active branch that is patched?
Comment #219
kreynen CreditAttribution: kreynen commented@inventlogic,
The entity processor branch hasn't been updated in 8 months... http://drupalcode.org/project/feeds.git/log/refs/heads/1033202-entity-pr...
So it's missing everything that's been committed to the 7.x-2.x branch in the last 8 months http://drupalcode.org/project/feeds.git/log/refs/heads/7.x-2.x
AND the improvements that have been added to this patch like being able to related entities to their parent Feed the way you can w/ nodes.
I think most developers who have contributed to the entity processor are now using the current 7.x-2.x brach w/ this patch.
Comment #220
inventlogic CreditAttribution: inventlogic commented@kreynen
Thanks for the info.
Reading through this issue I figured the patch at #116 was still the valid one to use against 7.x-2.0-alpha8.
However it failed on several chunks.
Which patch should I be using?
Comment #221
kreynen CreditAttribution: kreynen commented@inventlogic, I'm patching against the 7.x-2.x branch from git, but that is currently the same as the dev snapshot since nothing has been committed since July 6. The patch won't apply to 2.0-alpha8 because of the changes made between April and July. If you really want to help test this patch, I would encourage you to clone 7.x-2.x from git https://drupal.org/project/feeds/git-instructions.
Comment #222
kreynen CreditAttribution: kreynen commentedMedia 2.x went alpha1 yesterday!
Would be really great to have a non-patched solution to direct file_entity field mapping and imports. @twistor has been committing regularly to the 8.x branch, but nothing's been done with the 7.x branches in 30+ days. 1139 open issues in the queue is also a little overwhelming. Anything people interested in seeing this move forward can do to reduce that number makes it easier for maintainers to focus on issues that are RTBC.
Comment #223
petria CreditAttribution: petria commentedCan anyone help me to get this patch working with external images?
(https://drupal.org/node/2066741)
Comment #224
gmasky CreditAttribution: gmasky commentedI am trying to import data into an entity form.
I applied the patch at #217
At Processor-change-Entity processor Entityform - EXPERIMENTAL I get the following message
and
The Settings for Entity processor Entityform is blank and
Mapping gives me the error Fatal error: Call to undefined method FeedsMissingPlugin::getMappings() in C:\wamp\www\aloysius7\sites\all\modules\feeds\feeds_ui\feeds_ui.admin.inc on line 491
Any help will be highly appreciated
Comment #225
guillaumev CreditAttribution: guillaumev commented@gmasky Did you try clearing the caches ?
Comment #226
gmasky CreditAttribution: gmasky commentedYes I have cleared the cache. I get the message
on the front page after I clear the cache
Comment #227
jrao CreditAttribution: jrao commentedThis patch doesn't seem to work with entity properties of the type entity, the default values form validation can't handle it, and even if we bypass that, the mapping doesn't seem to work either.
Comment #228
jamesdixon CreditAttribution: jamesdixon commentedI was able to get this working with 7.x-2.x for my entity. I used kreynen's patch in #217 and added some code to fix:
EntityMalformedException: Missing bundle property on entity of type partprofile. in entity_extract_ids() (line 7693 of ...
Comment #93 was having this issue as well. The problem was the bundle was getting missed when the entity info hook has $info['entity keys']['bundle'] set.
Comment #229
tannerjfco CreditAttribution: tannerjfco commentedTested patch in #228 against latest 2.x-dev, I'm also getting the same error messages as mentioned in #224, along with the following warning:
Missing Feeds plugin FeedsEntityProcessorNode. See exhibitor_list. Check whether all required libraries and modules are installed properly.
Comment #230
kreynen CreditAttribution: kreynen commented@gmasky and @Tannerjf, can you provide the exports from your exhibitor_list and entity_importer Feed Importers or the steps someone can use to reproduce this error? Did these importers exist before the patch? Were they created with a previous patch applied?
Comment #231
apmsooner CreditAttribution: apmsooner commentedI applied the patch from #228 to most recent dev version of feeds. Patch applied fine, I see the experimental processors, etc... however I get the same message stated previously about missing plugin. This is what it says in my log entries:
Plugin Entity processor File - EXPERIMENTAL of plugin type feeds:plugins points to nonexistent file sites/all/modules/feeds/plugins/FeedsEntityProcessor.inc for class handler handler.
I didn't have feeds previously installed on this test site nor was there any previous patches applied. Only the one from #228. Hopefully that helps....
Also, these errors appear on the feeds admin ui:
Comment #232
apmsooner CreditAttribution: apmsooner commentedHere is my export although there really isn't much to it as I can't modify settings and mapping fields leads to WSOD.
Comment #233
jamesdixon CreditAttribution: jamesdixon commentedI apologize, I'm pretty new to writing patches. I forgot to include --staged in git diff so it would include the new file. Please try this one.
Comment #234
apmsooner CreditAttribution: apmsooner commentedThanks @jamesdixon, that patch took care of the issue. Not sure how to create a file entity processor though, it's asking for required fields for file id, file size, file owner and feed nid. What are you doing for that?
Comment #235
apmsooner CreditAttribution: apmsooner commentedI can put bogus values in there just to get the Entity Processor File settings form to save but trying to import gives an error of: Entity property name doesn't support writing.
If it can't write the fileid and it's required in the settings form, not sure how I could use the fileentity processor. Anyone else tried this and have tips?
Comment #236
jamesdixon CreditAttribution: jamesdixon commented@apmsooner: The required settings form may need work. I think what's happening is the EntityProcessor is allowing you to specify default values for each field if your Feeds Source doesn't have a value for that field.
It must be pulling these form items from the entity form config, and if the entity form config has a required value, then it will require a default value. We should probably make all those default value settings optional.
I have not tried using fileentity with this feed processor. What is the exact error message you are receiving, is it "Entity property name doesn't support writing."
Comment #237
apmsooner CreditAttribution: apmsooner commentedYup,
Entity property name doesn't support writing.
This error occurs even if I don't map anything but the guid. The default required values is the root of the problem from the start. Not sure what will happen after the required restriction is removed but happy to test out if we can get to that stage.
Comment #238
markusa CreditAttribution: markusa commentedGreat feature for the feeds module.
Already used the branch discussed in the issue to successfully import a list of taxonomy vocabularies.
To be able to import as any entity is awesome!!
Thanks for you your work
Comment #239
jshimota01 CreditAttribution: jshimota01 commentedWhat chance this patch (or the 116 patch) get put into the current dev for availability to non-programmers? I could really use this patch without having to learn how to patch!
Comment #240
gmasky CreditAttribution: gmasky commented@jshimota01 patching is not that difficult. This is helpful if you use Windows https://drupal.org/node/620014
Comment #241
scottalan CreditAttribution: scottalan commentedI have been trying to get the patch in #116 to apply but have been unsuccessful thus far. Has anyone else had an issue getting the patch to apply?
Tried using drush:
and
curl http://drupal.org/files/feeds-entity-processor-1033202-116.patch | git apply -
Errors:
Comment #241.0
scottalan CreditAttribution: scottalan commentedcrosslinked
Comment #242
jamesdixon CreditAttribution: jamesdixon commented@apmsooner - I believe the required fields issue existed earlier, and by opening up the allowed fields for mapping my patch made it worse. I've added some code to make those default mapping fields optional.
Also my two previous patches needed running together to work properly, I combined them into a single patch attached with the code to remove the required default mapping fields.
Comment #243
jamesdixon CreditAttribution: jamesdixon commentedComment #244
jamesdixon CreditAttribution: jamesdixon commentedThis last patch should fix any "Entity property [propertyname] doesn't support writing" people were having. Like #235
Comment #245
dobe CreditAttribution: dobe commentedPatch #244 fixed my "Entity property [propertyname] doesn't support writing" as well as cleaned up the default settings.
Comment #246
sisko CreditAttribution: sisko commentedApologies for this ultra newbie question but to which file(s) do I apply the patch?
Comment #247
sisko CreditAttribution: sisko commentedApologies for this ultra newbie question but to which file(s) do I apply the patch?
Comment #248
jamesdixon CreditAttribution: jamesdixon commented@sisko you download the 7.x-2.x-dev branch and apply the patch to it. Here is a link to download http://ftp.drupal.org/files/projects/feeds-7.x-2.x-dev.tar.gz.
Comment #249
sisko CreditAttribution: sisko commented@JamesDixon: Thanks for clarifying.
Just confirming the patch is executed against the entire Feeds folder I:E
patch feeds.patch feeds/*
???Comment #250
jamesdixon CreditAttribution: jamesdixon commented@sisko apply the patch inside the feeds folder. I am a noob when it comes to patching so I used git instructions to apply patches here https://drupal.org/project/feeds/git-instructions
Comment #251
sisko CreditAttribution: sisko commented@JamesDixon: Thank you for the assistance. It's very appreciated. I shall give your directions a try asap.
Comment #252
jamatulli CreditAttribution: jamatulli commentedI've been able to get the Entity Processor to show up for a custom table created with the Data module. Unfortunately the mapping can not be completed because it only shows the first field in the table.
Comment #253
jamatulli CreditAttribution: jamatulli commentedCorrection: It is getting the first field marked Primary Key.
Comment #254
jamesdixon CreditAttribution: jamesdixon commented@sisko Great let us know if it works for you.
@jamatulli I had the same issue with earlier patches. Please try version 7.x-2.x-dev with patch 244 and let us know if it works.
Comment #255
imclean CreditAttribution: imclean commented@jamatulli, there are a number uncommitted patches for the Data module which may help. Please try the label patch and the primary key patch from the Data module's issue queue.
#1538588: Add Entity Label field (Was: allow entity ref fields to point to data items)
#1671380: Data Entity allows entities to be created with entity ids that are not integers
Comment #256
jamatulli CreditAttribution: jamatulli commentedCurrently running 7.x-2.0-alpha8+8-dev which is the current dev and this was patch #244. I will take a look at those Data Entity issues and patches. It will most likely take me a couple days.
Comment #257
dobe CreditAttribution: dobe commentedThe patch worked good for using feeds to import some entities. I was unable to get it to work with commerce orders though.
Comment #258
agoradesign CreditAttribution: agoradesign commentedThe patch worked also good for us, importing some ECK-made entities
Comment #259
jlyon CreditAttribution: jlyon commented@imclean: Thanks! feeds-7.x-2.x-dev with the patch from #244 and your two patches from #255 seem to be working pretty well for me (I'm importing from and xml file into a data table). I had to manually apply the patch from #1671380: Data Entity allows entities to be created with entity ids that are not integers because it conflicted with the first one.
I'm still having getting a PRIMARY KEY error when I try to update the enties by running a second import without first deleting the existing data, though:
This is likely a data module issue, so I'll follow up in one of those queues once I've had some time to track down the cause.
Comment #260
imclean CreditAttribution: imclean commented@jlyon, good to hear. Could you please provide feedback in both those issues?
In regards to the integrity constraint violation, check the actual schema of your feeds_item table:
https://drupal.org/comment/6757420#comment-6757420
Comment #261
jamatulli CreditAttribution: jamatulli commentedFinally got around to applying the two patches for Data and I had no luck. I also had to apply the second one line by line and assumed they were additive.
I imagine what became data_entity.admin.inc:line 81 should be what is prescribed in the "label callback" patch (i.e. $header = array(t('Name'), t('Locked'), t('Required'), t('Label'));) as opposed to being additive (i.e. $header = array(t('Name'), t('Locked'), t('Required'), t('Label'),t('Entity ID'));). This seems rather inconsequential to the issue I am having though and predictably neither version had an impact.
I may try to restart from the beginning and see if I can get it work.
Comment #262
imclean CreditAttribution: imclean commentedBecause neither of the Data patches I liked to have been committed each one needs to be applied to a clean version of Data from git.
They're not large so the second one you do could be done by hand.
Comment #263
jamatulli CreditAttribution: jamatulli commentedSuccess. At least with the listing of particular fields from a Data table. I used #1538588: Add Entity Label field (Was: allow entity ref fields to point to data items) but not the other patch. It was probably something in my hand application that wasn't working.
Comment #264
lquessenberry CreditAttribution: lquessenberry commentedAre there pushes to implement this into Feeds permanently? I sure would love to be able to safely import CSV files into my own ECK Entities without patching.
Comment #265
dobe CreditAttribution: dobe commentedAs far as I know. But it needs testing from the community. Please patch and let us know if you find any issues with ECK.
Comment #266
lquessenberry CreditAttribution: lquessenberry commentedThanks! I am testing it now and so far it's doing really well.
Comment #267
bobojo CreditAttribution: bobojo commentedI have been working on this all day, and I just can't figure out what's wrong. I'm trying to create some forums using the Entity processor for taxonomy terms. It says that it has created the terms, but when I looked in MySQL, only the taxonomy_heirarchy table is being changed. I'm using Feeds SQL with some very simple mapping to set the tid, name, description, and parent. Any ideas why this wouldn't be working properly?
Edit: I just got it to work by removing the mapper to manually assign a tid. I need this mapper to work, so I'm going to keep testing it, but I thought it would be good to mention that in case it helps find the problem.
Comment #268
geek-merlinProfile2 mapper: Did not succeed updating profil2 entitite; mapped user id did not overwrite the predefined one.
EDIT: it looks like the problem is that i can't set UID as unique target.
Comment #269
geek-merlinNevertheless i'm all for the approach outlined in #212:
* commit this as experimental
* handle the finetuning in dedicated issues
So setting RTBC to attract maintainers.
Comment #270
kreynen CreditAttribution: kreynen commentedIn 10 days this issue will be 3 years old. Despite contributions from dozens of developers, this doesn't seem to be any closer to being committed. It has been moved to RBTC several times. The first was almost 2 years ago https://drupal.org/comment/5524356#comment-5524356
@twistor is the only active maintainer and hasn't responded to any of the patches that would add entity support in over a year (https://drupal.org/comment/6798294#comment-6798294).
The last push to get something committed was 5 months ago and despite reaching out to @twistor directly to review the patch, there was still no response https://drupal.org/comment/7710367#comment-7710367
Judging by @twistor's commits, he seems focussed on a D8 release. That is great, but doesn't solve the problem of entity support for D7.
If none of the current maintainers are interested in supporting this feature in this project, I think it's time to add additional maintainers or fork the project to entity_feeds or something like that. Trying to troubleshoot issues with specific entities based on patches really isn't an option.
We've been using a patched version of Feeds to import Media entities directly (vs. using something like https://drupal.org/project/media_feeds to add media to a field using a node) for almost 2 years. I can't speak for all possible use cases, but entity support is very reliable for Media.
Comment #271
twistor CreditAttribution: twistor commentedI shall take a gander soon.
Comment #272
alibama CreditAttribution: alibama commentedseriously - chuck this dude a few bucks. https://www.gittip.com/twistor/ just sayin'
Comment #273
twistor CreditAttribution: twistor commentedAlright, I made a few changes and went ahead and committed this with an experimental tag on the processors.
Let's start with the followups.
Comment #274
twistor CreditAttribution: twistor commentedBig thanks to everyone who helped. I know there was a lot of people.
http://drupalcode.org/project/feeds.git/commit/f6aa61d
Comment #275
guillaumev CreditAttribution: guillaumev commentedAwesome, thanks a lot !
Comment #276
fagoheureka!
Comment #277
Patrick Danielsson CreditAttribution: Patrick Danielsson commentedThanks, this is amazing!!
But I can't find the fields for Profile 2 when I want to map the fields, what is wrong? I can only choose the "standard fields", like ID, username etc.
I have chosen the Profile processor, and the Main profile.
Thanks!
Comment #278
dasjoyay, glad to see this in now :)
after 278 comments, i'd suggest opening one or more follow up issues because this one's already too big
Comment #279
geek-merlinYAY!!!!!
Comment #280
likewhoa CreditAttribution: likewhoa commentedMission Accomplished!
Comment #281
alibama CreditAttribution: alibama commentedwell - entityforms don't actually work yet, but it's at least a start...
Comment #282
kreynen CreditAttribution: kreynen commented@alibama Please open an issue specifically for any entity that can't be imported in the Feeds queue or that module that controls that entity.
To use the File Entity processor w/ Media requires an additional module like https://drupal.org/project/feeds_media_internet_files to handle mapping the Filename. Without that, Filename shows up the mapping UI, but is empty in the resulting file entity.
It's possible a helper module like that will be required for entityform, but I haven't look at that entity specifically.
Comment #283
ndf CreditAttribution: ndf commentedGreat!
Comment #284
scottrigbyExcellent!
twistor++
Comment #285
Patrick Danielsson CreditAttribution: Patrick Danielsson commentedMaybe i'm the one who does something wrong, but I can't add any Data Processors, i've done everything correct if I read the thread, and I have made a new Data table in the Data module, but still nothing is showing up in Feeds.
Comment #286
Leo Pitt CreditAttribution: Leo Pitt commented@Patrick Danielsson - are you using the 7.x-2.0-dev branch?
Comment #287
Patrick Danielsson CreditAttribution: Patrick Danielsson commentedI'm using this version of Feeds:
http://drupalcode.org/project/feeds.git/commit/f6aa61d
And i'm using Data 7.x-1.x-dev, is there another version I should use.
Comment #288
kreynen CreditAttribution: kreynen commentedPLEASE open new issues for problems w/ Data (or really problems w/ anything). This specific issue has been considered fixed since Entity support was added to the 7.x-2.x branch. 7.x-2.0-dev isn't a branch. It's a snapshot release of the 7.x-2.x branch. You should be using a dev snapshot for 2014-Jan-16 and use 7.x-2.x-dev when tagging the issue.
Comment #290
travist CreditAttribution: travist commentedThis commit causes some major problems when installing the Taxonomy module after the feeds module is enabled.
Please see #2223853: Installing taxonomy module after this module causes a fatal.
Should I reopen this issue, or just move forward with the issue mentioned above?
Comment #291
rooby CreditAttribution: rooby commentedStick with the other issue.
Comment #292
robertwb CreditAttribution: robertwb commentedPEBKAC error RESOLVED - sorry for the spam.
I am unable to see any parsers for entities such as Profile2 in my Feeds Importer config screens. Structure -> Feeds Importers - [My Feed Importer] -> Processor .
I have applied the most recent dev module and verified that the code from this patch seems to be in there (found "node_form_feedsentityprocessor_feeds_form_alter" anyhow).
Is there something really simple that I am missing?
Comment #293
crystaldawn CreditAttribution: crystaldawn commentedWhat needs to be done to get Profile2 entities to show up in the list of processors. I can see several but none list Profile2. These are a few I can see:
Content entity processor - EXPERIMENTAL
Node processor
The above 2 processors to me would indicate that this patch is functioning (I am using the dev branch) since I can see 2 processors that are essentially the same thing. Node processor and Content entity processor are pretty much identical. I would expect to see a Profile2 EXPERIMENTAL processor as well, but it's not in the list. Why is it not listed?
Is this left up to the profile2 project (which will likely never happen), or is it suppose to show all entity api defined entities (which is what I would expect, and thus this patch should display profile2 defined entities).
Comment #296
dsnopekNew issues should be new issues! Marking as closed again.
Comment #297
twistor CreditAttribution: twistor commentedFor those following along at home, this is being reverted. Weigh in here #2469219: Remove the Generic Entity Processor.
Comment #298
chintan4u CreditAttribution: chintan4u at Tieto commentedHi @twistor,
I gone through this whole post. Still I'll need some help on patches.
I have drupal 7 web application for support (With No Documention :( )
I updated feeds module 7.x-2.0-alpha9 + feeds-entity-processor-1033202-129_0.patch
Earlier this D7 Website was running on 7.x-2.0-alpha7 + feeds-entity-processor-1033202-129_0.patch
When I updated to 7.x-2.0-alpha9 + patch, Some taxonomy filed are NOT getting created using FeedsXPathParserXML, It was working file earlier.
Any suggestions ??
Comment #299
MegaChriz CreditAttribution: MegaChriz at WebCoo commented@chintan4u
It is recommended to use the Feeds Entity Processor module now (instead of the patch posted here). See also the Change record.
Comment #300
chintan4u CreditAttribution: chintan4u at Tieto commented@MegaChriz Thanks, I will give it a try.
Comment #301
chintan4u CreditAttribution: chintan4u at Tieto commentedHi All, hi @MegaChriz,
I gave try to sparate Feeds Entity Processor module but It won't help to create new taxonomy terms.
(Things are messed up with pathches and our old custom code)
I really don't have any problem with 7.x-2.0-alpha7 + feeds-entity-processor-1033202-129_0.patch as it works as per our need.
Lets say if I decided to patch security fixes on 7.x-2.0-alpha7 itself instead of updating to newer versions. Shall I add below commits to existing code ? Will this suffice ?
Changes since 7.x-2.0-alpha8:
#2495145 by twistor, cashwilliams, greggles, klausi: Possible XSS in PuSHSubscriber.inc
#2502419 by klausi: Log messages XSS attack vector
Thanks in advance.