This module is working really well for me, but I've had a number of problems which all boil down to a single issue. As stated in the README, the module requires a unique identifier to be mapped for the search result entities. I'm not sure the documentation makes it clear, but this also has to be an integer (or the entity system complains about a type error). My problem is that the data I am storing in SOLR does not have a guaranteed unique integer field. There are nearly-unique integer fields, and a unique String field, but that is all.
So, I am currently using one of the nearly-unique fields, but it is a bit of a hack. Ideally, I would like the Sarnia module to generate an unique entity ID for me. Is this possible?
Also, I guess I can assume the unique ID only needs to be unique in the scope of the search results? i.e. it does not matter if a node in Drupal that is not indexed in SOLR has the same ID number as a SOLR result? If so, would the SOLR results document index be a sufficient identifier?
Thanks for your time. Andy.
Comment | File | Size | Author |
---|---|---|---|
#13 | contextual_filters_as_strings.patch | 547 bytes | jmdeleon |
#10 | sarnia-readme-unique-integer-id-1687980-10.patch | 699 bytes | andrewmacpherson |
Comments
Comment #1
becw CreditAttribution: becw commentedIIRC, something in the entity system (either Drupal core or Entity API) assumes that entity ids will be integers. I'm not sure that it would make sense to generate integer entity ids using Sarnia (... and store them in the Drupal database?) Do you know if there's a way to generate a serial field in Solr? I'd consider that a hack too, though... ideally you'd be able to use your string field.
This issue needs some research into non-integer entity ids; if they're allowed now (Entity API is evolving), then we need to make sure Sarnia supports them; if they're still not supported, maybe there are some
Schema.xml
techniques that we could recommend.Also, how can we make the integer id requirement more clear in the Sarnia documentation? Maybe we need a section on
Schema.xml
configuration as well as configuring Search API for Sarnia?Comment #2
anj CreditAttribution: anj commentedIn the documentation, I guess "You must choose a field with unique values" could read "You must choose a field with unique integer values", which would at least mean that small but important piece of information is there and not only in the README.
There is an SOLR ord() function that very nearly does the right thing, but seems to be very memory hungry.
I guess I'll have to re-index and create a suitable integer for each item. I'd be happy to live with a hack for now, if anyone knows where in Sarnia the returned ID is mapped to the Entity.
Comment #3
becw CreditAttribution: becw commentedOk, I've updated the Sarnia documentation.
It looks to me like Solr's
ord()
function is a property-level function, not result-level: it returns the index of a value in a multivalue Solr property. Also, generating IDs based on a particular query would conflict with caching--the same Solr record could be returned with different ids depending on the query.Comment #4
krlucas CreditAttribution: krlucas commentedFirst, this module is the best thing ever.
Second, the documentation for the entity id property says:
Notice it says "numeric", not integer. My Solr index is composed of very similarly structured records from different sources with potentially overlapping integer ids. However, by adding an integer "source" id to my index I was able to to create a composite float id of "$source_id.$item_id".
So far so good...
Comment #5
Antti J. Salminen CreditAttribution: Antti J. Salminen commentedActually, as that quote says there is no requirement for entity ids to be numeric if they are not fieldable. See here: http://drupal.org/node/1027908#comment-4636438
Comment #6
remydenton CreditAttribution: remydenton commentedRight, only fieldable entities require numeric IDs. Unfortunately, that doesn't help us much since Sarnia entities are fieldable.
Also, whatever the documentation may say, entity_property_verify_data_type() validates fieldable entity IDs specifically as integers and will throw an error if you've got decimal points in there.
My solr index actually uses namespaces in the field that's supposed to contain the unique ID (so the unique value is something like "islandora:001"). Obviously, that won't pass muster as integer or number, and I can't even think of a good way to generate a unique integer from a text string like that in Solr. Does anyone have a clever solution for this?
The alternative is to just bluster on through and use strings as the entity ID anyway. Hardly ideal, but the only issue I've run into so far with this approach is that I get a validation error when I try to add the entity ID as a field in the view. As long as I remove that field, things seem to work okay. Maybe I just haven't done enough testing yet...
Comment #7
anj CreditAttribution: anj commentedFor what it's worth, this problem has more or less disappeared for me, because we're moving to Solr4. In order to support SolrCloud, each document has an internally generated _version_ long integer timestamp, and this can be used for the entity ID.
Comment #8
ibakayoko CreditAttribution: ibakayoko commentedyou need to install a solr 4, create a new collection and import the index from your solr 3 core.
and use sarnia to query the new collection.
Comment #9
tombraider CreditAttribution: tombraider commentedHi,
I'm trying to display data into my drupal website using sarnia to retrieve external/non-drupal indexed data from Solr. I followed the steps in this link https://drupal.org/node/1379476 but I'm somehow stuck with the "ID field" select box. My select box is not showing any option, therefore I cannot proceed with the next step. This is because I don't have a unique integer id. I only got mongoid which is a type string that I use as unique id from my non-drupal application. I'm currently using Solr 4.3 and I have the _version_ field of type long in my schema.xml file.
I'm wondering how we can use that as the entity id since no option is showing up in the select box. Any help regarding this would be very much appreciated.
Comment #10
andrewmacpherson CreditAttribution: andrewmacpherson commentedbecw's change to the handbook page (comment #3) is alos needed in the module's README.txt file.
Patch attached.
Comment #11
andrewmacpherson CreditAttribution: andrewmacpherson commentedComment #12
jmdeleon CreditAttribution: jmdeleon commentedPatch of corrected documentation accepted.
Comment #13
jmdeleon CreditAttribution: jmdeleon commentedAdding an additional patch to this: Since the unique ID field is no longer required to be an integer value, I made a small change to the contextual filter handler to treat all arguments as strings, where they can further be processed (and converted to integer if necessary).
Comment #14
jmdeleon CreditAttribution: jmdeleon commentedComment #15
jmdeleon CreditAttribution: jmdeleon commented