Automatic CCK introspection

To clarify your needs - you're writing custom search modules using the apachesolr.module, and need this functionality to be better abstracted to support your development? Looks very interesting, but please explain more.

I'm looking forward to the query construction object. It's high on my list of things that need massive refactoring. I'm glad you're working on it.

Let's move the query discussion to a new thread.

Hi pounard!

Thanks for this cool module! I wanted to store some cck fields in the solr index to render a better search result containig an imagefield and cck fields.

So I used your module.

Unfortunately it is not possible to store multiple values for one cck field in the index. The current API does not support this.
http://drupal.org/node/664866

This means, it is not possible to save the filepath, filemime, data description of an imagefield in the solr index.

Because of this issue I changed your module and added this behaviour for the special case "filefield".

Attached you will find my version of your module. Maybe we should start a new project for this module?

Thanks to both of you!
I am testing now.

The original code provided by pounard works fine, I do have many more of my cck fields in solr now.

I think the new addition from ayalon needs a tweak or I suppose it could be my imagefield, I dont have much time to look into it right now but it does seem interesting, if I figure something with it ill post it back.

When I tested ayalon's version and got some multivalue field errors in the dblog.

Indexing failed on one of the following nodes: 69672, 69673, 69674, 69675, 69676, 69677, 69678, 69679, 69680, 69681
"400" Status: Bad RequestApache Tomcat/6.0.18 - Error report HTTP Status 400 - ERROR: [84e3c3424f2e/node/69672] multiple values encountered for non multiValued field ss_filefield_path: [sites/dev.mysite.net/files/prop_img/448946/448946-1.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-2.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-3.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-4.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-5.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-6.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-7.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-8.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-9.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-10.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-11.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-12.jpeg]

type Status reportmessage ERROR: [84e3c3424f2e/node/69672] multiple values encountered for non multiValued field ss_filefield_path: [sites/dev.mysite.net/files/prop_img/448946/448946-1.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-2.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-3.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-4.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-5.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-6.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-7.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-8.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-9.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-10.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-11.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-12.jpeg]

description The request sent by the client was syntactically incorrect (ERROR: [84e3c3424f2e/node/69672] multiple values encountered for non multiValued field ss_filefield_path: [sites/dev.mysite.net/files/prop_img/448946/448946-1.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-2.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-3.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-4.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-5.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-6.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-7.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-8.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-9.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-10.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-11.jpeg, sites/dev.mysite.net/files/prop_img/448946/448946-12.jpeg]).Apache Tomcat/6.0.18

You have my attention. Don't create a new project quite yet. That may be the best way to go, but let me have a good look first. Thanks.

I'll have a look at this. It seems that you have multiple images attached to one node.

The starting module has a typo in the filemime callback. It will never work. filemime will never be indexed.

/**
 * @see apachesolr_cck_index_number_callback()
 */
function apachesolr_cck_index_filemime_callback($node, $field_name) {

    $fields[] = array('value' => $field['filemime']);

}

should be

/**
 * @see apachesolr_cck_index_number_callback()
 */
function apachesolr_cck_index_filemime_callback($node, $field_name) {

    $fields[] = array('value' => $value['filemime']);

}

But we need a solution for multivalue imagefields. I try to fix this.

Is the use of per-field really the best way to go? I have had success writing Solr/CCK mappings (before I stumbled across this issue) as:

$mappings['text'] = array(
  'text_textarea' => array(
    'index_type' => 'string',
  ),'text_textfield' => array(
    'index_type' => 'string',
  ),
);

and that worked fine. This has the advantage of updating automagically when new fields are added, without having to re-run apachesol_rcck_apachesolr_cck_fields_alter. I'm working on modifying it to behave as I have described, and will post a patch when I've got it working.

Also, using float for all of the number types seems wrong. The number_integer type should certainly be an integer type and number_decimal might even need to be a string (to preserve the exact value of the numeric type). I'll make these changes as well. A tdouble would probably be fine for number_decimal fields.

As far as multiple values per field, the solution being developed in #558160: date facet for cck field for the date module is to create a separate Solr field with '_end' appended to the end of the name (for the end date). That is, if you had a multi-valued date field with start and end times called times, you would have data that looked something like this:

<date name="tdm_cck_field_times">2007-01-05T13:05:00Z</date>
<date name="tdm_cck_field_times_end">2007-01-05T13:07:00Z</date>

<date name="tdm_cck_field_times">2007-01-08T02:05:00Z</date>
<date name="tdm_cck_field_times_end">2007-01-08T12:07:00Z</date>

In other words, you use parallel arrays. It's not a terribly pretty way of doing it, but Solr doesn't yet support fields with compound values (though it looks like it's in the works), so this is probably the best we have for now.

It looks like the patch in #558160: date facet for cck field has a fairly general way of specifying compound fields, so it would be better to use that than to create a bunch of special cases for filefield, since it is hardly the only field to use compound values.

It looks like the patch in #558160: date facet for cck field has a fairly general way of specifying compound fields, so it would be better to use that than to create a bunch of special cases for filefield, since it is hardly the only field to use compound values.

this is true.

I've been working on merging the bits from #558160: date facet for cck field in here as well. I have my work at http://github.com/haxney/apachesolr.

I've added a 'default_index' option to CCK field definitions which sets up a default 'indexing_callback' for the field type. This makes specifying types much easier when no additional processing needs to be done.

The definitions for the 'nodereference' type (ignoring the 'display_callback') would be:

$mappings['nodereference'] = array(
  'nodereference_buttons' => array(
    'default_index' => 'nid',
    'index_type' => 'integer',
  ),
  'nodereference_select' => array(
    'default_index' => 'nid',
    'index_type' => 'integer',
  ),
  'nodereference_autocomplete' => array(
    'default_index' => 'nid',
    'index_type' => 'integer',
  ),
)

This means that the value $node->$field_name[$x]['nid'] would be indexed (for each $x, if the field accepts multiple values). This is much simpler than having each field type define its own indexing callback.

Eventually (and it may be more work than it's worth), I plan to allow for 'default_index' to accept an array that looks something like

array(
  'name' => 'title',
  'website' => 'url',
);

For the example of the link module. This would mean "create two Solr fields, ${field_name}_name and ${field_name}_website, which will receive title and url, respectively." There are some cases, like the link CCK field, which are compound, but do not require any special processing before being handed off to Solr. There are probably only a few instances in which this will actually be useful, but it shouldn't be too hard.

Once this is stabilized (and I plan on writing some unit tests for this), it should make adding support for most of the CCK fields out there much more straightforward.

Woo! I now have CCK fields being indexed by Solr with a minimum of fuss!

Here are the fields which I now have working (in theory; I've only actually tested ca_phone, text, and email):

• nodereference
• number_integer
• number_decimal
• number_float
• text
• userreference
• email
• filefield
• link
• fr_phone
• it_phone
• ch_phone
• ca_phone
• cr_phone
• pa_phone
• uk_phone
• ru_phone
• es_phone
• au_phone
• cs_phone
• hu_phone
• nl_phone
• il_phone
• nz_phone
• br_phone
• hk_phone
• mo_phone
• ph_phone
• jo_phone
• eg_phone
• int_phone
• viewfield
• viewreference

Once again, the code is hosted on GitHub. What would make integrating it back into the CVS repo easiest? It's more changes than would make sense as a single patch, so how should it be handled? I'm handy with Git, so I can hack up the set of commits any way that is convenient.

There is still some polishing that needs to be done, as well as documentation (since it changes how hook_apachesolr_cck_fields_alter() works), and I'll take care of those in the next few days.

Subscribe

haxney...
I tested your version of apachesolr and while it seems to work there are no new fields being indexed.

The original post by pounard gets all those elusive cck fields to show on the filters page, in fact I didnt notice any difference when testing your version of apachesolr, does it also require pounards cck introspection module? I didnt test that combined with yours yet.

Cheers!

@macrocosm

I tested your version of apachesolr and while it seems to work there are no new fields being indexed.

What do mean? That the fields aren't being sent to the Solr server? What part is working and what isn't working?

It has definitely worked for me in testing (though I only am testing with 4 documents, so I don't know how it does with larger sizes), and you may need to reindex. I haven't really looked at what triggers a reindexing, so I usually rely on going to admin/settings/apachesolr/index and checking "reindex immediately" and hitting "Re-index all content." You could also try running cron.

Also, the best way to tell if the content is really being indexed is to run a simple query through the Solr admin interface and see if it includes the CCK fields (which will all be named something like "ss_cck_field_text").

The original post by pounard gets all those elusive cck fields to show on the filters page, in fact I didnt notice any difference when testing your version of apachesolr, does it also require pounards cck introspection module? I didnt test that combined with yours yet.

I also have all the CCK fields showing up in my "enabled filters" view on admin/settings/apachesolr/enabled-filters, so I don't know what's wrong.

Make sure that you are replacing the entire apachesolr project with my new version, as it modifies apachesolr.module and apachesolr.index.inc, among other core parts of the project. You can go here for a tarball of the current version of my changes.

Let me know if that helps.

@haxney

I did do a manual re-index which didnt seem to make a difference. I also did a full replacement with your module and ran update.php too.

I used the "enabled filters" page for a short cut to see if solr was picking up the other fields. It seems to me that while a lot of the fields are there well more than half were not. My main content type thats being indexed is a real estate mls listing and has many fields & types of fields. Seems apachesolr doesnt bring "CCK filefield field" or "CCK number_integer field" at all, and even on the "CCK text field" only about half the fields showed up. But like I said Pounards module at the top of the page brings in every one of them, with no errors that I have found.

Did you blend pounard's module into your arrangement? ..or is it a new direction?

@haxney
I will test your new version this weekend, thanks for posting it.
Cheers!

EDIT: I should also add that I am using the latest cck-6.x-2.x-dev since thats the only one that works with views 3.x

It's not a good idea to use the haxney fork on a production site right?

Well if you can get it to work and it tests ok I dont see why not... but haxney's version is really for testing deving im guessing.

That said the post by pounard in post #1 above does automatic CCK introspection well AFAICT.

No it's not a good idea. We should definitly create an new module.

@haxney
Please use the module from #8 and provide some patches so we can test this f.ex. the link field indexing.

@robertDouglass
Could we create a new module for this? We have now a lot of code and because we don't have a cvs repository, the whole module code breakes in to different forks.

I tested haxney's version on a sandbox installation and it worked for the text and nodereference fields I have, although the block is now rendering the nodereference options as <a href="/node/1">Premo Haero Quia</a> instead of just Premo Haero Quia as it was before.

I've committed #558160: date facet for cck field so you'll probably have to merge/re-roll. Please double check everything and then change status when I'm able to review. Thanks.

Subscribe

ApacheSolr is a GREAT module, but it lacks integration of CCK Fields. In my case I have one Integer Field and one Float Field in my CCK and it gets not automatically indexed by ApacheSolr (it works on 'some' text fields though). What I'm trying to achieve is simply sort by one of my CCK fields.

Of course I downloaded the apachesolr_cck extension but both (my integer field and my float field) do not get indexed. Instead of the numeric values it always has some random text in those fields. The text seems to be from the first text field in my CCK.

Subscribe

Subscribe

Doubtful this will be integrated into 6.x-2.x

So apache solr 6.x doesn't integrate with cck at all?

Nevermind, having re-read the issue, this now seems like something that can go into 6.x.

Note that 6.x already has CCK integration for most fields on which you would want to do faceting. This patch just adds more integration.

I don't see a patch. Allowing faceting on many CCK is not a good idea. Location CCK fields, for example, often have unique values, so faceting is not useful. For integer/float fields, we would want to add facet blocks that support ranged queries.

Anyway, 6.x-2.x will be unsupported soon, and 6.x-1.x is feature-frozen. 7.x has a different API. Please re-open a 7.x issue if you find Field API support in 7.x lacking. I think this should probably be its own module for 6.x.