Hi,

The setup:

I have a vanilla Drupal with 1 page and a CCK content type 'publication' with different fields (author, publicationyear,...) but no body or teaser. I added a dozen 'publication' nodes and configured cron to get hit every hour to index all the nodes and make them searchable. A fairly standard setup I think.

Modules I use:
CCK 2.2 and Views 2.2. Nothing else. Oh, and of course Drupal 6.9.

The problem:

CCK fields aren't indexed which makes them not searchable. The only fields that get indexed are the default 'Drupal core' fields like title and body. This means only words in the title of a node are available to search but we can't search on the name of the author or any other word that's saved in a CCK field.

Strange thing is that CCK provides a seperate interface under 'admin' to configure how CCK fields are indexed by search. So, there's actually support for field indexing in CCK itself. Yet tinkering with it doesn't seem to do much. (I kept checking the indexing table for words from the CCK fields to pop up). I also dove in the CCK code but found nothing out of the ordinary.

I also checked cron (works), permissions (all set correctly) and watchdog (no errors) but nothing that explains this odd behaviour.

After an afternoon of trying to get it to index those fields, I got it to work like this:

1. create new nodes
2. clear the cache under 'Site configuration' > 'performance' > 'clear cached data'
3. reindex the site: 'Search settings' > 'Reindex site'
4. hit cron.php manually through a browser (allthough crontab seems to work fine too)

All of a sudden, all the CCK fields on the nodes are indexed nicely! Even the cached ones. Now I can search on the name of the author and such.

Of course, I think this quite basic stuff and I was wondered why caching is preventing it from being indexed correctly. I can't find a solution or point at a module/piece of code that causes a conflict between Drupal caching, CCK and Search indexing. I've searched drupal.org and other sites extensively but I can only find issues that are related to this one and in older versions.

I've checked with another Drupal developer: he encountered the same behaviour but doesn't have a solution either.

Comments

dipen chaudhary’s picture

dipen chaudhary commented

few clarifications?

Did you enable caching before indexing the site for the first time?
Did you index the site before adding cck fields? If yes, did you re-index the site after adding cck fields data.

Once you clarify this, I'll try to reproduce this one.

netsensei’s picture

netsensei commented

1. Under 'performance' the 'caching mode' is disabled. It hasn't been turned on AFAIK (I'm not the only one working on this site)
2. The site wasn't indexed before the CCK fields were added. Cron was set up the content types were defined.

These tables contain data:

- cache
- cache_content (all the existing nodes of the site)
- cache_filter

The other cache_* are empty.

I also did the following:

1. add a few nodes with CCK content
2. click on 'reindex site' under 'Search settings'
3. hit cron.php (either manually or through crontab)

A search on words contained within CCK fields yielded no results.

netsensei’s picture

netsensei commented

I did some testing and searching this morning and came up with a cause and a solution:

To reproduce this:

1. make a CCK content type with fields
2. create several nodes
3. enable cron and let it index the site
4. start searching for data that's stored in the fields: you shouldn't get any results.

5. now go to 'user management' > 'permissions'. Under 'content_permissions module' make sure that the 'view' permissions for the CCK fields is explicitly set for the anonymous user.
6. go to 'search settings' and click the 'reindex' button of the site
7. wait until cron has run
8. initiate a new search: now the data in the CCK fields have been indexed and are searchable!

What's the cause?

The indexer get's data from the $node through two op's in hook_nodeapi: 'view' and 'update index'. In CCK only the 'view' op is implemented (so it seems). The 'view' op is also used to render data to the site. Therefor it is subject to the way permissions are set. If anon users are denied to view the CCK fields, then they won't get indexed by search.

Side effects:

Suppose you want to make sure all the data is indexed on the site through cron, but you don't want all the CCK fields visible for anon users, then you can't rely on the 'view' permissions in user management. You should set them all for the anon user in order to get all the data indexed, and show only those fields you want through customizing the correct template tpl for the theme.

If you change the 'view' permissions for the CCK fields after you have created nodes, the existing nodes won't be reïndexed to make the new CCK fields visible.you have to reïndex the entire site through 'search settings' and clicking the 'reindex' button. This means the entire site is reïndexed. If your site contains up to a few thousands nodes, this takes a while to do.

There's a very good reason why indexing in CCK is implemented the way it is: you don't want to make data/nodes pop up in the search results if you don't set the 'view' permissions for those fields. But then again: (manually) reindexing entire sites if you change the 'view' permissions seems to be a bit of a crude way to respond to changes.

dipen chaudhary’s picture

dipen chaudhary commented

That makes sense, I've never felt the need to turn on content permissions module. But if it does not include cck fields in result set of search for those who dont have permission, then its awesome. But does it include the cck fields for those who have the permission? I'll try to test this out if u have not tested it already :) (Like I said I never needed content permission :))

dipen chaudhary’s picture

dipen chaudhary commented
Status: Active » Closed (works as designed)
hnln’s picture

hnln commented

subscribe

vrteach’s picture

vrteach commented

I'm also seeing this. I'm using the "Reindex" module, as well as just the regular cron. Sometimes the reindex module shows that it is indexing the full number of nodes (about 1,900) but sometime it only shows about 125, and sometime 1. I've turned on view_* for all the cck fields for anonymous users.

vrteach’s picture

vrteach commented

Doh! My problem was simple. The big bunch of nodes that were not being returned by search were all migrated in using the migrate module. When they were created, they were not published, so the search did not return them even though I could see them as administrator.

davidredshaw’s picture

davidredshaw commented

I've had this issue. In my case I didn't want anonymous users to access any content but I did need search of CCK fields. I disabled access for anonymous users using the "access content" permission but enabled viewing of specific fields in order to search them.

paganwinter’s picture

paganwinter commented

Subscribing...

pvanerk’s picture

pvanerk commented

In my case I had to perform an extra step:

5. now go to 'user management' > 'permissions'. Under 'content_permissions module' make sure that the 'view' permissions for the CCK fields is explicitly set for the anonymous user.
6. go to 'search settings' and click the 'reindex' button of the site

Go to ?q=admin/settings/performance and hit the "Clear cached data" button.

7. wait until cron has run
8. initiate a new search: now the data in the CCK fields have been indexed and are searchable!

Thanks for the post! It helped me big time.

plastikkposen’s picture

plastikkposen commented

Hi

Its my first time I've experience of this problem as well. Did not diable any cache functions at all (normal cache). The first poster, mr. netsensei's recipi works as suggested;

(1. create new nodes)
2. clear the cache under 'Site configuration' > 'performance' > 'clear cached data'
3. reindex the site: 'Search settings' > 'Reindex site'
4. hit cron.php manually through a browser (allthough crontab seems to work fine too)

cdonner’s picture

cdonner commented

I suspected permissions as the issue (they always are :-), but forgot to check the field-level settings. Thanks for the detailed answer.

charlie-s’s picture

charlie-s commented

What about fields that cannot be viewable to anonymous?