Search index never gets to 100% if there are no helpers for attachments OR if some nodes have no visible fields

Thanks for making a new case out of this. That should have been my job :)

I'll do some tests and post more specs when I have time to debug this. For now, it seems like the indexing process goess into some kind of loop, doing all the same files every cron run (up till the maximum of items per cron run).

Eventhough new content is not added, each cron run now last minutes, after I upped the items per cron run to 500.

I researched this further and did a clean Drupal install. The problem is easily reproduceable.

1. Install clean D6.

2. Create a filefield called "files".

3. Download and enable search_by_page and search_files

4. Autodetect file helpers (I had all helpers that search_files supports installed and enabled)

5. Create a node with the files-field, and upload several filesincluding:
* some.txt
* some.doc
* some.jpg
* some.zip

6. Run cron.

7. Visit admin/settings/search and admin/reports/dblog. Notice:
* Search index doesnt get to 100% no matter how many times you run cron
* Watchdog-log is full of "Content was skipped - PID (xxx)".

8. The search works for some.txt and some.doc, as it should. However when there are too many non-indexable items and the itemlimit per cron run is smaller, even the normal files never get to be indexed.

Repeating 6. and 7. will result in the the same thing happing again and again. The same files generate each message, just incrementing the PID. I'm currently at PID 2082295. Yeah, over 2 million, thanks to my often run cron - despite the site having less than 1000 attachments!

It seems the underlying problem causing the loop is because of this logic.

search_by_page.module, L516:

    // Get page title from module and optional content
    $info = module_invoke($item->from_module, 'sbp_details', $item->modid, $item->environment);

    if (!$info) {
      // Module indicated not to index this page after all
      watchdog('search_by_page', 'Content was skipped - PID (%pid), path (%path)',
        array(
          '%pid' => $item->pid,
          '%path' => $item->page_path),
        WATCHDOG_NOTICE);

      _search_by_page_remove_path($item->pid);
      continue;
    }

.. which means "empty" index items will be cleared with:

/**
 * Internal function: removes a path item from the paths and search database.
 *
 * @param $pid
 *   ID of the path to remove.
 */
function _search_by_page_remove_path($pid) {
  db_query('DELETE FROM {sbp_path} WHERE pid=%d', $pid);
  _search_by_page_remove_searchinfo($pid);
}

/**
 * Internal function: removes a path item from the search database.
 *
 * @param $pid
 *   ID of the path to remove.
 */
function _search_by_page_remove_searchinfo($pid) {
  db_query("DELETE FROM {search_dataset} WHERE type='search_by_page' AND sid=%d", $pid);
  db_query("DELETE FROM {search_index} WHERE type='search_by_page' AND sid=%d", $pid);
  db_query("DELETE FROM {search_node_links} WHERE type='search_by_page' AND sid=%d", $pid);

}

So, if the submodule returns nothing, the index information will be removed. Next time we try to run cron, the process notices the missing indexes, and tries to recreate it - and fails again. The loop never ends. And if there are more non-indexable filetypes than the "index search items per cron run", the search index (probably) just never gets all the files indexed.

A simple fix which would work in this case is to just return something to $info, and we can avoid the problem. The something should of course be something relevant, such as the filename of a file and, that has unindexable content. Or the title of the node to which it belongs. Those strings should be search-index worthy (but currently afaik dont make it to the index)?

Another way I guess would be to just not delete the path with
_search_by_page_remove_path($item->pid);

.. but I don't know these modules well enough to be sure this wouldn't break anything. So I'll try to fix this the simple way first.

Hmm, after looking at the sbp_path table, I noticed the non-supported filetypes always have "0" in the "last_index_time".

One row for example:

I looked further, and it seems "last_index_time"=0 is true for all non-indexable files, but never indexable files! There was one exception: a file named test.txt. That makes sense though, because this particular text file was empty, so there was nothing to index. Even so, it had been indexed, but I assume the last_index_time was set to 0 anyway.

This means that the looping problem is not only about non-handled filetypes, but empty files aswell. And I guess any other failed index-operation may lead to the same problem. I think it's enough reason to mark this as critical.

Here is a patch for the "simple" solution. It just changes 2 lines in sbp_attach.module like this:

  return NULL;

to

  // We can still return at least the filename.
  $content = basename($fname);

I tested it, and it seems to work great. Cron runs finally index the content and no more "Content was skipped"-messages. Hooray.

.. However, I noticed a new problem: after all the normal indexing (for non-existent files) was done, there were still 9 unindexed items, and now the complaint was:
Content not rendered () - PID (250), path (node/66), realpath (node/66), language (fi)

Each cron run now results in 12 of these messages, which are from 6 different nodes (apparently each node creates two of these messages). The nodes are of a few different nodetypes, but they have no files attached.

It seems this new problem is not really related to the issue with the files, however it fits into the topic of this issue (incomplete search index).

Anyway, the problem with no helpers for some filetypes should be fixed with this patch. I somehow feel that the looping condition still might occur under other circumstances, so maybe it needs to be reviewed.

Setting to "needs review". I'm wondering why no one else seems to have encountered this problem?

More debugging, and seems like the "new" render-issue is caused by nodes that have no visible fields to render.

It just happens that all the 6 nodes have:
- bodytext disabled
- no visible cck fields

With visible I mean that some fields are set to hidden on the "Display fields"-page for the content type. I'm outputting those fields with other means (such as views blocks).

This piece of code apparently don't care about my views blocks, and sees the node content as empty:
$content = menu_execute_active_handler($path);
.. which would be totally ok, if it wouldn't lead to this endless loop.

A very easy and similar solution to the patch above, would be to simply add "some info" that is always available as $content, if there is no other content. Something like the $node->title, or such. There must be more elegant ways to fix this, but.. let's see.

I probably should have created a new case, but here goes: a patch that fixes the render errors.

As I said it simply puts node title as $content, if $content is empty and there is nothing else to render. Fixed my problem, and I FINALLY have search index at 100% without errors in my logs :D!

Again, this underlying logic that causes the loop in some cases still needs to be fixed, but these simplistics fixes worked for me.

Im not sure if my approach works as well as it should. It does help: my search index is finally full and I get no unnecessary notices in my logs, however the search_indexing cron is still qute slow. It averages on around 60 seconds, which is way too much for a site this size.

There is no "endless loop" anymore, however I believe search_by_page for some reason still goes through all the files/nodes during each cron run, even if they haven't changed. If I disable search_by_page (and the submodules), the search indexing lasts 1 to 4 seconds. So, 1 minutes long indexing when there is nothing new to index, doesnt sound like it's working as intended.

I will your patch tomorrow. (unfortunately I don't have a chance to do it sooner.

I tested your patch (well actually I just updated to the latest dev-version). Unfortunately it doesnt fix the problem.

My searchindex was at 100% before updating the module. After the update and running the hook once, it was "99% and 2 items missing". After a few test runs it's "99%, 18 items missing". I dont get any messages in my watchdog log though. With my previous patches it was at 100% (but running as slow).

The whole indexing operation still takes forever (40-150 seconds) even with no content changed - and the whole site slows down during the indexing. I started using elysia_cron, so I can execute invidual hooks at given time intervals (which allows me to index during the night + manually for testing).

I now understand sbp actually tries to reindex "everything" during each run. I have to wonder why and what will happen when using the module on very large sites (or often on smaller sites, like my case)?

Could there be an UI option to only index updated items, and not unchanged content? Or a timebased setting how often this is run? I should probably make a new case though, right :)? Or should I post it here: #738382: Better handling of decision on when to index?

Thank you jhodgdon, and sorry for being so slow to respond! I've been overworked for a while now.
I tried the latest version, ran cron a few times times, and search index fibally reaches 100%.

Still the search indexing process seems not very optimal for my use case, since I have no need the re-indes and pages every 5 minutes (thats how often I run cron). I will continue about that in the other case. Anyway thanks again for this. Also, I'm glad to hear you've become the new Drupal documentation project lead. Please keep doing hat you do :)