Indexing does not complete

You're right, this definitely is a case that should be handled by the module in some way. Force-splitting after 50 characters (or 255 and increasing the field size accordingly) is probably an easy solution, and there doesn't seem much else we can do. Maybe we should then also display a warning that such truncating has occurred during indexing.

However, I'm really curious as to what kind of words you have in your nodes? First I thought about some special language/alphabet that my processor wasn't built for, but … ja, anscheinend eher nicht. ;) Are you using the Tokenizer processor, maybe with custom settings?

Subscribing, I ran into this problem as well. Workaround: enable the tokenizer processor on the workflow tab of your index.

I think the DB search does not split words correctly.

Ah, now that I look at the code, the further tokenizing by the DB backend is only done when the data isn't already tokenized in some way. I assume you activated the HTML filter (which also tokenizes, but only according to HTML tags – therefore containing whole paragraphs as single tokens, if they don't contain HTML tags), but not the Tokenizer, which results in this error.
However, simply letting the DB backend tokenizing also run on already tokenized data isn't good either – this would make the Tokenizer processor rather useless, as you then couldn't configure the tokenizing anymore.
After all, I think people just shouldn't use the DB backend without the Tokenizer preprocessor, it's just not built for that. Maybe we can display a warning when too long tokens are encountered, which then checks whether the Tokenizer is enabled. How about that?

Had both of you the HTML processor enabled and the Tokenizer disabled?

Same problem with both processors enabled. Investigating.

In my case, we have some dirty search data in the external source material. e.g.

"scr9706.htm1596CryptoOutbreakLBENSON8570025100000251pkgATC" file Carchivescriptscr9706.htm uniqueID 1596CryptoOutbreakLBENSON8570025100000251pkgATC 19970714 8 1596CRYPTO OUTBREAK...germs spread by zoo fountain va 242 LB ATC A NAME"1596CryptoOutbreakLBENSON8570025100000251pkgATC" ItemSlug Writer Vers Word Read Audio Total Type Broadcasts 1596Crypto Outbreak LBENSON 8 570 0251 0000 0251 pkg ATC Headline Cryptosporidiosis Outbreak Sickens Kids at Minnesota Zoo Keywords Cryptosporidium, Minnesota Zoo, 

So here, the truncation is acceptable and just needs error suppression, I suppose. Thoughts?

Yes, in such cases, just catching the error and maybe displaying / logging some explanation is probably the best we can do. They shouldn't throw an exception, in any case.

So, to summarize my proposed solution:
- Check all text tokens (no matter whether tokenized by the processor or the service class) for length.
- When encountering a token that is too long, check whether the standard-tokenizing would solve this:
- - Yes: check whether the "Tokenizer" processor is enabled for the index.
- - - Yes: execute standard tokenizing on the token and set a message for the user to check her Tokenizer settings.
- - - No: execute standard tokenizing on the token and set a message for the user to enable the Tokenizer processor.
- - No: truncate the token to 50 characters and display a message explaining to the user that over-long words were encountered.

What do (all of) you think, would this be a sensible approach?

Or maybe we should also make the maximum word length configurable for database servers?

That all sounds right to me.

I think 50 is perfectly reasonable, at least in English. Quick look suggests the longest 'official' word is 45 characters.

https://secure.wikimedia.org/wikipedia/en/wiki/Longest_word_in_English

For other languages, there's this list: https://secure.wikimedia.org/wikipedia/en/wiki/Longest_words

There are a handful over 50, and it looks like 80 would handle all but the most edge cases (which are generally proper nouns).

Where would this logic be introduced? I might take a crack at it.

Thanks for the offer, but I've already written up a patch. Please see (and test) the attached patch.

I've also added at least rudimentary error handling for strings, which have similar problems with values longer than 255 characters.
If this problem is encountered on a production site (or, maybe rather "when"), we can probably fix this by dynamically increasing the field size in the database.

Hm. Upgraded to 7.x.1.x-dev, applied the patch and tried the drush command (which works without the patch) and:

WD php: Warning: preg_split() expects parameter 2 to be string, array given in SearchApiDbService->convert() [warning]
(line 439 of
/shared/vash/sandboxes/rickard/archiveportal/www/sites/default/modules/search_api/contrib/search_api_db/service.inc).
WD php: Warning: htmlspecialchars() expects parameter 1 to be string, array given in check_plain() (line 1476[warning]
of /shared/vash/sandboxes/rickard/archiveportal/www/includes/bootstrap.inc).

Fatal error: Cannot create references to/from string offsets nor overloaded objects in /shared/vash/sandboxes/rickard/archiveportal/www/sites/default/modules/search_api/contrib/search_api_db/service.inc on line 337

Call Stack:
    0.0007     203160   1. {main}() /shared/usr/local/drush/drush.php:0
    0.0171    3118416   2. drush_main() /shared/usr/local/drush/drush.php:41
    0.4517   47897464   3. drush_dispatch(array(23), ???) /shared/usr/local/drush/drush.php:101
    0.4517   47897464   4. call_user_func_array(string(13), array(2)) /shared/usr/local/drush/includes/command.inc:204
    0.4517   47897464   5. drush_command(string(1), string(2)) /shared/usr/local/drush/includes/command.inc:0
    0.4526   47897464   6. call_user_func_array(string(12), array(3)) /shared/usr/local/drush/includes/command.inc:721
    0.4526   47897464   7. drush_invoke(string(15), string(1), string(2)) /shared/usr/local/drush/includes/command.inc:0
    0.4529   47898056   8. drush_invoke_args(string(15), array(2)) /shared/usr/local/drush/includes/command.inc:58
    0.4535   47907632   9. call_user_func_array(string(21), array(2)) /shared/usr/local/drush/includes/command.inc:120
    0.4535   47908112  10. drush_searchapi_index(string(1), string(2)) /shared/usr/local/drush/includes/command.inc:0
    0.4716   50822296  11. search_api_index_items(class SearchApiIndex, string(2)) /shared/vash/sandboxes/rickard/archiveportal/www/sites/default/modules/search_api/search_api.drush.inc:143
    0.6139   51784448  12. SearchApiIndex->index(array(10)) /shared/vash/sandboxes/rickard/archiveportal/www/sites/default/modules/search_api/search_api.module:715
    1.0196   62629368  13. SearchApiServer->indexItems(class SearchApiIndex, array(10)) /shared/vash/sandboxes/rickard/archiveportal/www/sites/default/modules/search_api/includes/index_entity.inc:408
    1.0243   63300832  14. SearchApiDbService->indexItems(class SearchApiIndex, array(10)) /shared/vash/sandboxes/rickard/archiveportal/www/sites/default/modules/search_api/includes/server_entity.inc:196
    1.0244   63301120  15. SearchApiDbService->indexItem(class SearchApiIndex, long, array(23)) /shared/vash/sandboxes/rickard/archiveportal/www/sites/default/modules/search_api/contrib/search_api_db/service.inc:301


Variables in local scope (#15):
  $boost = '1.0'
  $e = *uninitialized*
  $field = array ('name' => 'Search text', 'type' => 'tokens', 'boost' => '1.0', 'value' => array (...), 'original_type' => 'text')
  $fields = array ('type' => array (...), 'title' => array (...), 'status' => array (...), 'promote' => array (...), 'sticky' => array (...), 'created' => array (...), 'changed' => array (...), 'search_api_language' => array (...), 'author:name' => array (...), 'author:roles' => array (...), 'body:value' => array (...), 'field_story_series' => array (...), 'field_story_program' => array (...), 'field_story_iptc_data' => array (...), 'field_search_text' => array (...), 'field_story_reporters:title' => array (...), ...)
  $focus = 0.496787204451
  $id = 40859
  $index = class SearchApiIndex { protected $server_object = class SearchApiServer { ... }; protected $callbacks = array (...); protected $processors = array (...); protected $added_properties = array (...); protected $fulltext_fields = array (...); public $id = '1'; public $name = 'Default node index'; public $machine_name = 'default_node_index'; public $description = 'An automatically created search index for indexing node data. Might be configured to specific needs.'; public $server = 'default'; public $entity_type = 'node'; public $options = array (...); public $enabled = '1'; protected $entityType = 'search_api_index'; protected $entityInfo = array (...); protected $idKey = 'id'; ... }
  $insert = *uninitialized*
  $item = array ('search_api_language' => array (...), 'type' => array (...), 'title' => array (...), 'status' => array (...), 'promote' => array (...), 'sticky' => array (...), 'created' => array (...), 'changed' => array (...), 'field_story_series' => array (...), 'field_story_program' => array (...), 'field_story_iptc_data' => array (...), 'field_search_text' => array (...), 'author:name' => array (...), 'author:roles' => array (...), 'body:value' => array (...), 'field_story_collections:title' => array (...), ...)
  $name = 'field_search_text'
  $query = class InsertQuery_mysql { protected $table = 'search_api_db_default_node_index_field_story_iptc_data'; protected $insertFields = array (...); protected $defaultFields = array (...); protected $insertValues = array (...); protected $fromQuery = NULL; protected $connection = class DatabaseConnection_mysql { ... }; protected $connectionTarget = 'default'; protected $connectionKey = 'default'; protected $queryOptions = array (...); protected $nextPlaceholder = 0; protected $comments = array (...) }
  $table = 'search_api_db_default_node_index_field_search_text'
  $token = 'Array'
  $txn = class DatabaseTransaction { protected $connection = class DatabaseConnection_mysql { ... }; protected $rolledBack = FALSE; protected $name = 'drupal_transaction' }
  $type = 'tokens'
  $v = *uninitialized*
  $value = 'nd'
  $values = *uninitialized*
  $word = array ('value' => array (...), 'score' => 1)
  $words = array ('selfcontained' => array (...), 'take' => array (...), 'sot' => array (...), 'name' => array (...), 'red' => array (...), 'river' => array (...), 'bridgeolson' => array (...), 'number' => array (...), 8812 => array (...), 'incue' => array (...), 'minnesota' => array (...), 'and' => array (...), 'north' => array (...), 'dakota' => array (...), 'trasportation' => array (...), 'officials' => array (...), ...)

Drush command terminated abnormally due to an unrecoverable error.                                           [error]
Error: Cannot create references to/from string offsets nor overloaded objects in
/shared/vash/sandboxes/rickard/archiveportal/www/sites/default/modules/search_api/contrib/search_api_db/service.inc,
line 337

Slight revision to the patch (in -p1 format).

Still throwing an error I can't track down.

rickard@saverem:~/sandbox/archiveportal/www/sites/default/modules/search_api$ drush sapi-i 1 1
Updating Default node index with limit of 1 items
WD php: Warning: htmlspecialchars() expects parameter 1 to be string, array given in check_plain() (line 1476[warning]
of /shared/vash/sandboxes/rickard/archiveportal/www/includes/bootstrap.inc).
WD php: Warning: htmlspecialchars() expects parameter 1 to be string, array given in check_plain() (line 1476[warning]
of /shared/vash/sandboxes/rickard/archiveportal/www/includes/bootstrap.inc).
WD php: Warning: htmlspecialchars() expects parameter 1 to be string, array given in check_plain() (line 1476[warning]
of /shared/vash/sandboxes/rickard/archiveportal/www/includes/bootstrap.inc).
htmlspecialchars() expects parameter 1 to be string, array given in check_plain() (line 1476 of              [warning]
/shared/vash/sandboxes/rickard/archiveportal/www/includes/bootstrap.inc).
htmlspecialchars() expects parameter 1 to be string, array given in check_plain() (line 1476 of              [warning]
/shared/vash/sandboxes/rickard/archiveportal/www/includes/bootstrap.inc).
htmlspecialchars() expects parameter 1 to be string, array given in check_plain() (line 1476 of              [warning]
/shared/vash/sandboxes/rickard/archiveportal/www/includes/bootstrap.inc).
An overlong word (more than 50 characters) was encountered while indexing: .Database search servers currently[warning]
cannot index such words correctly – the word was therefore trimmed to the allowed length.

No idea where check_plain() is getting invoked.

In this patch, I removed two issues:

1) $v should be $v['value'] in the preg_split().
2) In the indexItem() method PHP doesn't like &$token['values'] used by reference.

I also don't understand how or why the foreach ($value as $token) loop is supposed to work, given that inside that loop, $value is reset.

I have to apologize, that patch was really crappy. Sorry that you had to test it, and even tried to figure out what was wrong.
I now rewrote (and, this time, tested) it, should be OK now.

I've been there myself. Will test on Monday.

That patch works. I wonder if we can suppress the warning in some cases. When running this via the new drush commands, I could see thousands of these, which I don't really need.

Through the UI, we could also see up to a hundred messages, for instance.

The ability to flag verbose messages would be a nice addition.

Come to think of it, since indexing is mostly done in cron, using watchdog instead of displaying messages would be preferable anyways …

See the attached patch (although the modification is quite small).
Also interesting would of course be reviews by froezibaer and klausi, before I commit this.

Watchdog makes sense here. ++

Works for me. Can't say much about the code as I'm not that familiar with SearchAPI. I think it would be nice to link the error message "Check the logs for details." to admin/reports/dblog, but that is probably another issue.

I think it would be nice to link the error message "Check the logs for details." to admin/reports/dblog, but that is probably another issue.

As I understand it, watchdog isn't fixed on logging to the DB log, so we probably can't do this. Users will just have to know where to find their system logs. And yes, it would be another issue.

Anyways, good to know this works for you, too. Just committed the patch.

My indexing fails as well, but that was related to another issue. Linking to it for reference: #1346276: Tokenizer should only run on fulltext fields

Still get this error (resulting in not all nodes to be indexed > in fact : only 95% of my nodes are indexed)

An overlong word (more than 50 characters) was encountered while indexing, due to bad tokenizing. Please check your settings for the "Tokenizer" preprocessor to ensure that data is tokenized correctly.

Tokenizer preprocessor is enabled and only running on the Body text (fulltext)

Don't know what else to do.

edit : started tearing apart a node that was not being indexed and seemed that the word "één" was the word that the indexer kept on choking up on ...

Is this a documented issue ? > letters with accents not being treated well by the search api indexer ?
Edit : duplicate of http://drupal.org/node/1144620

#4 work for me. Thank you

I had a hard time to index some especially long WKT values (geodata). I ended up indexing them as full text instead of strings and not running any tokenizers or other filters on the field.

Thanks @basvredeling, your comment gave me the hint I needed.

I plan to move from Search API Solr to standard Search API with MariaDB and Redis for caching. But when I indexed nodes with Geodata values with the new Standard DB server, I got errors like this:

search_api_db  Warning  An overlong value (more than 255 characters) was encountered while indexing:

MULTIPOLYGON (((10.3634 57.3809,10.3599 57.3815,10.3546 57.3813,10.3546             
57.3790,10.3495 57.3793,10.3468 57.3757,10.3458 57.3760,10.3465 57.3778,10.3461     
57.3785,10.3440 57.3791,10.3440 57.3785,10.3431 57.3774,10.3424 57.3758,10.3414     
57.3758,10.3388 57.3761.

Database search servers currently cannot index such values correctly – the value was therefore trimmed to the allowed length.

Under the index I saw this:

Warning message
Some of the used data types aren't supported by the server's backend. See the data types table to find out which types are supported.

Under "Data types", the supported types were listed (Decimal, String, Date, Boolean, Integer, and Fulltext) and then all the unsupported Search API Solr types ...

The Geodata field was set to use "Storage-only", which works only with Solr:

Data Type: Storage-only
Description: A storage-only field. You can store any string and retrieve it from the index but you can't search through it. No
Supported:
Fallback data type: String

After indexing them as Fulltext instead of strings (the fallback data type for Storage-only), all is well, and there are no errors :)

Now, I just wonder ... should Fallback data type for Storage-only be changed from String to Fulltext?