HTTP Status 500 - Illegal character (NULL, unicode 0) encountered

That's odd - maybe there is a problem with the code intended to strip all such chars?

Is it possible some odd multi-byte char could have the 1st byte stripped and then leave a null byte?

Can you isolate the offention portion(s) of that node?

We're trying to isolate the offending node. Can you give us some direction to that end?

Here's what the search index page tells us:

The search index is generated by running cron. 3% of the site content has been sent to the server. There are 88328 items left to send.

The server has a 4 min delay before updates are processed.

Number of documents in index: 3266 (66 sent but not yet processed)

Numer of pending deletions: 0

.... We're thinking 3266 (or 3277) might be the offending node but we're not sure what order Solr indexes in.

Thanks,
Kevin

When Solr hits a bug like this, the module will not continue indexing as if everything is fine.

So I would recommend setting the nodes per cron run to 100 or something, and keep running it. When it craps out, change it to 50 or something. If it still craps out, reduce to 20, etc until you narrow in on the one which is causing the problem (in which case you are runnin 1 per cron run).

Alternately, you can facilitate this process a little if you can write a little code by getting into the error handling during the index operation, and spitting out whatever vars will help you identify the node or the raw XML that was sent.

HTH,
Jacob

Okay, so I'm guessing Solr just starts at the beginning of the node table (nid =1) and works it's way down the list. (apachesolr_search_node lists nodes in a similar fashion).

So, guessing again, when the index page says it was able to index 3266 documents, that should mean the problem is with the 3267th node in the node table.

But wait, Solr doesn't index documents with status = 0 so I'll need to exclude those from my count.

I think I understand. I'll work through the logic(?) listed above and see if I can pinpoint the problematic node.

I think I was able to find the problematic node and set it to unpublished.

The first try (counting nodes with status =1) did not work so my presumption above about Solr ignoring documents with status =0 was probably incorrect.

When I tried a new query, and ignored status, I was able to disable a range of nodes (just to be safe) that appears to include the one with the offending character.

I found the "range" with this query:

Then I "unpublished" nodes in the range by setting their status = 0:

Finally, I deleted the entire Solr index and waited for the magical 3267th node to be indexed. After 24 hours, we're now indexing node 41,000 so I believe the above steps were a success.

After the site has completed indexing (couple more days) I'll try to isolate the exact node that was causing the problems.

We did have similar problems previously with odd character encodings (hence the regex I posted above). Obviously we'd be happy to make it more robust if there is a way to do so.

Thanks pwolanin,

Do you know of a way that we can tell the sequence documents are indexed and which ones have (or have not) been indexed?

We're running into the same error at "Number of documents in index: 52345" but we're having a devil of a time locating the offending node.

The steps we took before (described above) are not working this time.

Hopefully re-indexing is not a step that we need to take ... having made it this far into the process, we're reluctant to start over.

TIA,
Kevin

The apachesolr module maintains its own table and an index into that table as a Drupal variable. Use the devel module or look in the variables table to find the value of 'apachesolr_index_last'. Assuming you can limit to a few nodes indexed per cron run this should let you narrow closely to the problem node, e.g.:

Where you have to sub in for %d the values that are stored in that variable. This will tell you then next 5 nodes in the queue to be indexed.

Note, that if you want to unpublish nodes and have apachesolr respect that, you must either use the UI or use node_save(). Touching {node} directly means that the {apachesolr_search} table is not updated.

> Note, that if you want to unpublish nodes and have apachesolr respect that, you must
> either use the UI or use node_save(). Touching {node} directly means that the
> {apachesolr_search} table is not updated.

Reviewing the code, it seems that _apachesolr_nodeapi_update() is called each time cron runs. Wouldn't that remove content from apachesolr_search if content is unpublished manually?

Oh, yeah, it will get fixed up on cron, thouh it was really intended for the case where an update to solr fails.

I applied the attached patch to track down the problematic node -- any chance of applying it or something like it? It dumps a list of up to 20 nodes in the chunk last sent to Solr when indexing fails.

(To track down the specific bad node, I temporarily hacked the following line in the same function:

changing 20 to 1 -- this resulted in a very specific error:
"Indexing failed on one of the following nodes: #" where # was the problematic node that we were unable to index.)

Indexing apparently failed due to the following characters in a comment:
"à uß%������˜‡í ����������������"

The characters do not appear to be copying and pasting correctly in my browser, however, so the above string may work for you in your testing.

Attachment	Size
apachesolr.module.patch	672 bytes

Yes, something like that patch looks like a good idea, though I might just concat that text with the watchdog call we are already making?

Try adding this additional processing to the indexing step prior to the step where we strip low bytes chars:

We tried this at one point but it did not actually strip low byte chars (e.g. char 30). If your problem is due to some funky encoding or multi-byte char, this might be useful as an initial clean-up to insure that the text looks reasonably like the expected utf-8

We will test this within the next few days and let you know whether or not it fixes this problem.

As for whether you concat the text or display it in its own watchdog is not important, just that the data is available to quickly debug and track down problematic nodes.

Where is the best place to put that code? I tried adding it to the top of apachesolr_strip_ctl_chars(), but it did not help. We still get the "500 - Illegal character" error.

There are 2 places it might be needed. See patch.

Attachment	Size
iconv-503644-15.patch	1.09 KB

Actually this might be better - it's a little silly to get the preg_replace in 2 places.

Attachment	Size
iconv-503644-16.patch	1.12 KB

I tried both in my previous testing, but neither worked. I also tried numerous variations involving combinations of iconv(), mb_convert_encoding() and mb_detect_encoding(), but none solved the problem.

Recently I unpublished the offending comment, and in doing so the problematic binary content was evidently stripped away. Re-publishing it we were able to index it without problems. I suspect now that this comment was either originally imported from another CMS, or created with a much earlier version of Drupal. In any case, I unfortunately no longer have the content available for easy testing.

Ok, forget those patches. d.o is having the same issue and I think I tracked it down - if there is an encoded character like #031 we decode it with html_entity_decode() and it slips though because we removed those characters before the decode step, rather than after.

Attachment	Size
reorder-503644-17.patch	1.67 KB

I just confirmed the fix locally for one known problem from d.o.

Confirm this patch works.

Added  to a node, upon saving the node ApacheSolr tries to index it, and watchdog reports the Java exception.

Adding in the patch and indexing that same node fixes the problem.

updated with the additional logging

Attachment	Size
reorder-503644-21.patch	2.65 KB

committed to 6.x

Deployed on drupal.org, seems to work as expected.

Automatically closed -- issue fixed for 2 weeks with no activity.