Based on the functionality diff, the MaxDoc directive in <autoCommit> is set to 2000 in Apache Solr Search Integration and not set at all in Search API Solr. The relevant discussion about setting the value to 2000 is located at #919694: AutoCommit MaxTime seems ridiculously long for normal use, let's change it to 20 seconds..

Comments

pwolanin’s picture

pwolanin commented

For production and depending on whether you have master/slave, how many docs you index, etc these values should possibly be varied. Generally people haven't complained since that issue, though having a "commit" button might be handy for dev if you don't change the config.

drunken monkey’s picture

drunken monkey
he/him
Primary language German
Location Vienna, Austria
commented

This change in the Search API was introduced because of #1225926-7: Fix performance problems in indexing workflow, where someone seemed to run into problems when maxDocs was set. Didn't you have any complaints in that direction? Or could maybe anyone figure out another reason for that? The issue was a bit mysterious, I admit, but removing maxDocs seemed to reliably fix this.

And I guess leaving too many new documents uncommitted has some performance impact?

cpliakas’s picture

cpliakas commented
Component: solrconfig.xml » Code

pwolanin, drunken monkey,

Thanks for the info and the background. I am still toying around as to whether or not this will have a module component to it, where it could do something like scan certain configs and provide recommendations based on your setup. Not sure if that is even possible, but could be cool.

Anyways, the default in 3.5 is now 10000, so 5x higher that the setting in Apache Solr Search Integration. It looks like the issue at [#1225926#comment-4918860] is the exceeded limit of maxWarmingSearchers=X error, which means that more than two commits were triggered at the same time given the maxWarmingSearchers setting of 2. I am assuming that what happened is that during the batch indexing more that 4k nodes were being sent at a time triggering more than 2 commits.

My thought is that we should increase to the 3.5 default. The Drupal modules should be smart about triggering commits anyways, so this gate shouldn't be run into often if ever. I am not in favor of removing it, because as we are seeing in Acquia Search people are starting to index non-Drupal data that is managed outside of Drupal. In that case we cannot assume that their application will be as smart about commits as the Drupal modules. Therefore having a higher maxDocs gate would at least catch exceptions.

cpliakas’s picture

cpliakas commented
Component: Code » solrconfig.xml

Adding new component flag.

drunken monkey’s picture

drunken monkey
he/him
Primary language German
Location Vienna, Austria
commented

OK, let's use the default again!

pwolanin’s picture

pwolanin commented
Component: Code » solrconfig.xml

an explicit 10k is ok for me.

pwolanin’s picture

pwolanin commented

an explicit 10k is ok for me.

cpliakas’s picture

cpliakas commented
Status: Active » Reviewed & tested by the community

Excellent! Marking as RTBC. Making great progress here, thanks everyone for your participation. I know this initiative isn't the most exciting of topics :-).

Chris

nick_vh’s picture

nick_vh
he/him
Location Ghent
commented

It actually is exciting!

cpliakas’s picture

cpliakas commented
Status: Reviewed & tested by the community » Fixed

Marking as fixed. Nothing to commit as it is the default setting.

drunken monkey’s picture

drunken monkey
he/him
Primary language German
Location Vienna, Austria
commented

Excellent! Marking as RTBC. Making great progress here, thanks everyone for your participation. I know this initiative isn't the most exciting of topics :-).

It actually is exciting!

I'm also pretty excited about finally being able to improve the config files with some competent help. ;) And having common configs definitely helps us all.

Also, Chris: You can link to specific comments in issues with [#NID-COMMENTNO] (where COMMENTNO is the number in the issue, not the cid (so 1, 2, 3, …)), in case you didn't know.

cpliakas’s picture

cpliakas commented

You can link to specific comments in issues with [#NID-COMMENTNO] (where COMMENTNO is the number in the issue, not the cid (so 1, 2, 3, …)), in case you didn't know.

I did not know that! Thanks for the formatting tip.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.