Should we be calling $solr->commit() after every update?

Background on autocommits and optimize

The following code in solrconfig.xml is where it happens:

<autoCommit>
      <maxDocs>10000</maxDocs>
      <maxTime>1000</maxTime>
 </autoCommit>

what this means is that it will run a commit operation when there are more than 10000 docs waiting to be committed OR 1 second (1000ms) after a document has been added (suppose if no other autocommit is in play). I really don't understand why one would want a config like this by default. A more sane value would probably be:

 <autoCommit>  
      <maxDocs>10000</maxDocs> <!-- maximum uncommited docs before autocommit triggered -->
      <maxTime>50000</maxTime> <!-- maximum time (in MS) after adding a doc before an autocommit is triggered -->
</autoCommit>

Unfortunately, this si commented out by default (at least in 1.2x)

This means that when apachesolr sends over some documents in a batch, as long as they don't send any more docs for 50 sec, a commit will happen. This seems pretty good. Then, if there is a backlog for whatever reason, it will run eventually @ 10,000 docs (although since most drupal sites are much smaller, this may be too large for our purposes).

I think we *should* leave the commit() call upon deletion or node_Access updates, because that should take effect ASAP.

Optimization is a different beast. I'm not sure, but I believe we aren't even doing any optimization currently. Perhaps we should suggest people to set it up on their own.

The cheap way is to optimize after every commit using a postCommit handler like this:

<listener event="postCommit" class="solr.RunExecutableListener">
      <str name="exe">optimize</str>
      <str name="dir">solr/bin</str>
      <bool name="wait">true</bool>
</listener>

But this can be heavy. It is probably preferred to optimize on cron once or twice a day. PostOptimize a commit is always called, and generally a snapshot is created for replication.

Practical changes Needed in the module

• Remove the ->commit() calls from the cron update_index section
• Change the documentation telling users to uncomment the autoCommit settings from their schema.xml
• Add documentation telling users to add their own optimize calls in cron

K, there it is, should we go for this?

I'm all for usability too, but there are a couple concerns here:

1. The setting isn't in schema.xml, but solrconfig.xml... This isn't a big issue, but instead of providing (and having to maintain a solrconfig.xml), I think just telling them to uncomment a few lines is not too bad. After all, they are also configuring a java server :)

2. Running optimize via drupal cron. While I like the premise, I don't know if this will fly. optimize commands can take a REALLY long time, and I'd be worried about that when cron is supposed to run every 15min on most CMSs. Also, it could cause PHP timeouts if the user has default timeouts set (unless we unset it). It could be done, but I don't like the idea of combining them. Someone running a solr server (which is different from using the module) will have to be a bit of sysadmin to get the thing going anyway. I think if our instructions are good, they should be able to add a cronjob for optimization...

What do you think?

That's true, we could use popen() or just system with a background indicator, but I think it is unneeded. Consider this:

1. Users who have drupal cron running can setup cron.
2. Users who are running a solr server have a dedicated box of VPS, so they are not using poormanscron
3. Users who are using a solrServer they do not maintain should not be sending index requests. The server should initiate these requests. It is possible for a client to initiate an optimize command from outside the box if the admin allows it, but not recommended.
4. optimize is not needed. Certainly it speeds things up, but it is only a 10% performance hit.

I'm okay going forward with it, but I think the larger issue is: should the indexing client be telling the server when to run operations needed for replication and for optimization, or should the server do it based on its needs.

From http://wiki.apache.org/solr/SolrPerformanceFactors:

Every time a new index searcher is opened, some autowarming of the cache occurs before Solr hands queries over to that version of the collection. It is crucial to individual query latency that queries have warmed caches.

The three relevant parameters:

* The number/frequency of snapshots is completely up to the indexing client. Therefore, the number of versions of the collection is determined by the client's activity.
* The snappullers are cron'd. They could run every second, once a day, or anything in between. When they run, they will retrieve only the most recent collection that they do not have.
* Cache autowarming is configured for each cache in solrconfig.xml.

So if a client is spamming commits, it will create a huge load on slave servers to keep updating, and it will also be a drag on disk space.

Optimize commands can be run more often, but is also a creator of CPU load.

What about this:

By default:
We provide a commit and an optimize command (perhaps backgrounded) after the entire cron operation has finished.
AND:
We create a new setting (radio) for whether you are handling this on the client side or the server side.

I see the need for usability, but from my discussions about this on the solr list, all the responses I've gotten are suggesting the later is a best practice.