I suggest we use this setup to allow any kind of replication (master, slave and repeater)

<requestHandler name="/replication" class="solr.ReplicationHandler" >
  <lst name="master">
    <str name="enable">${enable.master:false}</str>
    <str name="replicateAfter">commit</str>
    <str name="replicateAfter">startup</str>
    <str name="confFiles">schema.xml,mapping-ISOLatin1Accent.txt,protwords.txt,stopwords.txt,synonyms.txt,elevate.xml</str>
 </lst>
 <lst name="slave">
    <str name="enable">${enable.slave:false}</str>
   <str name="masterUrl">$MASTER_CORE_URL/${solr.core.name}/replication</str>
   <str name="pollInterval">$POLL_TIME</str>
 </lst>
</requestHandler>

using a solrcore.properties or 

-Denable.master=true
-Denable.slave=true

we can dynamically enable slave/master properties.
Source : http://wiki.apache.org/solr/SolrReplication#enable.2BAC8-disable_master....

You could also define the solrproperties file at runtime, but I should test if this works for all the cores. The only thing that you then still have to solve is the multicore replication path issue
-Dsystem.properties=conf\solrcore.properties

Currently at Acquia we use some text replacement methods that are just plain dirty, so if we could embed this clean way in both configs, we can make sure that we will have working configs for both modules.

Any objections?

CommentFileSizeAuthor
#20 1611504-20.patch1.51 KBNick_vh
#19 1611504-19.patch1.49 KBNick_vh
#18 1611504-18.patch1.07 KBNick_vh
#14 1611504-14.patch1.5 KBNick_vh
#8 1611504-8.patch1.79 KBNick_vh
#7 1611504-8.patch3.97 KBNick_vh
#6 1611504-6.patch1.34 KBNick_vh
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Nick_vh’s picture

Nick_vh
he/him
Location Ghent
CreditAttribution: Nick_vh commented
Issue summary: View changes

Updated issue summary.

Nick_vh’s picture

Nick_vh
he/him
Location Ghent
CreditAttribution: Nick_vh commented
Issue summary: View changes

Updated issue summary.

cpliakas’s picture

cpliakas CreditAttribution: cpliakas commented

I really like the idea of having the replication code in there by default to promote HA Solr configs. Great job finding this technique.

The one question I have is about the $MASTER_CORE_URL. Is that something that you have to replace custom, or is there some method in Solr that allows you to replace this using environment variables? My first thought is that we should not put anything in there that requires string replacements custom to Drupal.

Second thought, should we add a solrcore.properties file to the standard config?

Third thought, maybe the Apache Solr Common Configurations module provides a UI that allows for certain customizations, i.e. generating a custom config such as "Set up master slave", change time, etc. The goal would NOT be to make the module required by either Search API or Apache Solr Search Integration, but rather a tool to customize the defaults that are likely to work with both Search API and Apache Solr Search Integration? This is off topic, but I think relevant to the approach we take to the custom replacements. Added discussion about this at #1611532: Purpose of a Apache Solr Common Configurations module.

Chris

Nick_vh’s picture

Nick_vh
he/him
Location Ghent
CreditAttribution: Nick_vh commented

The replacement patterns are solr patterns so there is nothing Drupaly in there :-) See the docs page I linked to for more info in regards of there patterns.
The properties file can actually change almost anything in the solrconfig.xml if it is defined as a variable.

I don't think we should include the properties file since it is not required for a standalone server. It is one extra file people will have to copy and it does not really serve much purpose in a non-HA environment

cpliakas’s picture

cpliakas CreditAttribution: cpliakas commented

Oh cool. I think I get it. So everything in ${blah.key:value} is read from solrcore.properties? If that is the case, we could be more consistent with naming conventions and do a ${master.core.url:http://example.com} nomenclature or something.

I do like the idea of including a solrcore.properties file, because even though it is one more thing to copy, it would be much easier to say, "edit this file to enable HA" as opposed to create this file, add these key / values etc. Copying a file is something pretty much anyone can do, whereas creating a value and adding key value pairs, although simple, is more prone to error than just having to change a false to a true.

cpliakas’s picture

cpliakas CreditAttribution: cpliakas commented

In addition, if we are adding custom variables that rely on a configuration file, I would think that is something we should provide a default of.

Nick_vh’s picture

Nick_vh
he/him
Location Ghent
CreditAttribution: Nick_vh commented

Some good information how we can really leverage this
http://wiki.alfresco.com/wiki/Alfresco_And_SOLR#SOLR_core_properties_in_...

Nick_vh’s picture

Nick_vh
he/him
Location Ghent
CreditAttribution: Nick_vh commented
FileSize
1611504-6.patch1.34 KB
Nick_vh’s picture

Nick_vh
he/him
Location Ghent
CreditAttribution: Nick_vh commented
FileSize
1611504-8.patch3.97 KB

Now with the solrconfig.properties

I think I'm diffing with wrong branch. Will update asap

Nick_vh’s picture

Nick_vh
he/him
Location Ghent
CreditAttribution: Nick_vh commented
FileSize
1611504-8.patch1.79 KB

This one should be better :-) Compared to master

I tested this setup for a master/slave configuration and I can confirm it works as expected. Makes configuring the solr core much easier

Nick_vh’s picture

Nick_vh
he/him
Location Ghent
CreditAttribution: Nick_vh commented

We should look at more properties we want to define dynamically.
This could also be a perfect way to make the solrconfig.xml dynamic for both apachesolr and search_api

Nick_vh’s picture

Nick_vh
he/him
Location Ghent
CreditAttribution: Nick_vh commented
Status: Active » Needs review
drunken monkey’s picture

drunken monkey
he/him
Primary language German
Location Vienna, Austria
CreditAttribution: drunken monkey commented
Status: Needs review » Active

This seems like a brilliant idea to me! Making advanced setups easier is definitely a good thing.

Regarding we should include a solrcore.properties file, I'm not really sure. On one hand I think it'll be confusing to people who want a simple setup, we would have to document this well. On the other hand, of course you're right that we should make this as easy as possible.
Maybe we can put an example online (probably here in the repo) and link to it in the README.txt?

Also, one doesn't really need a properties file but could also just specify -DMASTER_CORE_URL=foo on startup, right?
I'm with Chris here, though, we should keep the variable names consistent unless there's a good reason not to.

We should look at more properties we want to define dynamically.
This could also be a perfect way to make the solrconfig.xml dynamic for both apachesolr and search_api

Yes, definitely! This is something I wondered anyways, and I think this would make it much easier for people to customize their Solr servers, especially regarding later module updates.
New issue for that?

And: Is this also possible for schema.xml? Could be great, e.g., for localisation, or even for dynamically selecting the type used for text fields.
The less users will need to edit the config files directly, the better.

drunken monkey’s picture

drunken monkey
he/him
Primary language German
Location Vienna, Austria
CreditAttribution: drunken monkey commented
Status: Active » Needs review

Patch looks good, and if you even tested this, great!
It just complains about a missing newline at the end of the properties file.

Nick_vh’s picture

Nick_vh
he/him
Location Ghent
CreditAttribution: Nick_vh commented
Status: Needs review » Needs work 
+++ b/conf/3.x/solrconfig.xmlundefined
@@ -1031,19 +1031,19 @@
+     <str name="masterUrl">${master_core_url}/${solr.core.name}/replication</str>

I would actually go for master_core_only. I tried implementing this in our Solr cloud and it becomes too tricky if the cores that are replicating need to have the same name.

Also, we should take out the conf-files I think. But not sure about that just yet

Nick_vh’s picture

Nick_vh
he/him
Location Ghent
CreditAttribution: Nick_vh commented
Status: Needs work » Needs review
FileSize
1611504-14.patch1.5 KB

Update

Please review

cpliakas’s picture

cpliakas CreditAttribution: cpliakas commented

Looks good to me.

Stupid nit-picky stuff, but I think we should conform to how Solr is formatting these variables. They have some work to do on consistency, but it appears that groups are broken up by "dots" and camelCase is used to separate words in between dots.

For example: solr.data.dir and solr.abortOnConfigurationError

I know this is minor, but I feel we should be consistent and move away from underscores.

Nick_vh’s picture

Nick_vh
he/him
Location Ghent
CreditAttribution: Nick_vh commented

masterCoreUrl is perfectly ok for me ;-)

cpliakas’s picture

cpliakas CreditAttribution: cpliakas commented

ThatsGreatThanksForResponding.

Nick_vh’s picture

Nick_vh
he/him
Location Ghent
CreditAttribution: Nick_vh commented
FileSize
1611504-18.patch1.07 KB
Nick_vh’s picture

Nick_vh
he/him
Location Ghent
CreditAttribution: Nick_vh commented
FileSize
1611504-19.patch1.49 KB
Nick_vh’s picture

Nick_vh
he/him
Location Ghent
CreditAttribution: Nick_vh commented
FileSize
1611504-20.patch1.51 KB
drunken monkey’s picture

drunken monkey
he/him
Primary language German
Location Vienna, Austria
CreditAttribution: drunken monkey commented
Status: Needs review » Reviewed & tested by the community

Fine by me.

Nick_vh’s picture

Nick_vh
he/him
Location Ghent
CreditAttribution: Nick_vh commented
Status: Reviewed & tested by the community » Fixed

Committed

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

Anonymous’s picture

(not verified) CreditAttribution: commented
Issue summary: View changes

Updated issue summary.