Tough one... The schema.xml

Thanks Nick. This will be a tricky one. The diff is much appreciated, and thanks for the explanations.

Adding new component flag.

Yes, that really is a tough one. I'd have said we should leave that for when we have the solrconfig.xml figured out, but I guess we might as well start now. However, we probably should split this as well?

In any case, I guess having common type definitions might be a good sub-ordinate target and also achievable.
For the actual field definitions, a quick look tells me there won't be much chance of those being unified. The approaches are just too different, as far as I can tell. Maybe unifying some fields, e.g., the required ones, might be possible, but I'm not sure how much of an advantage that would bring us.

+++ b/solr-conf/schema-solr3x.xmlundefined
@@ -52,15 +54,11 @@
+    <!-- numeric field types that can be sorted, but are not optimized for range queries -->
+    <fieldType name="integer" class="solr.TrieIntField" precisionStep="0" omitNorms="true" positionIncrementGap="0"/>
+    <fieldType name="float" class="solr.TrieFloatField" precisionStep="0" omitNorms="true" positionIncrementGap="0"/>
+    <fieldType name="long" class="solr.TrieLongField" precisionStep="0" omitNorms="true" positionIncrementGap="0"/>

Search API Solr also uses Trie fields, but we didn't rename the type. This way, people can still use the "normal" types, even though by default only Trie fields are used.

Not sure when this was added, but it probably solved an existing problem :-)

Well, you'll only very rarely want case-sensitive matching in fulltext searches, so I guess that makes sense. And since this field type isn't used by default, I guess adding it to the Search API side wouldn't hurt, either.

Why was this commented out? Seems like we want processing of the, by default, English language?

Why? Why should English be the default?
In the end, no matter what language we choose, most people will have to change it. Maybe some won't need any setting at all, and in any case that will (very probably) be much better than having stemming for a wrong language.
These are about the thoughts that lead me to that decision.

In regards of the english stemming. I think we should design for the 90% and those that want to modify it can do so afterwards. Having a sensible default only seems logical to me. On a sub-note, I am natively Dutch but I do expect things to work as fluent as possible in English and have a set of clear instructions somewhere that explains how to modify it to support another language.

I highly doubt that 90% is anywhere near the right figure.

Adding a defaultSearchField is always useful when you want to debug solr directly in the admin interface.
We should probably discuss this since Search Api does not have such a field

Since there is no field like "content", which will always contain sensible data, in Search API Solr, we cannot easily do that. The best we could do is specify "id" or something like that, to at least avoid errors. Don't know if that makes much sense, though, as the error more clearly shows that you should explicitly specify a field.

This diff combines both schema's without making a cut on functionality or backwards compatibility.

Nick, that's awesome!

Great work!
It makes the schema pretty cluttered, especially in comparison with the Search API, but I guess if this really has no disadvantages regarding functionality, that's probably worth it. We'll just have to unify the type definitions now, up to a certain extent. Specifically, we seem to use a very different approach regarding strings. We might have to introduce two different dynamic fields for those …
Other than that, how compatible are the types used (or, to what extent are they equal)? If they differ in other aspects, maybe we'll have to generally introduce prefixes for our dynamic fields (i.e., all apachesolr fields start with "a_", all Search API ones with "s_").

However, if we have to change field names, or to what fields what data is indexed, this would necessitate re-indexing for users and also make updating more complicated (on code update, users would have to immediately replace the config files, restart Solr and re-index, or things would break).
So maybe forcefully (completely) unifying the schema, too, won't be worth it after all? Having uniform type definitions (and then maybe use different types for the same fields) would be a good first (and maybe only) step here.

I've been talking to a lot of people about this initiative and they all vote yes to have this. Now, I've also asked if a default English would be a problem or not and not a single one of them was against the idea of having a default English schema.

I really think and press on the importance of an English default schema and moving the multilanguage issue to another issue. This way we can enable the stopwords/protwords/... at once and allow an easy integration. A readme could be included that explains the possible configuration options for different languages

Hm, I guess it would be OK to do that …
However, at the very least we should make the "English" a variable, so people can easily change the language (and then just replace the stopwords, etc., files).

+++ b/solr-conf/schema-solr3x.xml
@@ -390,15 +407,19 @@
+	 <!-- This field is used to build the spellchecker index -->

Tab.

+++ b/solr-conf/schema-solr3x.xml
@@ -432,6 +453,8 @@
+   <!-- Search Api needs a t_* field for backwards compatibility -->

It's not really for backwards-compatibility if we still use it.
But by the way, is there any merit to distinguishing single-valued from multi-valued text fields? I couldn't really think of any, thus only t_*.

+++ b/solr-conf/schema-solr3x.xml
@@ -443,6 +466,12 @@
+   <!-- These fields are used for facetting on string fields, so case and
+        special characters are correctly displayed. -->

We should probably also note that these are only used by the Search API.

If you want to say they are for facetting, it conflicts with the namespace philosophy that the namespace is a abbreviation of the field type (string, int/...).
Could we rename them somehow? Having a fss and f_ss is not so nice.

So does, e.g., sort_*, and so do the three-letter prefixes Apachesolr uses when compared to the Search API.
We should maybe just make the distinction clearer, which fields are used by both modules and which only by one. This way, f_ss_* and fss_* would be more clearly separated and I wouldn't see such a problem with that.
(That is, provided we choose to unify the field definitions at all, that is.)

There are two trailing spaces, but apart from that I can live with this. ;)

Amazing to see we really managed to do this!