schema.xml defines two fieldTypes text and textTight. Both use the same resources synonyms.txt, stopwords.txt and protwords.txt.
But this doesn't work because text applies a MappingCharFilterFactory using mapping-ISOLatin1Accent.txt which converts some characters. For text this needs to be respected in the resources synonyms.txt, stopwords.txt and protwords.txt. For textTight it must not be respected. So both fieldTypes must use different resources!

Due to the fact tat textTight is not used by default I recommend to simply remove textTight from schema.xml

CommentFileSizeAuthor
#7 750426-remove-textTight-7.patch3.09 KBpwolanin
#4 750426.patch1.41 KBmkalkbrenner

Comments

robertdouglass’s picture

robertdouglass commented

Agree. Will point this out to Peter as well.

pwolanin’s picture

pwolanin commented
Version: 6.x-1.0 » 6.x-1.x-dev

I think this was from some ancient default schema - if we remove it, maybe we can think about a better substitute?

mkalkbrenner’s picture

mkalkbrenner
Primary language German
Location 🇩🇪
commented
Version: 6.x-1.x-dev » 6.x-1.0

I think for 6.x-1.x it should be removed because it's not used. The situation for 6.x-2.x might be different.

I attached a patch to remove it. I already removed support for fieldTight from Apache Solr Multilingual which currently uses Apache Solr Search Integration 6.x-1.x.

mkalkbrenner’s picture

mkalkbrenner
Primary language German
Location 🇩🇪
commented
Version: 6.x-1.0 » 6.x-1.x-dev
StatusFileSize
new 750426.patch1.41 KB

sorry. changed back version and attached patch.

robertdouglass’s picture

robertdouglass commented

We didn't start using it in 6.2 either, so I'm thinking this is a no-brainer.

mkalkbrenner’s picture

mkalkbrenner
Primary language German
Location 🇩🇪
commented
Title: fieldType textTight conficts with fieldType text » fieldType textTight conficts with fieldType text and textSpell

Same issue with textSpell, I think:

   <fieldType name="textSpell" class="solr.TextField" positionIncrementGap="100">
     <analyzer>
       <tokenizer class="solr.StandardTokenizerFactory" />
       <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/>
       <filter class="solr.LengthFilterFactory" min="4" max="20" />
       <filter class="solr.LowerCaseFilterFactory" /> 
       <filter class="solr.RemoveDuplicatesTokenFilterFactory" /> 
     </analyzer>
   </fieldType>
pwolanin’s picture

pwolanin commented
StatusFileSize
new 750426-remove-textTight-7.patch3.09 KB

committing patch in #4 plus schema version change (attached) - maybe need to think more about how to fix textSpell.

robertdouglass’s picture

robertdouglass commented
Version: 6.x-1.x-dev » 5.x-2.x-dev

Thanks.

pwolanin’s picture

pwolanin commented
Status: Active » Patch (to be ported)
jpmckinney’s picture

jpmckinney commented
Status: Patch (to be ported) » Fixed

Fixed in 5-2 http://drupal.org/cvs?commit=361662

mkalkbrenner’s picture

mkalkbrenner
Primary language German
Location 🇩🇪
commented
Version: 5.x-2.x-dev » 6.x-1.x-dev
Component: schema.xml » solrconfig.xml
Status: Fixed » Needs work

See comment #6. The conflict still exists for textSpell in solrconfig.xml.

jpmckinney’s picture

jpmckinney commented

I assume you mean schema.xml. Should we just change stopwords.txt to another filename for textSpell? (We will need to modify the installation instructions, I assume.)

jpmckinney’s picture

jpmckinney commented
Title: fieldType textTight conficts with fieldType text and textSpell » fieldType textSpell conficts with fieldType text
Status: Needs work » Active

No patch. Setting to active.

jpmckinney’s picture

jpmckinney commented
Version: 6.x-1.x-dev » 6.x-2.x-dev
mkalkbrenner’s picture

mkalkbrenner
Primary language German
Location 🇩🇪
commented
Version: 6.x-2.x-dev » 6.x-1.x-dev
Component: solrconfig.xml » schema.xml

yes. it's schema.xml. Renaming the file to stopwords_spell.txt seems to be the easy solution. The downside will be that no such file comes with the apache solr example where we get stopwords.txt from.

BTW If we decide to use to different files it will be easy to generate these different files from one source using Apache Solr Multilingual.

BTW I still think this issue a bug in 6.x-1.0 and should not be moved to 6.x-2.x.

jpmckinney’s picture

jpmckinney commented

I fix bugs in 6.x-2.x and then backport (same as the Drupal project), so that the most recent branch is never out of date.

mkalkbrenner’s picture

mkalkbrenner
Primary language German
Location 🇩🇪
commented
Version: 6.x-1.x-dev » 6.x-2.x-dev

I rethought this issue. The mapping-ISOLatin1Accent filter only causes a serious issue for non-English solr indexes.

So give me some feedback on this approach:

- textTight has been removed already from cvs, fine.
- All the configuration that ships with the apachesolr module is only suitable for English indexes. The mapping-ISOLatin1Accent filter doesn't matter for stop words as long as they are English stop words. So we should not touch the default definition of fieldType textSpell.
- Apache Solr Multilingual is able to create two different stop word files on the fly from one source and to change schema.xml accordingly.

So we should close this issue here and leave it to Apache Solr Multilingual to solve it.

jpmckinney’s picture

jpmckinney commented

I'm fine with delegating responsibility to apachesolr_multilingual for now. I hope sometime in the future we can consider merging apachesolr_multilingual back into apachesolr.

mkalkbrenner’s picture

mkalkbrenner
Primary language German
Location 🇩🇪
commented
Status: Active » Fixed

see #788842: fieldType textSpell conficts with fieldType text

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.