Currently the Search API stores a lot of unnecessary stuff in the options settings (making #1228726 necessary). This makes it painful when working with Features and introducing some small changes to the index that then might lead to large differences in the export code.
Ideally, only the fields that get indexed are stored in the options settings.

Comments

Title:Reduce Search API index options setting sizeReduce size of stored index fields settings
Category:bug» task

(Clarification: We won't touch options besides the fields settings here, I think.)

Current plan:
- Remove "name" and "indexed" key from field settings.
- Only store indexed fields in settings.
- Provide an API method (SearchApiIndex::getFields()) to access the previously available information (with unindexed fields, names and additionally descriptions).
- Related fields will be added via a special $index->options['additional fields'] entry. (This also means unused related entities will now automatically vanish again when the fields form is saved.)

Patch is under construction, it's not too trivial.

- Store "boost" only for fulltext fields, and if it is different from "1.0".

Title:Reduce size of stored index fields settingsReduce size of stored index settings

(Clarification: We won't touch options besides the fields settings here, I think.)

Disregard that – just checked, and for me about 2/3 of memory size (after the above changes to the fields settings) come from the „Aggregated fields“ data alteration alone. While, for testing purposes, I do have several aggregated fields defined, this is of course a tremendous waste of space.
So I'm gonna apply the above general rules also to those settings. This shouldn't have any impact on other modules, however, as I guess none of them manually checks the internal settings of that data alteration.

Status:Active» Needs review
StatusFileSize
new41.52 KB

Most processors also have a "fields" setting, this was also shrunk down to the essentials.

The attached patch works for me (with patches to several other modules, which I'll post in a minute to those) and reduces the stored size of the index options by a factor of about 10 (again, for me).

Status:Needs review» Needs work

Tested your patch (in combination with the Solr Search) and the Features export code for Search API index fields was reduced from around 1000 to 25 lines, yeah :)

One small problem I've found so far:

Notice: Undefined index: name in theme_search_api_index() (Zeile 761 von search_api/search_api.admin.inc).

--> the indexed field names in the search index view page are missing.

Status:Needs work» Needs review
StatusFileSize
new43.07 KB

Ah, you're right, missed that one.

Status:Needs review» Fixed

Committed.

StatusFileSize
new28.61 KB

I am get a problem with this commit, becouse field names are deleted from index config, but search_api_views is not fixed.
See attach for details

Status:Fixed» Needs work

I had a lot of issues with that too. It seems that this commit partly removed the fields settings from the index configuration (?!?).

I had a lot of issues with that too. Also with search_pages modules and the extend_searche_page.

Status:Needs work» Needs review
StatusFileSize
new3.52 KB

I am get a problem with this commit, becouse field names are deleted from index config, but search_api_views is not fixed.
See attach for details

Ah, yes, sorry, missed that spot. Patch attached.
The fix is already contained in #1231512: Use real Relationships instead of level magic in Views integration, too.

I had a lot of issues with that too.

You'll have to be a bit more specific there, I fear.
However, I've already spotted two bugs myself, which should be fixed by the attached patch.

It seems that this commit partly removed the fields settings from the index configuration (?!?).

As previously listed:

- Remove "name" and "indexed" key from field settings.
- Only store indexed fields in settings.
- Provide an API method (SearchApiIndex::getFields()) to access the previously available information (with unindexed fields, names and additionally descriptions).
- Related fields will be added via a special $index->options['additional fields'] entry. (This also means unused related entities will now automatically vanish again when the fields form is saved.)
- Store "boost" only for fulltext fields, and if it is different from "1.0".

Field selection (in "Processor settings" section) is lost after saving settings in index workflow page. Is it possible to fix this?

StatusFileSize
new4.43 KB

Ah, yes, thanks for catching that! I wholly blame Form API's many pitfalls.

Extended patch attached.

Thank you. Th problem #15 is fixed.

Status:Needs review» Needs work

Just realized that for one index some of my facet blocks are gone (I guess it's related to this issue). This only affects those from related entities (e.g. Profile -> Field collection -> Taxonomy terms), the others ones (fields on the base entity) are still there.

Furthermore, all my views field with '(indexed)' (in combination with #1231512: Use real Relationships instead of level magic in Views integration) disappeared.

Just realized that for one index some of my facet blocks are gone (I guess it's related to this issue). This only affects those from related entities (e.g. Profile -> Field collection -> Taxonomy terms), the others ones (fields on the base entity) are still there.

Define „gone“.
If they still appear on the „Facets“ tab, but (when enabled) nowhere on the „Blocks“ page, this also happens to me from time to time and seems to be a weird bug in the Facet API. Something that should never be NULL is suddenly set to NULL for some facet blocks and chaos ensues.

However, I can't reproduce either of these two, so I'll need a bit more details. Let's just discuss this this afternoon.

StatusFileSize
new4.83 KB

I had several "Undefined index: value" and "Undefined index: original_value" error preventing correct indexing when using an Features exported index in SimpleTest that I traced to usage of $this->options['fields'] in SearchApiIndex::index. Replacing it with $this->getFields() fixed the issue.

Here is an updated version of the patch in #16 with this additional fix.

I had several "Undefined index: value" and "Undefined index: original_value" error preventing correct indexing when using an Features exported index in SimpleTest that I traced to usage of $this->options['fields'] in SearchApiIndex::index. Replacing it with $this->getFields() fixed the issue.

If you executed update.php properly, this shouldn't happen. Using $this->options['fields'] should be completely correct in this case, as we only need the indexed fields' 'type' keys.
How exactly did you trace this? And what do you mean with „using an Features exported index in SimpleTest” – did you try to import an index that was exported before this change? This of course won't work, like with all changes that require DB updates.

Also, I'm pretty sure you mean original_type, not original_value.

Status:Needs work» Needs review
StatusFileSize
new6.5 KB

Spotted the bug in getFields() that caused the whole trouble for Matthias. And once again it was the type of bug where you wonder afterwards that anything at all worked … Quite a miracle all worked for me – maybe I should rethink my testing setup.

Anyways, patch with the fix attached, please test!

Status:Needs review» Reviewed & tested by the community

Thanks Thomas for taking a look at it. My views fields and facets are back now. And if you need some complex configs for testing, let me know ;)

Status:Reviewed & tested by the community» Fixed

Great, committed.
Hopefully that's all fixed now.

Status:Fixed» Needs review
StatusFileSize
new748 bytes

No, not yet. :-/

See #1111852-105: Add a "More like this" feature – the „More like this“ config form has still some wrong usage. Patch attached.

Neither API change nor release blocker now, though.

Fixed the problem for me. Cheers.

Status:Needs review» Fixed

Great, committed.
Thanks for testing!

Is this in any available releases such as stable or dev - or not yet?

It's in RC 1 and dev.

Status:Fixed» Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.