Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.Problem/Motivation
Pagination / Number of items broken for MLT block
introduced a bug in our feeds (they don't have a pager and are now limited to 10 items)
In SearchApiQuery 'limit' is set to NULL when there's no pager:
// Views passes sometimes NULL and sometimes the integer 0 for “All” in a
// pager. If set to 0 items, a string “0" is passed. Therefore, we unset
// the limit if an empty value OTHER than a string “0” was passed.
if (!$this->limit && $this->limit !== '0') {
$this->limit = NULL;
}
Then in SearchApiSolrBackend.php rows is set to 10 if limit is NULL:
// In previous versions we set a high value for rows if no limit was set
// in the options. The intention was to retrieve "all" results instead of
// falling back to Solr's default of 10. But for Solr Cloud it turned out
// that independent from the real number of documents, Solr seems to
// allocate rows*shards memory for sorting the distributed result. That
// could lead to out of memory exceptions.
$solarium_query->setRows($query->getOption('limit', 10) ?? 10);
Proposed resolution
We should set a higher limit (e.g. 1000) but probably not too high to run into memory issues?
| Comment | File | Size | Author |
|---|---|---|---|
| #8 | 3068714_8.patch | 4.41 KB | mkalkbrenner |
Comments
Comment #2
mpp CreditAttribution: mpp at AmeXio for District09 commentedSee https://github.com/mkalkbrenner/search_api_solr/pull/43
Comment #3
mpp CreditAttribution: mpp at AmeXio for District09 commentedComment #4
mkalkbrennerI see your point. But I‘m not yet sure about the best solution.
If you use Solr Cloud and run ten shards, the queried node reserves the memory for 10x1000 results for sorting.
So what is the limit that makes most sense?
That’s why I decided to set the Solr default limit when mo limit is set. Solr itself would react like this if we don’t pass a limit at all.
Comment #5
mpp CreditAttribution: mpp at AmeXio for District09 commentedFrom a solr perspective it makes sense to set it to 10.
The problem is that we also set it to 10 when the "Display all items" pager is configured in views so we break views behaviour.
Imo we have two options:
A) Either we need to set a "high enough" limit (guess, 100 or 1000). It is up to people using solr cloud to fix memory performance issues by forcing them to set a limit (in code or views). I guess this could be an advanced option.
B) or we extend/swap the default views pager (Drupal\views\Plugin\views\pager\None) with a configurable default limit of 10 so it is clear from the UI that a limit applies.
Imo leaving the default "Display all items" pager is not an option.
Comment #6
mpp CreditAttribution: mpp at AmeXio for District09 commentedComment #7
mkalkbrennerI think we should introduce a config option per server to set the default limit.
I prefer an option per server rather than per index because an index could be moved between servers and it’s the server connector which technically influences the max limit.
Comment #8
mkalkbrennerComment #9
mpp CreditAttribution: mpp at AmeXio for District09 commentedLooks good, thanks!
limit, max_rows, default_limit, I was in doubt which oké to use.
limit is correct from drupal/views perspective but we're actually setting the maximum rows in the solr query.
I'll let you decide.
Comment #10
mkalkbrennerYou're right. 'rows' is the better name.
Comment #12
mkalkbrennerComment #14
potassiumchloride CreditAttribution: potassiumchloride commentedFor other users who are finding this thread because of a search like "Drupal views display shows ten results instead of all items when the pager is set to 'Display All Items' " -
Here are steps to make the change to your Solr server configuration:
Comment #15
guaneagler CreditAttribution: guaneagler at CI&T commented@potassiumchloride
Thanks for your information. It helps me a lot.
Comment #16
ressa CreditAttribution: ressa at Ardea commentedI searched for site:site:drupal.org "Display all items" solr 10, found this issue, and can confirm that increasing "Advanced" > "Default result rows" to 100 in the Solr server settings fixes that only 10 nodes were returned from Solr.