Regression: the pager no longer lets me jump to pages, nor shows total number of pages

Sorry, the count() queries were too much load on the DB. We should look for better long-term fixes, but this is quick and stabilizes the site.

no, we currently do not have that.

Not having a pager is preferable to not having a website, so we went for that.

Right, the count queries (which would be required to accurately give you those sorts of links) were crashing our DB server. Until we can serve the issue queue views via something other than SQL, we have to render these views like so. The logic that knows if we need a next/previous link (which works) is triggered by simply trying to request N+1 records if the page is configured to display N, so we don't have to count every possible result to accurately add a next link.

Also, most of the problems that are causing you to want/need to go to specific pages are already resolved or being fixed in the near future.

Finally, if you absolutely must go to a specific page for some reason, you can always add &page=N to your URL. I know it sucks, but it wouldn't be the first time that getting around parts of d.o was most easily accomplished by crafting the URLs yourself. :/

Just filed #2135511: Issue queue pager no longer shows total # of pages :( as a dupe. :( Added details to the title. This pager change has busted functionality I literally use every single day.

I definitely support the site working at all over not working and having this feature, but I'd love to at least leave open the possibility of adding this back once other performance enhancements are made (e.g. entitycache module or whatever), so moving back to active tentatively.

You can also use the amount of comments as metric here.

Would it be practical to add the page numbers (and/or total number of results) via an AJAX call only when they were specifically requested?

I need multiple pager links too. At least bring back the "first" link, please? And the current links are too subtle. It took me a while to even see the go forward/back links.

For those of us with a zilliion issues in our personal issue queue... we relaly need this. Echoing webchick. I use this every day and the new pager is totally worthless.

Here's a short video showing a before/after in D6 vs. D7 to emphasize the loss in workflow here.

https://dl.dropboxusercontent.com/u/10160/D.o%20UX/DoUX-queue-totals.mov

Before: ~5 seconds.
After: ~8 seconds + 1 server round-trip per page. (I lucked out that the queue I was checking in this case only required 2 of these, so only represented a ~250% regression in time taken to complete the task. It could've been much worse.)

We're working to bring the pager back.

Another bug, not sure if it's caused by this pager or the issue queue views themselves, but in addition to not showing my total # of pages, I can't actually even see what all of the issues themselves are, because when I page to the next one, it might re-show me the same issues again. :( It seems like it sorts in random order if they're of the same e.g priority. :(

Found this on https://drupal.org/project/issues/search/drupal?version%5B0%5D=8.x&issue... when trying to figure out what the "normal" issues were in that list. It showed me the same two issues on both pages.

As a stop-gap, could we permission-base the pager perhaps? Anonymous users see the performant pager, people with git permissions get the old one back so we can work?

The random sorting is either due to Search API, or more likely just the general problem of sorting by a value that's equal, not the pager. That's an entirely separate problem from this issue. I'm not even sure that's a bug. AFAIK, there's no mechanism for defining a secondary SORT via Views. Sounds like you really want to filter by priority then sort by something else.

Meanwhile, there's no custom magic here. They're just standard pager settings on Views. So I'm not sure if it's possible (without lots of custom magic) to get the conditional pager behavior you're describing.

Ultimately, nnewton needs to agree to any changes to the pager. We only deployed this to help keep our DB server alive. I know this isn't ideal, and It Would Be Better(tm) to have the full pager again, but having d.o up at all is Better Still(tm).

Sorry,
-Derek

Views definitely supports secondary sorting, in general.

But not via exposed sorts, right?

Anyway, we're totally off-topic here. This issue is about the pager. The sorting problem is separate. If people care about that, let's continue in a new issue to keep this one focused on the pager itself.

Thanks,
-Derek

From what I understand, the real solution here is a different data backend for the issue listing. However, in the short term a solution that might work is to remove the text search and some of the filtering/sorting options from the non-advanced search. This would allow us to control the query this view is creating and then the full pager might not be so painful. The issue in general is very simple, via this view you can currently create a query that _only_ runs via the LIMIT. The full version of it could never complete. The COUNT(*) obviously causes issues with a query like this.

This is actually not a new issue and existed on the D6 site to the point that we also introduced the lite pager there several times during high-traffic periods. However, it is now worse and it is no longer practical to run with the full pager at all on this version of the view.

Spun off #2148433: Define an explicit secondary sort for issue views when table sort is active.

Removing the search/filters from non-advanced views is not a solution, IMO. It just means more people will hit the advanced views to do their queries, which will probably only further create performance issues. :(

Do we need to spiff up and fix https://drupal.org/project/pbs or perhaps https://drupal.org/project/mv in order to restore pager functionality? It feels like we're surely hitting issues that other big D7 sites must've hit.

"Removing the search/filters from non-advanced views is not a solution, IMO. It just means more people will hit the advanced views to do their queries, which will probably only further create performance issues. :("

Right, but that one would have the limited pager.

"Do we need to spiff up and fix https://drupal.org/project/pbs or perhaps https://drupal.org/project/mv in order to restore pager functionality?"

The searchapi tables are already denormalized, so no. More optimization around these tables maybe possible, but its A. pointless as long as there is a text search box there and B. pointless if we are just going to switch to Solr.

"It feels like we're surely hitting issues that other big D7 sites must've hit."

Depends on what you mean. These pager issues are certainly very widespread and have been for a long time. If you mean the general use case of a very active issue listing with fairly flexible sorting/searching...most D7 sites would probably not back that fully in MySQL or would write something very custom. The plan with searchapi was to not have it fully backed in MySQL, however, that portion is not done yet. (is my understanding, I was not involved enough to really know)

pbs was a proof of concept from a sprint 4 years ago that was never developed into something that actually works. And even if it did, that still couldn't help with text search at all. Similarly, mv was basically a custom hack for d.o that was subsequently abandoned, and ultimately does the same thing -- tries to put "everything" in a single table. That's why we're using Search API. It's a well-maintained, actively developed version of what both of those modules claimed/promised to do. Remember, we're trying to use more community-supported solutions, instead of things that no one uses but d.o which the DA is on the hook to maintain.

The problem is that for now, we're using the SQL backend for Search API (path of least resistance to get the site live, closest to the D6 implementation, no time/resources to explore alternatives), and there are limitations with search_api_db. Text search is basically never going to work on a site the size of d.o served via DB tables. There are just too many issues on d.o (706586 out of 1153518 total nodes (61% of all content on d.o) at the time of this comment). search_api_db is JOINing on separate search index tables (the only thing it can possibly do) and that's just unworkable at this scale.

Ultimately, the only way to make this work at all is to use a different backend that's designed for this sort of problem, like sphinx or solr. We looked at solr while working on the upgrade, but decided that the latency of updating the solr index after you touch an issue was too great and there'd be an avalanche of critical bugs about how "I updated issue #X to set it to 'needs work' but it still shows up as 'needs review' in the issue queue." All of this discussion is available in #949372: Port issue views to Search API so we can have a performant backend if anyone's really interested. Thankfully, Search API is *exactly* designed to make these sorts of changes easier. Now that we're using search_api_db, it should be *much* easier to move to search_api_X, once we decide on which X we want to use and can setup the backend index appropriately.

Finally, the parent issue here, #1457256: Enable Lite Pager On Project Issues Advanced Search was originally from almost 2 years ago. It just never went live before the D7 launch due to people making "hot fixes" to the views on the live d.o site and me not being able to deploy the change nnewton wanted (#1478350: Manually-overridden views on d.o make deploying fixes impossible). So this basically has nothing to do with the D7 upgrade itself. This has been a big problem for a very long time. The kind of functionality people have come to expect is basically impossible to provide on a site this big just relying on SQL tables.

At this point, I think the only viable approach here is to call this postponed until #2136119: Text search on issue queues is slow and sometimes WSODs is resolved (which will probably be where we move to a new Search API backend). Until that's done, I don't believe we can allow the COUNT() queries back on d.o. As nnewton joked in IRC:

nnewton: or just put the whole thing back and I'll disable nagios ;)
...
nnewton: hey, I'm totally fine bringing it back. I just don't want to be on call with it back :)

"At this point, I think the only viable approach here is to call this postponed until #2136119: Text search on issue queues is bringing the site down is resolved (which will probably be where we move to a new Search API backend). Until that's done, I don't believe we can allow the COUNT() queries back on d.o."

Or somehow limit filtering options for the default view. Which I'm quite fond of anyway.

NOTE: In case it sounds like dww and I don't want this fixed....we definitely do. I believe we have just been trying to explain the depth of this issue and how long its been a problem. I've been gotten up countless times in the middle of the night because a bot was messing around with ridiculous pager values on the D6 site. It has been a pain point for awhile.

Thanks for the background info. That's very helpful.

Or somehow limit filtering options for the default view. Which I'm quite fond of anyway.

I'd love to gather some data of some kind (I guess Apache/ngnix logs?) to see what percentage of requests use the filters vs. don't to help us decide if that's a viable option. What I *suspect* will happen if those filters go away is the issue queue will be useless to > 80% of its users. I pretty much never use any issue queue view, apart from /project/issues/user, without setting some filters of some kind. Most often, though, it's not the full text search, but things like priority + version + tags. But unfortunately, it's exactly those custom filtered views upon which I (about 10-15 times per week) need to be able to see how many things are left in them. :( So switching to the current pager on filtered views doesn't actually solve my problem.

And while if full text search is the root issue, it's tempting to remove full text search from the main listing and shove it somewhere else, all that is likely to do is greatly increase the frequency with which duplicate issues are reported, which is its own rage-inducing thing. Then, worse than a 250% regression in productivity, it becomes a XXXX% regression in productivity (I haven't recorded before/after in D6 vs. D7 comparison yet to know precise numbers, but it's pretty severe) in order to close these issues as duplicates, thanks to the new issue queue workflow which hid this common behaviour behind a page refresh.

So it sounds like indeed the only thing we can do right now is suck it up and wait, unless there's a way for the community to help with Solr.

> And while if full text search is the root issue, it's tempting to remove full text search from the main listing and shove it somewhere else,

This is purely based on my own usage patterns, but my perception is that listing and searching for issues breaks down like this:

A. full overview of the issue queue
B. full overview of the issue queue, filtering by versions / components / status / tags
C. text search for duplicates
D. text search for 'where's that issue I saw 2 weeks ago about foobar'

It seems to me that the text search is less time-dependent. Certainly for case D, it doesn't need to be that fresh. Case C is maybe a problem, as when a problem first comes up, people need to be able to find duplicates.

And also, it seems to me that cases A and B are the ones where you want the pager. For text search, if the first page of results doesn't have what you want, you probably need to refine or rephrase your search.

So I can certainly see a case for splitting issue queue listings into listing with filters and a full text search, where the text search comes with a warning that it might not be entirely up to date.

To mitigate the problems with duplicate searching, is it possible to prioritize the indexing, so critical issues get indexed first?

Yes, that's my workflow, and I suspect most other project maintainers' workflows as well. I'm concerned though about the the other 99.97% of the community who are not project maintainers and are instead going into the issue queue to find/report bugs. The kind of people who post about their off-topic module problems on an issue about fixing the Drupal.org packaging system. :P~ I worry that hiding the search box from them just means more aggravation for project maintainers having to triage their mistaken duplicates.

However, +1 to the statement that full-text search does not need a fancy pager. It's the filter-based searches (components, versions, tags, etc.) that really need it.

Hm. What about this? Keep the current pager on the default views, keep the full-text search on the default views for the 99.97%. On advanced views (for the project maintainers), remove the full text search, and just use the regular pager. Would that work?

> I worry that hiding the search box from them just means more aggravation for project maintainers having to triage their mistaken duplicates.

True, but isn't the main pathway to the full text search the single text box on the project page? That can send people through to the unpaged text search, and they can refine and alter their text search terms there.

#25... I don't agree that the main pathway to full text search is the box on the project page necessarily. I personally always start with project/issues/[name] and search from there. But then again, I am not the average user. :)

Since the "pestilence" of duplicates came up more than a couple of times in this issue, I want to shamelessly promote #19386: Automatically search for duplicate issues/questions before submitting new issue/question now that we upgraded to D7 (notice how that issue has a 5-digit node number to get an idea of how back this problem goes).

I think it would be okay to do a temporary partial solution now: show the pager unless there is a text search. Text search still needs to wait on #2136119: Text search on issue queues is slow and sometimes WSODs. Count queries for the others are much more simple.

This does need testing for deployment. Searches on issue tags, participants, and followers (your issues), each have a table to JOIN on. It is possible those count queries are still too heavy for now.

That's great news!

It would be a huge step forward to have the pager just for:

- all issues for project foo
- project foo issues filtered by component or status

Yeah, I can live with that too. Better have a partial solution in place than none at all.

I needed to know how many 8.x in each state: active, needs work, needs review, rtbc and postponed issues there were. This used to be, select that status, jump to last page, count and multiply number of pages -1 + num on last page. Now I have to click through all the pages to get to the last. :( (or do some kind of binary search and edit the url https://drupal.org/project/issues/search/drupal?text=&assigned=&submitted=&participant=&status[0]=1&&&version[0]=8.x&&issue_tags_op=%3D&issue_tags=&page=1 &page=1 part, like guess there are 70 pages and edit the url, then guess maybe 30, then 45, then 53...)
Putting pager back on searches that are not text (this was not a text search) would let me do this faster like I used to.

I would be inclined to say -- though others should feel free to tell me I don't have the full picture -- that having the pager on component & status would satisfy the 80-20 rule and pretty much fix the pain this is currently causing.

As a contrib maintainer, I feel that having the pager on tag searches would be a 'nice to have' only. I don't know whether in core it's useful to know how many issues there are with a tag.

I don't think I've ever used the pager on text search, having it for just filtering via select list is all I need!

I don't know if this is related, but this was mentioned in misc other pager threads, some pages have schizophrenic pagers

https://drupal.org/project/usage - no pager
https://drupal.org/project/usage?page=2 - pager
https://drupal.org/project/usage?page=3 through ?page=6 - no pager
https://drupal.org/project/usage?page=7 - pager

PS: Is Drupal supporting the NSA fight back thingie or has the site be hacked?

Discussion on Joining The Day We Fight Back is at https://drupal.org/node/2188053. It is intentional.

Project usage pagers are not related to this issue.

http://drupalcode.org/project/drupalorg.git/commit/0c2487a restores pagers for issue queue listings, except for text searches, and has been deployed. For text searches, #2136119: Text search on issue queues is slow and sometimes WSODs needs to be fully resolved.

thanks!

We now have significantly faster database servers.

As mentioned in #2136119-126: Text search on issue queues is slow and sometimes WSODs, I think we can close this one out. There have been no complaints since the fix was committed back in Feb. to allow seeing the full pager on filtered listings, and that's the vast majority use case. Especially if allowing full pagers on text searches could introduce further performance issues.

Then we won't ~double the query time on issue text searches.