This is a note to myself to write a patch to refactor apachesolr.index.inc so that there is only one db_query to get a batch of entities (without changing any functionality).

Once this has been committed, I will then be able re-roll the patch for https://drupal.org/node/1828014 so that it is easier to read (currently it deletes two near-identical queries and replaces them with it's own, slightly different, query).

Comments

ianthomas_uk’s picture

Status: Active » Needs review
StatusFileSize
new4.8 KB

Progress so far. I've not been able to test this myself yet, but I'm uploading it for the testbot to check.

I'm combining an "indexing query" (gets the list of entities to index) and a "remaining count query" (gets a count of the entities to index).

Changes to indexing query

The remaining count query included ->condition('aie.status', 1), which the indexing query didn't. I assume these entities were be rejected later in the indexing process due to being unpublished, although I've not actually verified that yet. If they are not, then that means we are indexing unpublished entities which would be a bug. This condition will now apply in both places where the combined query is used, and may result in slightly faster indexing when there are unpublished entities.

Changes to count query

We now make two calls to apachesolr_get_indexer_table() as we can't reuse the variable. This is a fast function, so should not be a concern.
We are also asking db_query() for more fields and to order the results, but as we are only doing a count for this query I imagine these are optimised out by the db layer. Even if not, this should not add significantly to the load time of this page.

ianthomas_uk’s picture

StatusFileSize
new1.26 KB

As far as I can tell this patch is correct. I've attached a slightly modified version that just adds an underscore to the start of the function name as it's not meant to be public.

pwolanin’s picture

Status: Needs review » Needs work

wrong patch?

ianthomas_uk’s picture

Status: Needs work » Needs review
StatusFileSize
new4.8 KB

Sorry, I was late for a call.

ianthomas_uk’s picture

As I discussed with nick_vh yesterday it would be great to get this reviewed and committed, as it should make https://drupal.org/node/1828014 much easier to review. The patch can still be applied to 7.x-1.x HEAD.

When reviewing, please consider the changes that I have mentioned in #1 and confirm that you agree these are either insignificant or for the better, and confirm that I haven't made any other changes in my patch.

nick_vh’s picture

@@ -501,32 +481,7 @@ function apachesolr_index_get_entities_to_index($env_id, $entity_type, $limit) {
-    ->orderBy('aie.changed', 'ASC')

These orderBy statements now appear in both queries. Since they have an index these don't have a massive performance impact. However, I'm doubting if we should add an extra index for changed only. I did an explain query on this and it seems like it is using the index properly so I'm 99% sure it's fine.

nick_vh’s picture

Status: Needs review » Fixed

Committed to 7.x-1.x and 8.x-1.x

ianthomas_uk’s picture

Status: Fixed » Reviewed & tested by the community

Thanks for the review.

I can see this on 8.x-1.x but not 7.x-1.x.

ianthomas_uk’s picture

Status: Reviewed & tested by the community » Needs work

I'm getting kicked out of the room now, but I've spotted that the function I added was missing the following line:

$bundles = apachesolr_get_index_bundles($env_id, $entity_type);

I'll try and put that into a patch as soon as I can.

nick_vh’s picture

oh indeed - please do :-)

ianthomas_uk’s picture

Status: Needs work » Needs review
StatusFileSize
new1007 bytes

Here's a patch. It won't apply at the moment as the earlier patch hasn't made it to git on drupal.org.

nick_vh’s picture

Version: 7.x-1.x-dev » 6.x-3.x-dev
Status: Needs review » Patch (to be ported)

Committed, thanks. Putting status back to needs backport

ianthomas_uk’s picture

Version: 6.x-3.x-dev » 7.x-1.0
Status: Patch (to be ported) » Needs work

You've committed the follow-up patch, but not the original patch from #4 (which will probably need to be committed first).

ianthomas_uk’s picture

Version: 7.x-1.0 » 7.x-1.x-dev
Assigned: ianthomas_uk » Unassigned
nick_vh’s picture

Hmm, let's see if this is better then. Redid some commenting but essence is the same

nick_vh’s picture

Status: Needs work » Needs review
nick_vh’s picture

Version: 7.x-1.x-dev » 6.x-3.x-dev
Status: Needs review » Patch (to be ported)

Finally all got committed, putting the backport tag on again

ianthomas_uk’s picture

Version: 6.x-3.x-dev » 7.x-1.x-dev
Status: Patch (to be ported) » Needs review

That will still need #11 to be applied. #15 will correctly remove the line from the previous commit of #11, as git has added it to the old location of the file.

nick_vh’s picture

Version: 7.x-1.x-dev » 6.x-3.x-dev
Status: Needs review » Patch (to be ported)

Committed both - This patch was quite a mess to get right it seems. The backport has to be a bit cleaner next time :-)

ianthomas_uk’s picture

Don't forget to apply #11 to 8.x

mkalkbrenner’s picture

Assigned: Unassigned » mkalkbrenner
Issue summary: View changes

This mess of patches doesn't simplify the backport :-(

Mark_’s picture

Status: Patch (to be ported) » Needs review
StatusFileSize
new5.33 KB

Here is a backport for D6.

Mark_’s picture

It looks like some of #1828014: Mass re-indexation can miss (a lot of) content crept into that last patch I posted.

3.x currently uses queries with a where clause like this:
WHERE aie.status = 1 AND ((aie.changed > %d) OR ((aie.changed <= %d) AND (aie.entity_id > %d)))

Whereas #1828014: Mass re-indexation can miss (a lot of) content changes the "<=" to just an "=":
WHERE aie.status = 1 AND ((aie.changed > %d) OR ((aie.changed = %d) AND (aie.entity_id > %d)))

So, I am submitting this patch that retains 3.x's "<=" operator, and once this patch is reviewed and accepted, I'll submit a patch on #1828014 that fixes the query.

mkalkbrenner’s picture

StatusFileSize
new5.13 KB

Your patch looks good. I just simplified it a little.

  • mkalkbrenner committed 3c4dde1 on 6.x-3.x
    Issue #2011492 by ianthomas_uk, Nick_vh, Mark_, mkalkbrenner: Remove...
mkalkbrenner’s picture

Status: Needs review » Fixed

Locally all tests passed. Committed to git.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.