Closed (fixed)
Project:
Apache Solr Search
Version:
6.x-3.x-dev
Component:
Code
Priority:
Normal
Category:
Feature request
Assigned:
Reporter:
Created:
4 Jun 2013 at 09:40 UTC
Updated:
31 Jul 2014 at 14:50 UTC
Jump to comment: Most recent, Most recent file
Comments
Comment #1
ianthomas_ukProgress so far. I've not been able to test this myself yet, but I'm uploading it for the testbot to check.
I'm combining an "indexing query" (gets the list of entities to index) and a "remaining count query" (gets a count of the entities to index).
Changes to indexing query
The remaining count query included
->condition('aie.status', 1), which the indexing query didn't. I assume these entities were be rejected later in the indexing process due to being unpublished, although I've not actually verified that yet. If they are not, then that means we are indexing unpublished entities which would be a bug. This condition will now apply in both places where the combined query is used, and may result in slightly faster indexing when there are unpublished entities.Changes to count query
We now make two calls to apachesolr_get_indexer_table() as we can't reuse the variable. This is a fast function, so should not be a concern.
We are also asking db_query() for more fields and to order the results, but as we are only doing a count for this query I imagine these are optimised out by the db layer. Even if not, this should not add significantly to the load time of this page.
Comment #2
ianthomas_ukAs far as I can tell this patch is correct. I've attached a slightly modified version that just adds an underscore to the start of the function name as it's not meant to be public.
Comment #3
pwolanin commentedwrong patch?
Comment #4
ianthomas_ukSorry, I was late for a call.
Comment #5
ianthomas_ukAs I discussed with nick_vh yesterday it would be great to get this reviewed and committed, as it should make https://drupal.org/node/1828014 much easier to review. The patch can still be applied to 7.x-1.x HEAD.
When reviewing, please consider the changes that I have mentioned in #1 and confirm that you agree these are either insignificant or for the better, and confirm that I haven't made any other changes in my patch.
Comment #6
nick_vhThese orderBy statements now appear in both queries. Since they have an index these don't have a massive performance impact. However, I'm doubting if we should add an extra index for changed only. I did an explain query on this and it seems like it is using the index properly so I'm 99% sure it's fine.
Comment #7
nick_vhCommitted to 7.x-1.x and 8.x-1.x
Comment #8
ianthomas_ukThanks for the review.
I can see this on 8.x-1.x but not 7.x-1.x.
Comment #9
ianthomas_ukI'm getting kicked out of the room now, but I've spotted that the function I added was missing the following line:
$bundles = apachesolr_get_index_bundles($env_id, $entity_type);I'll try and put that into a patch as soon as I can.
Comment #10
nick_vhoh indeed - please do :-)
Comment #11
ianthomas_ukHere's a patch. It won't apply at the moment as the earlier patch hasn't made it to git on drupal.org.
Comment #12
nick_vhCommitted, thanks. Putting status back to needs backport
Comment #13
ianthomas_ukYou've committed the follow-up patch, but not the original patch from #4 (which will probably need to be committed first).
Comment #14
ianthomas_ukComment #15
nick_vhHmm, let's see if this is better then. Redid some commenting but essence is the same
Comment #16
nick_vhComment #17
nick_vhFinally all got committed, putting the backport tag on again
Comment #18
ianthomas_ukThat will still need #11 to be applied. #15 will correctly remove the line from the previous commit of #11, as git has added it to the old location of the file.
Comment #19
nick_vhCommitted both - This patch was quite a mess to get right it seems. The backport has to be a bit cleaner next time :-)
Comment #20
ianthomas_ukDon't forget to apply #11 to 8.x
Comment #21
mkalkbrennerThis mess of patches doesn't simplify the backport :-(
Comment #22
Mark_ commentedHere is a backport for D6.
Comment #23
Mark_ commentedIt looks like some of #1828014: Mass re-indexation can miss (a lot of) content crept into that last patch I posted.
3.x currently uses queries with a where clause like this:
WHERE aie.status = 1 AND ((aie.changed > %d) OR ((aie.changed <= %d) AND (aie.entity_id > %d)))Whereas #1828014: Mass re-indexation can miss (a lot of) content changes the "<=" to just an "=":
WHERE aie.status = 1 AND ((aie.changed > %d) OR ((aie.changed = %d) AND (aie.entity_id > %d)))So, I am submitting this patch that retains 3.x's "<=" operator, and once this patch is reviewed and accepted, I'll submit a patch on #1828014 that fixes the query.
Comment #24
mkalkbrennerYour patch looks good. I just simplified it a little.
Comment #26
mkalkbrennerLocally all tests passed. Committed to git.