Make better use of the entity cache [#1898166]

The exclude function was doing an unnecessary static cache clear when the loaded entity was actually in the memory. I moved some functions around and I think this will actually improve the speed of the indexing.

Please test and let me know the results.

Comment	File	Size	Author
#4	Screenshot_1_25_13_11_56.png	99.07 KB	Nick_vh
#5	Screenshot_1_25_13_12_04.png	80.09 KB	Nick_vh
	node_load_issue.patch	1.74 KB	Nick_vh

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Comment #1

janusman CreditAttribution: janusman commented 24 January 2013 at 22:31

Looks good on first glance!

Comment #2

Nick_vh

he/him

Ghent

CreditAttribution: Nick_vh commented 24 January 2013 at 23:02

With 500 items of d.o, and drush and the patch

nickveenhof@Nick-Veenhofs-MacBook-Pro:~/Sites/dev/drupalorg_d7/site/sites/all/modules/apachesolr$ time drush solr-index
Inspected 500 of 50481 entities. Submitted 500 documents to Solr [ok]
500 items successfully processed. 500 documents successfully sent to Solr. [status]

real 1m29.258s
user 1m0.704s
sys 0m2.059s

Without the patch

Inspected 500 of 49981 entities. Submitted 500 documents to Solr [ok]
500 items successfully processed. 500 documents successfully sent to Solr. [status]

real 2m0.622s
user 1m24.346s
sys 0m3.041s

I'll do 10 of those tests to see if this is actually a real improvement

Comment #3

janusman CreditAttribution: janusman commented 24 January 2013 at 23:24

Looks like a pretty good improvement if the remaining tests come out similar.

Functionally-wise, we should make extra sure things are indeed being removed from the static cache, and if so this is a no-brainer. Great work!

Comment #4

Nick_vh

he/him

Ghent

CreditAttribution: Nick_vh commented 25 January 2013 at 11:05

File	Size
Screenshot_1_25_13_11_56.png	99.07 KB

Here's some tests I did. It seems a better use of the entity cache gives us an average speedup of 114%, median (not super significant here) 115%.

The tests have been done on a D7 port of Drupal.org so the nodes were not generated. The procedure was the following

time drush solr-delete-index && time drush solr-mark-all && time drush solr-index

This means that we always index from scratch and thus do the same test over and over again. Each test was done 5 times to get a more realistic view.

Average time it took to index 500 entities without the patch :
158.941 seconds, which is 3.16 entities per second.
Average time it took to index 500 entities with the patch :
139.185 seconds, which is 3.59 entities per second.

Definitely an improvement! (please ignore the attachment, it selected the wrong column)