Improve performance of log parser

Yeah, we definitely need to improve performance.

IMHO the real(more impact) solution for all the exec problem should be done with #973890: Create a reposync plugin using libgit2 php binding.

About the shell_exec calls, one static cache for that which shell invoke is definitely needed, patch attached for it(probably 1 instead of 7000 calls).

I need to review in detail other points to actually make an informed opinion :-p

Patch at #2 committed.

Assigning.

Last night I did a bunch of experimenting with replacing exec() calls to the git binary with calls to the Glip library. The only one I was able to get a significant improvement from was replacing all of the git ls-tree commands. Since it's the most used git command during log parsing that's a good thing.

This patch is paired down to the smallest set of changes and the way it's done it makes project/git_deploy a dependency (for Glip). This can be made even more efficient by only loading the repository once but I haven't done that here since that requires changing logic or adding a new global.

You can try it out and see what it does to your times.

What I'd really like to do in the future is go over the database tables and understand the way all of the data fits together and why we collect each part the way we do. Maybe there is a more efficient way to grab it.

I've also included the caching change sdboyer recommended.

Leaving this open as it may become enough or may need work depending on the out come of performance testing.

Taking myself off this for now. +1 for replacing exec calls with some sort of php lib like Glip. If dependency is an issue, an uglier alternative to Glip integration would be to try to combine the work to be exec'd into fewer OS calls (by building some sort of temporary script and parsing the result). There is a lot of overhead to exec calls (loading & unloading the shell environment).

In performance testing I was using Diaspora which exhausted the available memory on my MacBook.

I've reduced the number of exec() calls from thousands down to about 3 for initial import. Here are some times for fresh imports on my dev station. Down to ~15 seconds from about 110 minute for core.

I think it's pretty close and I should be able to grab what we are missing as it's identified.

halstead@avarice:/var/www/git-dev-skinny/drupal/sites/all/modules/versioncontrol_git$ time drush vcapi-parse-logs zen
Beginning synchronization of repository zen                                                                                               [ok]
Successfully synchronized repository zen                                                                                                  [success]

real	0m0.740s
user	0m0.460s
sys	0m0.160s
halstead@avarice:/var/www/git-dev-skinny/drupal/sites/all/modules/versioncontrol_git$ time drush vcapi-parse-logs views
Beginning synchronization of repository views                                                                                             [ok]
Successfully synchronized repository views                                                                                                [success]

real	0m3.352s
user	0m1.970s
sys	0m0.170s
halstead@avarice:/var/www/git-dev-skinny/drupal/sites/all/modules/versioncontrol_git$ time drush vcapi-parse-logs drupal
Beginning synchronization of repository drupal                                                                                            [ok]
Successfully synchronized repository drupal                                                                                               [success]

real	0m13.866s
user	0m8.050s
sys	0m0.170s

The highest memory usage while processing Drupal core was 157MB of resident memory and about 10 MB on disk.

Still to do,
Not exactly pretty but at least it's quick! (I wrote it with no sleep.)
Still missing types (FILE/DIR).
Needs to be tested against both tight and loose packs. (Should work.)
Need 40 zeros for empty parents.
Currently decides on tag/branch in a silly wrong way. :S Should be easy to make work correctly.

I still do not try the patch, but using git-fast-export seems like the obvious way to do imports, that I completely failed to notice :-p, naturally it reduce the number of exec's so, hoping to take a look at this soon.

Thanks halstead!

We now use DBTNG for all of our INSERTS. Yay.
Line numbers are included. Branches/tags are better. File operations are better. Probably took care of some more.

Still to do, process annotated tags. (These come in as tagger: in the export and we can use the same name for both author and committer.) Annotated tag data should be tracked in versioncontrol_operations
using these constants.
define('VERSIONCONTROL_OPERATION_COMMIT', 1);
define('VERSIONCONTROL_OPERATION_BRANCH', 2);
define('VERSIONCONTROL_OPERATION_TAG', 3);

I still need to do auto-increment checking to make sure the assumed values are what actually went in.

At this point processing Drupal core now uses 534MB of resident memory and takes about this long:

real  0m47.837s
user	0m39.170s
sys	0m0.670s

Views repository uses 148MB of memory and takes:

real	0m7.657s
user	0m5.930s
sys	0m0.180s

Don't load the whole export file into a single array. Replaced odd dept concept with a more standard counter. Replaced if/continue with an easier to read if/else structure.

Minor fix. Avoid strict warning. No need to include the auto_inc column.

Added error checking to make sure we are getting errors due to concurrency.
Watchdog $type cannot be longer than 16 characters so I've shortened it.

Not working but we have all of the data we need for {versioncontrol_operation_labels} in
$log->name_block->names[$log->commit] on line 194.

Core now takes 58 seconds.
Changed watchdog to a better name.
Call git rev-list once per branch to build our vc_op_label mapping.
Added variables for autoincrement settings.

The db_last_insert_id() function was returning zeros so giving false errors. Backing out that change.

Now with automatic error recovery.

Just the error recovery attempt by itself.

Change to handle empty repositories.

I don't know if it will make any difference but maybe start the transactions manually.

Okay everything since #26 plus changed drupal_set_message to watchdog().

This patch rolled against 6.x-2.x fixes the off by one error we were seeing with the auto increment settings on git-dev.

So a work in progress but...
Removed DBTNG from queries involving auto increment and added the table locks back in. With these changes memory usage is cut in half and there are no issues with multiple workers. On my dual-core desktop machine this code imports all ~8900 repos in 8 minutes 4 seconds with 0 errors.

Just wanted to get this out there, db_last_insert_id definitely won't work with DBTNG since it's using an entirely separate DB driver. Apparently DBTNG does not support the return value for the last insert with multiple value inserts, only single value inserts.