Problem/Motivation

Our current AMIs contain a local copy of the Drupal core repo.

We could use this repo for checkouts instead of cloning the repo across the network.

Proposed resolution

  • Experimentally add this behavior to the checkout plugin.
  • Determine whether there is a benefit.

Remaining tasks

User interface changes

API changes

Data model changes

Original report

Checking somebody's comment, I took a look at the logs of the fastest testbots cloning contrib projects for contrib tests, and found that some clones were in fact taking 2-3 minutes, when they should take 5-10 seconds. I was baffled because a manual clone didn't take that long. However, when I added --reference /var/cache/git/reference, the checkout took twice as long for views. However, it took 3 minutes and 45 seconds for commerce.

It's possible that this is a regression in the (old) version of git we're pinning to because PIFR 6.x-2.x doesn't have the ability to provide a valid patch for current versions of git. I'll check that...

root@www:/tmp# time git clone git://git.drupal.org/project/views.git
Cloning into views...
remote: Counting objects: 52456, done.
remote: Compressing objects: 100% (12131/12131), done.
remote: Total 52456 (delta 38684), reused 52133 (delta 38472)
Receiving objects: 100% (52456/52456), 15.09 MiB | 875 KiB/s, done.
Resolving deltas: 100% (38684/38684), done.

real	0m21.716s
user	0m1.416s
sys	0m0.096s
root@www:/tmp# rm -rf views
root@www:/tmp# time git clone --reference /var/cache/git/reference git://git.drupal.org/project/views.git
Cloning into views...
remote: Counting objects: 50329, done.
remote: Compressing objects: 100% (11245/11245), done.
remote: Total 50329 (delta 37612), reused 49738 (delta 37374)
Receiving objects: 100% (50329/50329), 14.89 MiB | 438 KiB/s, done.
Resolving deltas: 100% (37612/37612), done.

real	0m46.655s
user	0m3.692s
sys	0m0.208s

Comments

rfay’s picture

Issue summary: View changes
rfay’s picture

Issue summary: View changes
rfay’s picture

Changing the git version to 1.7.10.4 does not seem to affect this dynamic.

I suspect we should consider losing --reference altogether, or changing PIFR to populate the reference cache and then clone, which has significant disk space impact.

The tradeoff here is that core testing (drupal core only) currently clones in a couple of seconds, as opposed to 30 seconds without the reference dir.

isntall’s picture

Project: Drupal.org Testbots » DrupalCI: Dispatcher (Modernizing Testbot Initiative)
jthorson’s picture

Title: Reference repo slows down contrib clone for testbots » Determine whether we should use a reference repo

We don't currently have a reference repository in DrupalCI.

We should investigate whether we want/need one.

Mixologic’s picture

Project: DrupalCI: Dispatcher (Modernizing Testbot Initiative) » DrupalCI: Test Runner
Component: Code » Codebase Build
Mixologic’s picture

Project: DrupalCI: Test Runner » DrupalCI: Dispatcher (Modernizing Testbot Initiative)
Component: Codebase Build » Code

If there's a reference repo, its going to be baked into the AMI's. Moving to appropriate queue.

Mixologic’s picture

Project: DrupalCI: Dispatcher (Modernizing Testbot Initiative) » DrupalCI: Test Runner
Component: Code » Testrunner Codebase
Status: Active » Needs work

There is a copy of drupal core baked into the AMI's that could be used as a reference repo (provided we did a git pull first).

We should explore this in the checkout plugin to see if it saves much.

Mile23’s picture

Issue summary: View changes