Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.Running Drush on large sites can feel pretty slow (2-10 seconds in many cases). Clearly much of this is due to the time PHP takes to parse all the code. I would love to see if there are ways this can be improved.
PHP running on the web we can run APC or similar opcode caching to improve performance with dramatic results, however on the CLI you can't easily take advantage of that. You can set apc.enable_cli=1, but the opcode cache is dropped at the end of the CLI request, so that isn't really of direct use (except for testing APC key-value caching).
I did some investigating, however and found that recent versions of APC do in fact have a function to allow you to stash the opcode cache in a file and retrieve it later (this is intended for warming the caches of high performance web servers before coming on line). In principle that should work for Drush - see patch - but at least in my tests it always seemed to segfault PHP. Bummer.
The other idea I experimented with was running Drush as a daemon, and having a simple "client" app send commands and process user I/O. This is a little like the backend functionality, except that it runs commands in the same process, so you lose the isolation (of course many commands don't need the isolation) but you gain a huge speed advantage - subsequent commands took a fraction of a second because all the code is still in RAM. This approach worked OK for non-interactive commands, but for interactive stuff we would need to write/reuse some kind of protocol for handling the interaction, which may be quite a lot of work/complexity. The other problem is that if you run this as a system daemon you end up with all sorts of sketchy permissions issues, so you would probably need one daemon per website/user. Attached files are a very hacked up sketch of this.
I also thought a little about using the fcgid or native PHP cgi process management for the "daemon", but (f)CGI is not really suited to interactive stuff, so we end up with the same problem.
Does anyone have any thoughts on the above approaches, or other ideas?
| Comment | File | Size | Author |
|---|---|---|---|
| #20 | apc.patch | 1.1 KB | Owen Barton |
| #11 | drush-use_apc_cache-1051748-11.patch | 1.66 KB | pillarsdotnet |
| #5 | apc-segfault-php.txt | 1.13 KB | Owen Barton |
| #3 | drush_apc_testing.patch | 2.27 KB | sdboyer |
| drushd-client.php_.txt | 408 bytes | Owen Barton |
Comments
Comment #1
moshe weitzman CreditAttribution: moshe weitzman commentedNot that familiar with it, but q0rbon came up with https://github.com/q0rban/drushd for a recent project. There are a couple more daemon projects at http://drupal.org/project/modules?filters=tid%3A4654&solrsort=sis_projec...
Comment #2
Owen Barton CreditAttribution: Owen Barton commentedYes - I had thread forking working in my code also - it was pretty simple and worked fine, I took it out for testing though. The hard parts seem to be code reload/invalidation (perhaps have the fork live start transparently in the background and time out after a while), permissions (if systemwide), and most of all handling interactive user control (yes/no, select # etc). While I think it cold work well as a high performance cron type daemon or backend request handler alternative the drushd approach will get complex if you want to have it act exactly as it does on the shell (interaction, strout/err, colors, cwd response, table widths etc etc)
I think the APC approach is neater and simpler for regular execution. I was actually thinking that it would be best if APC implemented what I described internally - basically fixed the dump/load functions so they didn't segfault, and added a setting that saved the opcode at the end of each cli request in ~/.php.apc.cache (or whatever - it would have to be per user, for security though) and transparently loaded it for subsequent requests. Anyone know any APC developers ;)
Comment #3
sdboyer CreditAttribution: sdboyer commentedI did some more tests on this. Turns out there are some fairly specific problems in APC, ones that'll hopefully be fairly easy to resolve. Assuming the bug report I filed is adequate.
I made some small additions to grugnog's original script, and noted that the segfaults only occur when we attempting to load dumps containing, specific files (specifically commands/core/cache.drush.inc and includes/table.inc). Have fun with the array_slice() I left in the patch, you can see it for yourself; I did my tests against revision 7c582a0 of drush. There are other errors that occur when attempting to dump a lot of the other files which do appear when you get the segfault-causers out of the way, but it seems like the files still dump in those cases.
Comment #4
sdboyer CreditAttribution: sdboyer commentedPECL bug is at http://pecl.php.net/bugs/bug.php?id=22679
Comment #5
Owen Barton CreditAttribution: Owen Barton commentedNice detective work - I can confirm that I have same two problem files!
I dug a bit further and I think the problem when an undefined constant is used as a function argument default - for example in core.cache.inc if I change:
to
- it works even if that file is not excluded from the cache.
I found a different issue in table.inc - php segfaults whenever a constant (even a defined one) is used inside a class.
I have produced a reduced testcase for both these issues - I will attach this to the PECL issue. Looks like gopalv responded on PECL and fingers crossed may take a look next week.
Comment #6
moshe weitzman CreditAttribution: moshe weitzman commentedWe've added the commandfile cache which avoids a lot of dir scanning but all the converting to opcode is still present ... Could anyone prod that PECL person?
Comment #7
Owen Barton CreditAttribution: Owen Barton commentedCouldn't find Gopal in IRC (nick may be t3rmin4t0r it seems), so I e-mailed him just now. I'll update here if we hear anything back.
Comment #8
Owen Barton CreditAttribution: Owen Barton commentedI heard back from Gopal on IRC (he is no longer working on APC for his day job) and I e-mailed him the details. He also said "Will probably fix stuff in svn, it'll take me another month or so to land back home (Aug 12th), so don't expect a release right away.", so I am leaving this as postponed for now.
Comment #9
sdboyer CreditAttribution: sdboyer commentedWell hey, that's something, at least...
Comment #10
moshe weitzman CreditAttribution: moshe weitzman commentedComment #11
pillarsdotnet CreditAttribution: pillarsdotnet commentedLooks like this is fixed in trunk builds, at least.
Unfortunately, after applying the attached patch, glibc still barfs:
Comment #12
Owen Barton CreditAttribution: Owen Barton commented@pillarsdotnet - did you have apc.enable_cli=1 and apc.stat=0 in your php.inc for cli? Otherwise it might look like it works, but this is cause the cache dump is not dumping anything (it only dumps non-stat'ed files). If this is the case, then you might try chopping out files/functions to determine what the new cause is (as we did before).
Comment #13
pillarsdotnet CreditAttribution: pillarsdotnet commentedYes. Without
apc.enable_cli=1the code doesn't run (see my patch), and withoutapc.stat=0I get very different error messages.Not today; maybe sometime later.
Comment #14
Owen Barton CreditAttribution: Owen Barton commentedGot a nice e-mail from Gopal, the APC developer we have been e-mailing with, and he found some time to work on this during OSCON and committed the fixes to svn - Gopal++ :)
I just tested it out very briefly, using APC svn head, and it appears to work for me. The most recent patch at #11 did not work for me (empty cache file - I think it is taking array() literally, perhaps?) but the patch in #3 did. some very quick tests showed that this has a pretty reasonable effect - cutting execution time of "drush status" from about 1 second down to ~750ms - so roughly 25%, which is not an order of magnitude or anything, but I think still quite worthwhile. It would be interesting to hear from some others what effect this has on other systems.
Comment #15
sdboyer CreditAttribution: sdboyer commentedAWESOME.
Did you look at memory usage at all? I'm suspecting usage would be the same for APC vs. non-APC, but I can hope...
I think a 25% boost on something like
drush statusis nothing to cry about - at least assuming you were running that outside of a bootstrappable drupal instance. The real benefits here pay off when drush is running on a big site with a lot of code to load, after all.Comment #16
geerlingguy CreditAttribution: geerlingguy commentedI'm running drush commands via cron every minute or so, and have been noticing memory and CPU usage starting to increase as our server's load and bootstrap sizes have increased. Any update on the status of this? I'm running APC 3.1.9 on dev and production—would that be a good version to do some testing on?
Comment #17
sunTagging.
Comment #18
moshe weitzman CreditAttribution: moshe weitzman commentedD8 is a bloated pig and really requires APC to load all those files. What are the next steps here?
Comment #19
geerlingguy CreditAttribution: geerlingguy commented@moshe - looks like the patch in #3 needs a re-roll, and will work with APC 3.1.9 or later. If I get a chance, I'll try to test on one of my servers too.
Comment #20
Owen Barton CreditAttribution: Owen Barton commentedHere is the patch.
The bad news is that it appears PHP has regressed again - I am using APC 3.1.14 (the latest) with php 5.4 and get a similar "Exceeded bounds check in apc_bd_alloc_ex" message, both with the attached patch and the simplified test case. I have confirmed the APC fix for the previous issue http://news.php.net/php.pecl.cvs/16099 is still present in this version of PHP, so this must be a different underlying issue. My guess (looking at the changelogs) is that some change in 3.1.12 may have been the culprit, as this seems to have involved quite a bit of work on the dump/load functions.
Also, as I mentioned above - when I tested this, the improvement seemed very incremental - 25% at most. I think we should research other options as well.
- Zend OPcache (previously ZendOptimizerPlus) https://github.com/zendtech/ZendOptimizerPlus - this has been accepted into PHP 5.5 core and seems likely to become the most popular opcode cache. I was not able to identify if it has similar dump/load functions (I don't think so). This fork looks pretty promising however - https://github.com/TerryE/opcache/wiki/MLC-OPcache-details
- We can also investigate the "daemonized drush" approach discussed in the description. It occured to me that if we used the backend code to send requests and parse responses, it seems like this may be more achievable. It would still require a fair amount of work, but I think would be interesting.
Comment #21
greg.1.anderson CreditAttribution: greg.1.anderson commentedThis issue was marked
closed (won't fix)because Drush has moved to Github.If desired, you may copy this task to our Github project and then post a link here to the new issue. Please also change the status of this issue to
closed (duplicate).Please ask support questions on Drupal Answers.