I had a chance tonight to do some xdebug profiling of the generate-content drush command that ships with devel (drupal7). See http://drupal.org/node/659980 for the technique used. I created 100 page nodes with these results (will look much better on the web versus email):

Showing the 20 most costly calls sorted by 'memory-own'.

                                               Inclusive        Own
function                               #calls  time     memory  time     memory
-------------------------------------------------------------------------------
drupal_load                                38  0.2045 10930424  0.1774  9292976
drush_get_context                       13568  0.6879  4471928  0.3943  4471928
_drupal_bootstrap_full                      1  0.2864 15767416  0.0505  4352608
_drush_get_option                       12836  0.8102  3005792  0.4696  3005792
PDOStatement->execute                    1130  0.4732  2908040  0.4732  2908040
DatabaseCondition->compile               1324  0.2457  3849592  0.1327  2749264
_drush_find_commandfiles                    3  0.3846  2201712  0.0258  1981488
drupal_bootstrap                            4  0.3682 19872320  0.0432  1692504
unserialize                               114  0.0057  1640768  0.0057  1640768
drush_context_names                      1072  0.0200  1528040  0.0200  1528040
include_once                               33  0.0251  1626472  0.0223  1523720
drush_get_commands                          7  1.9546   918280  0.0954  1463656
PDO->prepare                             1130  0.0817  1596928  0.0318  1443248
array_merge                               722  0.0141  1337736  0.0141  1337736
DatabaseCondition->condition             1107  0.0417  1146328  0.0324  1146328
drush_drupal_version                      123  0.0126  1070328  0.0115  1064312
_registry_check_code                       13  0.0183  1393816  0.0108   986024
func_get_args                            1017  0.0148   935464  0.0148   935464
DatabaseConnection->defaultOptions       1130  0.0224   894992  0.0224   894992
InsertQuery_mysql->__toString             500  0.1415  2545992  0.0732   837008

Can we do something to drush_get_context(), drush_get_option(), and _drush_find_commandfiles() to slim down RAM usage?

Comments

moshe weitzman’s picture

moshe weitzman commented
Component: Code » PM (dl, en, up ...)

This has bit me again, this time during a migrate module import which is a long running, memory consuming operation. All I can think of is for us to change how we store the drush_log() items so they are grouped by $type. Then we add a helper function which migrate can call to flush all notices. Don't think will change the API at all. I don't see any core code that relies on an unkeyed array in the log.

apotek’s picture

apotek commented

The static map for a long-running process for many items will continue to use up memory with no limit. An alternative would be log this to an actual file, and then, if the entire log is requested, to read it from that file.

Instead of this:

function _drush_set_log($entry = NULL) {
  static $log = array();
  if ($entry == NULL) {
    return $log;
  }
  else {
    $log[] = $entry;
    return _drush_print_log($entry);
  }
}

It would be:

function _drush_set_log($entry = NULL) {
  static $logfh = NULL;
  if (is_null($logfh)) {
    // not sure how to find the tmp folder or the pid.
    $logfh = fopen(OS_TMP_FILE . OS_PID . '.log', 'a');
  }
  if ($entry == NULL) {
    return $log;
  }
  else {
    fwrite($logfh, serialize($entry . OS_NEWLINE);
    return _drush_print_log($entry);
  }
}
moshe weitzman’s picture

moshe weitzman commented
Status: Active » Postponed (maintainer needs more info)

I don't see this as a significant problem. Are you doing a lot of logging?

apotek’s picture

apotek commented

It's not a problem under most cases, but if you are processing millions of rows (such as in a migration), static variables that have no upper limit mean that memory consumption has no upper limit.

greg.1.anderson’s picture

greg.1.anderson commented
Version: » 8.x-6.x-dev
Status: Postponed (maintainer needs more info) » Closed (won't fix)
Issue tags: +Needs migration

This issue was marked closed (won't fix) because Drush has moved to Github.

If desired, you may copy this task to our Github project and then post a link here to the new issue. Please also change the status of this issue to closed (duplicate).

Please ask support questions on Drupal Answers.