remove hardcoded drush pm extension cache

Yes, I need to fix the hardcoding of the extension cache, currently in /tmp/cache.inc; however, this is not really a temporary file, and therefore should not go in /tmp at all. A few changes are needed here. Caching the structure in a context would be a really good idea. Making the cache pluggable, so that it could be stored in a database would also be good. The eventual model of operation is that drush will check in with drush.ws or drupal.org when it needs to map from an extension name to the project that contains it.

So, in short, the changes to pm.drush.inc suggested here are not appropriate. A better fix (moving to a non-temp location) is needed.

Can anyone review the changes in drush.inc? I don't do Windows. :(

Related to this, and just to understand, what's the use of cache.inc?

Because right now, and because it's created on /tmp/cache.inc, if I use drush with an user and then want to use it with a different one, it cannot write to cache.inc because the owner was the other user, and although tmp is world writable, once a file is owned by someone he can only owned it? Maybe set it wordwritable as a quick workaround?

@hanoii

This is where persistent cache of extension -> project mapping first showed up (as git blame showed). Greg's commit is here.

cache.inc is used to save mappings from module name to project name. Right now, everything is just all mixed together, which is not right; the cache should be keyed off of the Drupal major version.

I think I am going to rewrite it as a hook; no hook == no cache. That should do in the short term.

that would mean no creating cache.inc? without following that, if drush is going to create a file that should be use server-wide, I guess it should have the proper permissions, just worried about this scenario in which drush is run by different users within the same server.

Up!

I don't have understood what is the purpose of /tmp/cache.inc, but if it's useful for every drupal installation on a server, it should be chmod 777, if not, it should be created as /tmp/cache-$USER.inc. Otherwise, drush dl stop working after using it one time, and i should manually delete /tmp/cache.inc.

Yes, this is terrible.

sub

The drush dl command seems to work even with these errors. Is there anything to worry about using drush dl before this problem is fixed.

Shai

No, this "cache" is only used to map from extension (module) back to the project it came from; in its current form it is not really useful. I'm just going to take it out until a better implementation is available.

I agree with removing this for now.

Rather than remove it, I relocated the cache to $HOME/.drush and fixed up the code a bit. drush now keeps a 'missing' list of extensions that were needed to resolve a dependency at pm-enable time, but that could not be found. If these extensions are later found (via drush pm-download missing-project), then they are removed from the 'missing' list. There are 'TODOs' in the place where an external service could be called to look up / cache extension:project mappings.

Or I could just remove this.

I tested this and it seems to work. We should commit this after a couple tweaks.

1. Add a comment at top of cache file stating that the file is safe to delete.
2. I'm a little worried about stale wntries in the cache file. Project contents change even during same drupal release. Extensions can show up and leave on a whim. I didn't look into how we would handle that. If Greg thinks we are OK here, then no problem.

I tested this and it seems to work. We should commit this after a couple tweaks.

1. Add a comment at top of cache file stating that the file is safe to delete.
2. I'm a little worried about stale wntries in the cache file. Project contents change even during same drupal release. Extensions can show up and leave on a whim. I didn't look into how we would handle that. If Greg thinks we are OK here, then no problem.

#14.1: Good idea; done.

#14.2: I considered making a case that everything is okay; if a new extension 'foo' shows up in project 'fooproj', then it will overwrite any old entry for extension 'foo' in, say 'foo-proj'. Since there are << millions of modules for Drupal, it's probably not too much of a big deal if things 'drift' a little bit.

However, it did vaguely bother me that if an extension 'foo' were renamed to 'fooplus', then 'foo' would remain in the extension cache forever. Again, probably not a huge problem, since it won't hurt anything, but still vaguely bothersome. The enclosed patch keeps another data structure in the extension cache that keeps track of how long an item has been in the cache. This value is updated any time the project containing the extension is downloaded. If an item gets to be > 4 weeks old, then it disappears from the cache.

Hmm. Maybe that 4 weeks expiry is not needed. If we direct the user to the wrong project, and he downloads it, then we will overwrite the stale cache. So we make only one bad recommendation. Isn't that right?

It depends on what changed.

Case 1: If the module name stayed the same, but the project name changed, then the cache will be overwritten when the user downloads the correct project. In this case, if we direct the user to the wrong project and he downloads it, he will either get a 'project not found' error, or he will get a project that no longer contains the module he is looking for. In either case, the wrong entry stays in the cache until the correct project is found.

Case 2: If the module name changes, but the project stays the same, then other projects that depend on that module will have incorrect dependency information. When the user goes to download the project we recommend, looking for a module that no longer exists, then the name of the new module names will be written to the cache, but the old module names will stay there. Users will keep looking for the wrong module until the dependencies for modules that use it are all corrected, regardless of what is in the cache.

I could make the code clear out all of the cache entries where the value == the project downloaded whenever items are added to the cache, but that would not solve either issue above. It would get the bad value out of the cache in case 2, but as mentioned, that has no direct affect on the user. The expiry is the only way to get the value out of the cache for case 1, and it will also get the value out of the cache eventually for case 2, so my opinion is that the existing code is close enough.

The alternative is to leave the bad items in the cache forever; as I mentioned in #15, this won't really cause too many problems, but is vaguely troubling from a philosophic point of view anyway. The expiry code isn't too long.

Is #15 okay?

The other alternative would be to get rid of the extension-map in the cache, and keep only the 'missing' list locally, relying wholely on the (not-yet-existant) external service for mapping.

Since there is no agreement here about how it should work, all of the caching functions should just be yanked out for now, as this is causing problems for people. I'm sorry I committed this code in the first place; it was not ready to go in and should not have been put in.

I'll remove it as soon as I can, or someone else can.

Removed extension cache in commit http://drupalcode.org/project/drush.git/commit/01b584b

Should backport to 4.x.

subscribe

Backported.

@msonnabaum,

Thanks so much.

Are you going to be pushing out a v 4.5 soon? Or should I replace my install with the latest -dev version.

Thanks,

Shai Gluskin

I'm not sure the backport actually worked.

Temporary directory has been successfully set to /home/user/local/tmp from within the Drupal 7 interface at:
http://devel.host.com/#overlay=admin/config/media/file-system

$ cd host.com
ln -s sites/default/files host

cd sites/all/modules
$ drush status
 Drupal version                :  7.4
 Site URI                      :  http://default
 Database driver               :  mysql
 Database hostname             :  mysql.host.com
 Database username             :  drupal_host_adm
 Database name                 :  drupal7_host
 Database                      :  Connected
 Drupal bootstrap              :  Successful
 Drupal user                   :  Anonymous
 Default theme                 :  bartik
 Administration theme          :  seven
 PHP configuration             :  /etc/php5/cgi/php.ini
 Drush version                 :  4.4
 Drush configuration           :
 Drush alias files             :
 Drupal root                   :  /home/user/devel.host.com
 Site path                     :  sites/default
 File directory path           :  host
 Private file directory path   :  /home/user/d7_private

drush pm-download date wysiwyg imce google_analytics
Install location /home/nowaste/devel.host.com/sites/all/modules/date already exists. Do you want to overwrite it? (y/n): y
Project date (7.x-2.0-alpha3) downloaded to /home/nowaste/devel.host.com/sites/all/modules/date.                                 [success]
Project date contains 6 modules: date_views, date_api, date_popup, date_repeat, date_tools, date.
WD php: Warning: file_put_contents(/tmp/cache.inc): failed to open stream: Permission denied in drush_pm_put_extension_cache()   [warning]
(line 2419 of /home/nowaste/usr/drush_7.0/commands/pm/pm.drush.inc).
Project wysiwyg (7.x-2.1) downloaded to /home/nowaste/devel.host.com/sites/all/modules/wysiwyg.                                  [success]
WD php: Warning: file_put_contents(/tmp/cache.inc): failed to open stream: Permission denied in drush_pm_put_extension_cache()   [warning]
(line 2419 of /home/user/usr/drush_7.0/commands/pm/pm.drush.inc).
Project imce (7.x-1.4) downloaded to /home/user/devel.host.com/sites/all/modules/imce.                                        [success]
WD php: Warning: file_put_contents(/tmp/cache.inc): failed to open stream: Permission denied in drush_pm_put_extension_cache()   [warning]
(line 2419 of /home/user/usr/drush_7.0/commands/pm/pm.drush.inc).
Project google_analytics (7.x-1.2) downloaded to /home/user/devel.host.com/sites/all/modules/google_analytics.                [success]
Project google_analytics contains a module named googleanalytics.
WD php: Warning: file_put_contents(/tmp/cache.inc): failed to open stream: Permission denied in drush_pm_put_extension_cache()   [warning]
(line 2419 of /home/user/usr/drush_7.0/commands/pm/pm.drush.inc).
file_put_contents(/tmp/cache.inc): failed to open stream: Permission denied in drush_pm_put_extension_cache() (line 2419 of      [warning]
/home/user/usr/drush_7.0/commands/pm/pm.drush.inc).
file_put_contents(/tmp/cache.inc): failed to open stream: Permission denied in drush_pm_put_extension_cache() (line 2419 of      [warning]
/home/user/usr/drush_7.0/commands/pm/pm.drush.inc).
file_put_contents(/tmp/cache.inc): failed to open stream: Permission denied in drush_pm_put_extension_cache() (line 2419 of      [warning]
/home/user/usr/drush_7.0/commands/pm/pm.drush.inc).
file_put_contents(/tmp/cache.inc): failed to open stream: Permission denied in drush_pm_put_extension_cache() (line 2419 of      [warning]
/home/user/usr/drush_7.0/commands/pm/pm.drush.inc).

You're using 4.4. The fix is in 4.x, and soon 4.5.

Okay - thanks. I'll move back to drush 3 until it gets fixed.

@gr33nman, Drush 4 dev works great!:

wget http://ftp.drupal.org/files/drush-All-versions-4.x-dev.tar.gz

Shai

Thanks Shai,

I was able to find it at:
http://ftp.drupal.org/files/projects/drush-All-versions-4.x-dev.tar.gz

I see that it was just updated as of July 5, 2011. Looking forward to 4.5.

I'm curious - the current version that's 'recommended' (green) is 7.x-4.4. I wonder why all the others go by "All-versions-" except the current one.