The initial draft is here - enjoy and comment. See TODO-s in the code.

# completions for Drupal's drush module - put this in /etc/bash_completion.d/drush  or source it from .bashrc or /etc/profile
# currently this is very slow after you press TAB and expects that drush is invoked with "dr" cause it makes some checks via "dr ...". Otherwise because of "complete -F _drush_completion d dr drush drush.php" this completion is triggered on several drush names...

# v 0.1
# authot: rsvelko - Vladimir Radulovski from Segments.at

_drush_completion() {
  # local vars
  local cur prev words drupal_root

  # init the array with the reply
  COMPREPLY=()
  # part of currently completed word
  cur=${COMP_WORDS[COMP_CWORD]}
  # prev completed word
  prev=${COMP_WORDS[COMP_CWORD-1]}


# TODO - determine how to call drush - dr, drush, drush.php ?? Maybe we should standartize the best practice of deploying drush
#+ -> alias or symlink or in the $PATH?

# I chose to use dr status instead of
#+ dr eval "print drush_get_context('DRUSH_DRUPAL_ROOT') . \"\n\" "
#+ cause it is more flexible in the future and less code;
#+ performance is the same

# TODO: the fastest way to get drupal root!


drupal_root=`dr status | grep "Drupal Root" | awk -F ' : ' '{ print $2 }'`
# this outputs /var/www/vhosts/example.com


# echo $prev
#######################################################################
#
# drush commands/options or .info files (projects and their modules)
case $prev in
@(dl|enable|disable|uninstall|updatecode|update|info))
# @ means exactly one occurrence of any pattern. Deliberately no spaces or "" here

# now find all things with .info files
words=`find $drupal_root/modules/ $drupal_root/themes/  $drupal_root/sites/ -type f -name "*.info" -exec basename '{}' .info  \; | tr '\n' ' '`

# TODO: output only enabled modules when disabling and so on ... via a drush command will be best -
#+ !! I need a way to list enabled modules so that this output ca be fed to an enable command... the same with disabled ones and enable...
#+ Remember drush_mm ??

# TODO: I need a standartized way to know what :
#+ - is drush expecting (the basic commands/opts)
#+ - are the enable/disable/cron/--options - expecting ... (cron expects nothing, enable/disable expect modules/themes/etc. and --root expexts a path ..
#+ and --url expects a url)
#+ It will be best if the drush help command gives us this (or maybe it is just the human readable representation of the beneath array and here we eat the array itself - with context and expected args)
;;

*)
# list all drush commands/options based on the output of drush without args
 words=`dr | grep -A 10000 "Options:" | grep "^  " | awk -F ' ' '{ print $1 }'| tr -d ',' | tr '\n' ' '`

;;

esac




words=`compgen -W "$words" -- $cur`
  COMPREPLY=( $words )

}

complete -F _drush_completion d dr drush drush.php

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Owen Barton’s picture

Oh this looks fantastic, and very funny, because I was just thinking that I wanted to investigate that about half an hour before I came online and saw this issue :)

This will need some work (in the completion script, and in drush core) to be clean and usable. Perhaps we could add an "autocomplete" backend (similar to the JSON) one that just lists available auto-completion words at a given point. The performance stuff is more tricky - I am wondering if there is a way we could use some kind of cache to speed this up - alternatively we could force auto-completion to a more limited bootstrap level, which might mean you miss a few things but would be good enough for most purposes (and very fast).

I think "drush" is a good standard name. We have statusmodules command that shows modules enabled/disabled status.

moshe weitzman’s picture

if we have to bootstrap the modules, this is worthless. I think we need a 'no training wheels' mode where we discover all commands just via scan of sites dir. that means we get commands from disabled modules to. i was opposed to this at first but i think it woul be useful for autocomplete and also for a faster version of `help`.

I won't be working on this - hope Grugnog2 and rsvelko and polish and commit.

rsvelko’s picture

EDIT1 : bolded some txt and changed my opinion on the best way to deploy drush (via symlink in the path)

@moshe : if we think of an especially fast way to get the same output as drush status we can use the Drupal Root dir (using find command) or mysql user:pw (using drupal's system table) to find just the enabled modules ...which avoids bootstraping.

= Here I give some todo-s/questions from the above script:

# TODO - determine how to call drush - dr, drush, drush.php ?? Maybe we should standartize the best practice of deploying drush

= ok - drush is the standart way to call this command

#+ -> alias or symlink in the $PATH?

= let's leave to people to decide - the question was meant to shorten the README if one way is better than the other - but I am 50/50 on this so .... (personally I use aliases)

= I've just changed my .bashrc by removing all aliases to the drush.php in favor of a symlink in my $PATH - thus I can call drush from bash scripts - so I propose wechange the README to say that this is the prefered while aliases are still possible

# I chose to use dr status instead of
#+ dr eval "print drush_get_context('DRUSH_DRUPAL_ROOT') . \"\n\" "
#+ cause it is more flexible in the future and less code;
#+ performance is the same
# TODO: the fastest way to get drupal root!

= posted a parallel issue about that

# TODO: output only enabled modules when disabling and so on ... via a drush command will be best -
#+ !! I need a way to list enabled modules so that this output ca be fed to an enable command... the same with disabled ones and enable...
#+ Remember drush_mm ??

= drush_mm does this by using db_query and the system table (time dr mm list : real 0m1.617s)
= time dr statusmodules : real 0m11.356s (yes I know that this command is not the same thing - )
= time find $drupal_root/modules/ $drupal_root/themes/ $drupal_root/sites/ -type f -name "*.info" -exec basename '{}' .info \; | tr '\n' ' ' : real 0m0.288s - FASTEST ( I wish this command knew the multisite to look into - so we look in all/ and in example.com/ under sites/ ... )

# TODO: I need a standartized way to know what :
#+ - is drush expecting (the basic commands/opts)
#+ - are the enable/disable/cron/--options - expecting ... (cron expects nothing, enable/disable expect modules/themes/etc. and --root expects a path ..
#+ and --url expects a url)
#+ It will be best if the drush help command gives us this (or maybe it is just the human readable representation of the beneath array and here we eat the array itself - with context and expected args)

= here we are speaking of this JSON backend for autocompletion

-----------------
@Grugnog2:
statusmodules is way too slow and gives the info not as we want it - we need it the way drush_mm outputs it :

enabled: module_one module_two .....
disabled: ...

separated with a space it is usable by enable/disable commands :)

rsvelko’s picture

ok, shortened and polished the code a bit. posted some new issues to separate the work to pieces.

here is the newest version - see inline for the rest of my post :

# Shell auto completion for Drush ( http://drupal.org/project/drush ) ( http://drupal.org/node/437568 - the issue that we use for this  )
# v 0.2 - 090422
# author: rsvelko - Vladimir Radulovski from Segments.at

# It auto-completes when you invoke drush with - 'd', 'dr', 'drush' or 'drush.php' .

# INSTALL: make a symlink from /etc/bash_completion.d/drush -> /path/to/drush/drush.auto.completion or source it from .bashrc or /etc/profile

# REQUIREMENTS: "drush" named symlink in your $PATH . It uses it to get some info.

# TODO: currently this is very slow after you press TAB
#   TODO: the fastest way to get drupal root, drush commands and enabled/disabled projects!
# ( http://drupal.org/node/437888 )
#
# TODO: I need an automatically generated list of possible auto-completions that drush gives me. Will use the xxx.drush.inc files for that.

_drush_completion() {
  local cur prev words drupal_root

  # init the array with the reply
  COMPREPLY=()
  # part of currently completed word
  cur=${COMP_WORDS[COMP_CWORD]}
  # prev completed word
  prev=${COMP_WORDS[COMP_CWORD-1]}

  # downwards code needs work >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

  drupal_root=`dr status | grep "Drupal Root" | awk -F ' : ' '{ print $2 }'`
  # drupal_root="/var/www/vhosts/versichern24.org.test"
  # this outputs /var/www/vhosts/example.com


  # drush commands/options or .info files (projects and their modules)
  case $prev in
    @(dl|enable|disable|uninstall|updatecode|update|info)) # @ means exactly one occurrence of any pattern. Deliberately no spaces or "" here
    # find all things with .info files
    words=`find $drupal_root/modules/ $drupal_root/themes/  $drupal_root/sites/ -type f -name "*.info" -exec basename '{}' .info  \; | tr '\n' ' '`
    ;;

    *)
    # list all drush commands/options based on the output of drush without args ( nice hack :) works fine )
    words=`dr | grep -A 10000 "Options:" | grep "^  " | awk -F ' ' '{ print $1 }'| tr -d ',' | tr '\n' ' '`
    ;;

  esac

  # upwards code needs work >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

  # generate completion suggestions based on already typed string
  words=`compgen -W "$words" -- $cur`
  COMPREPLY=( $words )

}

complete -F _drush_completion d dr drush drush.php


rsvelko’s picture

the above is the not-so-perfect-but-working variant using find shell command for .info files and parsing drush help's output for commands names...

Two lines of development are left:

1. this issue here - for more inteligent completion I need someone to hint me how to take the $items array from each pm_drush_command() hook implementation in .drush.inc files into my auto-completion script above ...

2. performance and modules listing tasks are dealt with in the http://drupal.org/node/437888 issue.

moshe weitzman’s picture

We have been considering using .info files for commands instead of a php array. Just an FYI. Adrian seemed to long for this recently.

moshe weitzman’s picture

rsvelko’s picture

I am working on the release candidate for the drush.completion file ... Will be ready in 3 days. Will be fast and smart (due to --pipe and --backend).

moshe weitzman’s picture

i'm over here scratching the walls, waiting for this :)

rsvelko’s picture

today is the day for it.

rsvelko’s picture

had attempted to overcome the "completion of commands with spaces in their names" - problem by means of escaping/field separator variables in bash .... but no success after almost 3 hours ... (only my bash knowledge has gone up ... :) )

So, now I am going to realise sth else - for the sql commands for example -

I will do a 1st stage completion that will complete "sql" and then if "sql" has just been completed - look for all sub sql commands ...

Question: are the sql commands structured or "sql dump" is a standalone command?

Will publish the next version later today. (I live in Vienna so I am London + 1 hour.)

moshe weitzman’s picture

Question: are the sql commands structured or "sql dump" is a standalone command?

sql dump is a standalone command.

note that contrib modules will provide own commands and usually use 2 words like sql does. so we can't treat sql as special unless contrib modules can also become special.

rsvelko’s picture

I was supposing so - for "sql dump" being standalone and for contribs that would add new commands.

Luckily there is a way to parse the initial command list and make a suitable autocomplete routine. Working on it today too.

moshe weitzman’s picture

Any progress? We are doing triage to see what is in and whats out for 2.0 release. Not much time left.

rsvelko’s picture

ok, the progress is nice here - I moved almost all the things out of bash and into a php glue code file. A little polish and in 24 hours will submit it.

How much time do I have till I'm late for 2.0 btw ?

rsvelko’s picture

So,

here it is - a working Bash Completion script that can be included in drush 2.0.

It is suited for usage, but lacks many things. See the code for the rest. Install it. Test it and tell me what do you think.

# ABOUT         : BASH Shell auto completion for Drush - it auto-completes when you invoke drush with - 'd', 'dr', 'drush' or 'drush.php'.
# WEB RESOURSES : See http://drupal.org/node/437568 - the issue that we use for this thing.
# VERSION      : 0.6 (entirely bash based, no php helper file or dedicated backend)
# DATE         : 090522
# INSTALL      : (valid for debian with bash) make a symlink from /etc/bash_completion.d/drush -> /path/to/drush/drush.completion or source it from .bashrc or /etc/profile if you do not have privs for the symlinking

# REQUIREMENTS : "drush" named symlink in your $PATH . It uses it to get some info by calling drush.

# TODO         : Limitations : 
#                : 1. for now does not completes the 'dump' part of 'sql dump' - instead shows all commands - because of the whitespace between 'sql' and 'dump'
#                : 2. better modules/projects completion on dl/enable/disable ... for now we do a `find $drupal_root/modules/ $drupal_root/themes/  $drupal_root/sites/ -type f -name "*.info" -exec basename '{}' .info  \; | tr '\n' ' '` ...

# TODO         : Roadmap :
#                 : create an internal use --completion option for drush that will make completion suggestions

# The func that bash uses for drush completion.

_drush_completion() {
  local cur prev

  # init the array with the reply
  COMPREPLY=()
  
  # part of currently completed word
  cur=${COMP_WORDS[COMP_CWORD]}
  
  # prev completed word
  prev=${COMP_WORDS[COMP_CWORD-1]}


  drupal_root=`drush help --verbose | grep -A 1 "root directory at " | tail -1`
  # results to sth like "/var/www/vhosts/example.com" - no new line at its end
  # this is quite faster than `drush status --pipe` or `drush status --backend`

  drush_commands=`drush help --pipe | sed s/\"\ \"/\:/g | sed s/\"//g` # makes them :-delimited

  # output drush commands or .info files (projects and their modules)
  case $prev in
    @(enable|disable|uninstall|updatecode|update|info))
    
    # these above are the module commands
    # @ means exactly one occurrence of any pattern. Deliberately no spaces or "" here
    # find all things with .info files below
    
    word_list=`find $drupal_root/modules/ $drupal_root/themes/  $drupal_root/sites/ -type f -name "*.info" -exec basename '{}' .info  \; | tr '\n' ':'`
    ;;

    *)
    
    # list all drush commands
    
    word_list=$drush_commands
    ;;

  esac

  # generate completion suggestions based on already typed string
  
  oldIFS=$IFS # we need to hack IFS because completion of space-containing drush commands is not a nice task the least ;)
  IFS=:
  
  completions=`compgen -W '$word_list' -- $cur | tr "\n" ":"`
  #echo "<$completions>" # gives them one on per line

  COMPREPLY=( $completions )
  IFS=$oldIFS

}

complete -F _drush_completion d dr drush drush.php

rsvelko’s picture

For discussion:
1. the name of the auto completion backend option for drush - see the code above - I propose "--completion"

2. It would be awesome if drush commands didn't have space in them . This is easily changeable for core and contrib drush commands - do you think we should do this , enforce it on coder and devel modules for example?

rsvelko’s picture

FileSize
2.77 KB

The same file as above attached here.

moshe weitzman’s picture

I put a reference to this file in my bashrc but am getting a syntax error when i open a new shell:

-bash: /Users/mw/contributions/modules/drush/drush.completion: line 39: syntax error near unexpected token `('
-bash: /Users/mw/contributions/modules/drush/drush.completion: line 39: `    @(enable|disable|uninstall|updatecode|update|info))'
rsvelko’s picture

I have deleted the symlink in /etc/bash.comple.... to this file and have written:

". /path/to/file/..." in my .bashrc

Have you done it the same way?

rsvelko’s picture

Category: task » feature
Status: Active » Needs review
FileSize
3.28 KB

btw here is a new version 0.7 - it autocompletes commands like "sql dump" nicely now. This was quite a challenge....

Test this one instead. I suppose (could not replicate) you've not sourced it from .bashrc correctly with the ./source command. See the file attached for install instructions.

rsvelko’s picture

For now I won't implement new features - let's freeze it a bit - just usability enhancements, bugs and docs... So if install goes ok and is easy enough (and if autocompletion works well out of the box :) ) please tell me - did I manage to go into drush 2.0 ?

moshe weitzman’s picture

Priority: Normal » Critical

I am getting the same errors, now on line 42. We need someone else to try this. I am sourcing the file correctly in .bashrc AFAICT

rsvelko’s picture

figured it out - you are trying to execute the file instead of to source it with the "." bash command (also known as "source") :

# cd drush_completion_bash/
# chmod +x drush.completion
# ./drush.completion
./drush.completion: line 42: syntax error near unexpected token `('
./drush.completion: line 42: `    @(enable|disable|uninstall|updatecode|update|info))'

I apologize that above I have written "./source" - I meant (. OR source casue "."="source" in bash ...) So try either ". /path/to/completion script" or "source /path/to/script" written in your .bashrc .... This was written in the script itself in the header section under "INSTALLATION" ....

PS . the completion script is not for standalone execution...

moshe weitzman’s picture

I'm still missing something. This is the final line in my .bashrc and I get the error described earlier

source /Users/mw/contributions/modules/drush/drush.completion

rsvelko’s picture

I am on a debian 4.0 box. Yours? Give more info pls.

rsvelko’s picture

bash version, linux version

rsvelko’s picture

try also the alternative install method with a symlink

moshe weitzman’s picture

Title: Autocmplete awesomeness for drush » Autocomplete awesomeness for drush

Mac OSX v10.5.6.
GNU bash, version 3.2.17(1)-release-(i386-apple-darwin9.0)

I do not have a /etc/bash_completion.d directory.

FWIW, I copied the snippit below into my .bashrc a long time ago - works well. Encouraged by that, I copied this drush.completion code directly into my .bashrc but get similar errors.

# see http://www.macosxhints.com/article.php?story=20080317085050719

_complete_ssh_hosts ()
{
        COMPREPLY=()
        cur="${COMP_WORDS[COMP_CWORD]}"
        comp_ssh_hosts=`cat ~/.ssh/known_hosts | \
                        cut -f 1 -d ' ' | \
                        sed -e s/,.*//g | \
                        grep -v ^# | \
                        uniq | \
                        grep -v "\[" ;
                cat ~/.ssh/config | \
                        grep "^Host " | \
                        awk '{print $2}'
                `
        COMPREPLY=( $(compgen -W "${comp_ssh_hosts}" -- $cur))
        return 0
}
complete -F _complete_ssh_hosts ssh

rsvelko’s picture

Title: Autocomplete awesomeness for drush » Autocmplete awesomeness for drush
FileSize
3.37 KB

one more thing :

the "drush help -v -b " gives me the drupal root in a structured way so I decided to pull drupal root dir from there... so here is v.0.8 attached.

In v.0.7 I used the help command without the -b and got an empty drupal root output on some sites...

rsvelko’s picture

FileSize
3.62 KB

aha I see - removed the @() bash magical stuff from the case ...

now works here when invoked direclty and should work both install ways in your bash... Feedback on the attached 0.9 version appreciated.

rsvelko’s picture

the problem was not in the source command but in the strange bash syntax that worked on debian and not on Mac OSX

moshe weitzman’s picture

OK, this last one works for me. Thanks for your persistence, rsvelko. I think this is RTBC. Anyone have comments?

I noticed is that we could give a nicer error when you use it outside of a drupal site. It currently errors with

find: /modules/: No such file or directory
find: /themes/: No such file or directory
find: /sites/: No such file or directory

Also, the list of modules to enable should differ from the list to disable which should differ from all. Not sure we care.

I would love to shave a few milliseconds off of the completion time. Can you think of a good way to do that? Is most of the time spent in gathering data from drush or is it in find? compgen?

I have no problem making a single drush command that returns the enabled module and the root in one go.

Owen Barton’s picture

I think this needs a bit more polish. It basically works for me, although I get a "tr: warning: an unescaped backslash at end of string is not portable" message when attempting to auto-complete.

Also, the general approach looks a kinda kludgy in places - I don't like the idea of doing a "find" to get info files - this doesn't differentiate projects and modules so gives invalid results for some commands, and I am pretty sure that a system table query that drush could output in exactly the right form a lot faster and corrector.

The script does a string replacement to change the delimiter to ":" - we should just change the drush output to be directly usable I think.

I am going to have a quick play on the drush side of things and see if I can come up with some more useful output.

Owen Barton’s picture

It appears that PHP (at least on my Ubuntu system) does something to break completion/readline when invoked via a bash function.

#!/bin/bash
_drush_completion() {
  local cur

  COMPREPLY=()
  cur=${COMP_WORDS[COMP_CWORD]}
  words=$(drush --pipe)
  COMPREPLY=( $( compgen -W '$words' -- $cur ) )
  return 0
}

complete -F _drush_completion d dr drush drush.php
#complete -W "$(drush --pipe)" d dr drush drush.php

The first case will autocomplete the first suggestion (eg "status" on "st"), but won't show other options (e.g. "statusmodules) as it should, and further breaks readline by not allowing backspace to delete characters on the line. No idea what is happening here - but it is nothing to do with bash output - it occurs whenever php is executed (I tried within a bash subshell, with an exec in front as an independent command - adding no input to the auto-completion). If we can figure this out, I think we should be able to have a very simple way of wiring together generic auto-completion that all commands can add to without needing to extend the bash script.

The second case (commented out) actually works great, although only for help commands (with a small patch to do $pipe[$key] = str_replace(' ', '-', $key); and drush_print_pipe(implode(" ", $pipe)); for testing purposes). This can't really go further however, because AFAICS the currently completion text is not available in this bash context.

rsvelko’s picture

At this point we :
1. have a rough draft that works (I am using debian 4.0)
2. need to pave the way strategically to solve this problem
- drush backend commands specification
- the best way to eat this backend info and use it in a tiny bash script

If you look into the code I've attached last you will notice it is quite hacky/complicated at times (the IFS part mostly). Beleive me this was the only way I could succeed in autocompleting the "sql query" - space delimited type of commands ....

According to me the rght way to go (mentioned before) is to carefully think of the drush backend commands and let the bash script just do the final complete statement or 1-2-3 more lines ...

Frando’s picture

To speed things up, couldn't we just cache the autocompletion lists in a file? So you'd check if there's a file .drush-completion-cache or something in the current directory, and if so, use that, and only recreate it if it's not present? So no more find on each time you use drush.

rsvelko’s picture

what mechanism to refresh the cache would you suggest - cron?

Frando’s picture

Well, I'd let drush do it (cron might lead to permission problems if it's not executed as the same user who is invoking drush). So I'd clear it on all pm operations (dl, update) and maybe on cache clearing or also additionally time based.

naught101’s picture

Title: Autocmplete awesomeness for drush » Autocomplete awesomeness for drush

Pity there's no auto complete for drupal.org, eh? :D

Owen Barton’s picture

I discovered what I am pretty confident is the issue with autocomplete on Ubuntu - this specific PHP build is doing something non-standard with the way it opens stdin. For licence compatibility reasons the Ubuntu PHP builds link against (the seemingly quite broken) libedit for readline functionality, rather than libreadline, and it is likely this is why we are seeing this problem here. I have attached a note and a simplified test case to https://bugs.launchpad.net/ubuntu/+source/php5/+bug/322214 - please add your thoughts there too. There is similar issue at https://bugs.launchpad.net/ubuntu/+source/php5/+bug/230030 where the note about licensing and libedit being generally broken is.

Given that Ubuntu is a pretty major segment of Drush users (up there with OSX for sure), I am not sure what the best approach is:
1) Go ahead with autocompletion but tell Ubuntu users not to set it up for now
2) Try and fix libedit support ourselves and/or wait for it to be fixed upstream (probably 6 months at least)
3) Ask all Ubuntu users to upgrade to a non-Ubuntu build (5.3 from http://php53.dotdeb.org/ for example does not have this issue)
4) Try an alternate route that doesn't involve calling php in a complete -F function. For example - if we implemented a "complete" command/parameter that output all commands (and possibly command followups like modules etc, although that needs whitespace mangling) we could use something as simple as complete -W "`drush complete`" -f drush

intuited’s picture

FileSize
5.49 KB

Hi folks,
I took version 0.9, from may 28th, and ended up doing a complete rewrite. The results are:

  • it's much faster.
    • I found that the version 0.9 script was taking well over a second to run.
      • this may be due to something in my drush/drupal setup: the drush calls were nearly half a second each. Is this normal?
    • This one takes about .1 seconds worst-case (ie when modules are involved). for pure command completion it's well under that, unless a cache refresh is needed.
    • the speed increase is mostly due to not calling drush except when absolutely necessary.
    • the use of a commands cache allows most of those drush calls to be avoided
    • also the drush root dir is determined by searching for index.php in current & parent directories rather than by calling drush.
    • it's still kind of slow when getting modules; this could be remedied by using dnotify to update a cache.
  • if there were problems when parsing command lines containing option parameters, these problems have been resolved.
  • command names given after "drush help " are completed.

Things to keep in mind:

  • I haven't tested it for portability, though it should be fine. I run ubuntu linux. I haven't had any problems with stdin-related issues.
  • due to the way the detection of drupal root dir works, module/theme completion won't work if the current directory is more than 6 levels below drupal root.
  • commands are cached. if you upgrade to a new version of drush that has different commands, delete the cache at "~/.drush-complete/command-cache".
    • the cache will be expired every 24 hours otherwise. this is easily-enough changed by setting a constant.
  • it depends on the file "index.php" being unique to the root drupal directory.

Coding notes:

  • I dispensed with the colon-delimitation and rewrote it to delimit items using newlines. this makes it easier to use tools like grep and sed.
  • looking through the code again I realize there are some extraneous variables. They won't cause problems except in the readability area.

The new code:

#!/bin/bash
# ABOUT         : BASH Shell auto completion for Drush - it auto-completes when you invoke drush with - 'd', 'dr', 'drush' or 'drush.php'.
# WEB RESOURCES : See http://drupal.org/node/437568 - the issue that we use for this thing.
# VERSION      : 0.10 (now usably fast)
# CHANGELOG    :
#   : 0.10 - (--17jun2009 7h04--) basically a complete rewrite.
#          - made it a lot faster by avoiding calling drush whenever possible and by using sed instead of basename.
#          - works with option parameters
#          - cleaned up the code, putting stuff in functions.
#          - the script now creates a ~/.drush-complete directory where it stores a command cache.
#          - this rev courtesy of Ted Tibbetts (username "intuited" at the google mail location)
#          ! not tested for compatibility; please let me know if you have problems.  I run ubuntu linux.
#          ! it won't work if the current directory is more than 6 levels below drupal root.  this seems unlikely to be a problem.
#   : 0.9 - made it Mac OSX compatible I hope; renamed some vars
#   : 0.8 - changed the way drupal_root is determined
#   : 0.7 - we handle 'sql dump'-like commands correctly

# DATE         : 090617
# INSTALL      : (valid for debian with bash) 
#    : EITHER 1. cd /etc/bash_completion.d/; ln -s /path/to/drush/drush.completion
#    : OR     2. mcedit .bashrc; then write ". /path/to/drush/drush.completion" in it ('.' = source)

# REQUIREMENTS : "drush" named symlink in your $PATH . It uses it to get some info by calling drush.

# TODO         : Limitations :
#                : 1. better modules/projects completion on dl/enable/disable ... for now we do a `find $drupal_root/modules/ $drupal_root/themes/  $drupal_root/sites/ -type f -name "*.info" -exec basename '{}' .info  \; | tr '\n' ' '` ... which returns ALL present modules/themes

# TODO         : Roadmap :
#                 : create an internal use --completion option for drush that will make smarter completion suggestions


##constants
__DRC_SETTINGS_DIR="$HOME/.drush-complete";
__DRC_COMMAND_CACHE="$__DRC_SETTINGS_DIR/command-cache";
__DRC_COMMAND_CACHE_EXPIRE=86400;  # expire the cache every 24 hours


##support functions

#set up the settings directory where the command cache is stored
function __drc_setup_settings_dir {
  if ! mkdir -p "$HOME/.drush-complete"; then
    echo "drush-complete: couldn't create settings directory." >&2; return -10;
  fi;
}
  
#get a word list from cache unless it's nonexistent or expired
# word list is returned on stdout, one per line.
function __drc_get_commands {
  if ! __drc_setup_settings_dir; then return $?; fi;
  if [[ ! -e "$__DRC_COMMAND_CACHE" ]] || (( $(stat -c %Y "$__DRC_COMMAND_CACHE") < $(date +%s) - $__DRC_COMMAND_CACHE_EXPIRE )); then
    drush help --pipe | sed 's/\"\ \"/\n/g' | sed 's/\"//g' | sort | tee "$__DRC_COMMAND_CACHE"; 
  else
    cat "$__DRC_COMMAND_CACHE";
  fi;
}

#find all directories containing .info files that start with $1
# list is returned one per line
function __drc_get_modules {
  # if we can't find the drupal root, then return unsuccessfully -- we're not in a drupal directory.
  if ! drc_drupal_root="$(__drc_get_drupal_root)"; then return -1; fi;

  find $drc_drupal_root/modules/ $drc_drupal_root/themes/  $drc_drupal_root/sites/ -type f -name "$1*.info" | sed -r 's_.*/(.*).info_\1_';
}

#get the path to the drupal root install.
# ! assumes that the file called "index.php" that is closest to the current directory
#   (tracing upwards through the directory structure)
#   that contains '// $Id: index.php,'
#   is in the Drupal root install directory.
# results to sth like "/var/www/vhosts/example.com" - no new line at its end
function __drc_get_drupal_root {
  local root;
  INDEX_PHP_CANDIDATES=(./index.php ../index.php ../../index.php ../../../index.php ../../../../index.php ../../../../../index.php ../../../../../../index.php);
  if ! root="$(dirname $(grep -l  '//\s*$Id:\s*index.php,' "${INDEX_PHP_CANDIDATES[@]}"  2>/dev/null) 2>/dev/null)"; then
    #if we did not find Drupal's index.php, return unsuccessfully -- we're not in a drupal directory.
    return -1;
  fi;
  echo "$root";
}


##The func that bash uses for drush completion.
_drush_completion() {
  oldIFS="$IFS";

  #set up completion data structures
  local word;
  local drc_word_list;
  # get drush command list
  IFS=$'\n';
  drc_commands=("$(__drc_get_commands)");
  drc_commands_re="($(echo "${drc_commands[*]}" |tr '\n' '|'))";
  # init the array with the reply
  COMPREPLY=();

  #parse existing command line -- parts of commands get precedence.
  set -- "${COMP_WORDS[@]}"; shift;  # shift out the drush command

  if [[ $1 =~ -.* ]]; then shift; fi;    #ignore option parameters

  # test for help request
  local help;
  if [[ $1 == help ]]; then
    help=1;
    shift; if [[ $1 =~ -.* ]]; then shift; fi; #shift and ignore option parameters
  fi;

  local prefix="$(echo "$@" |sed 's/\S*$//')";
  local matching_commands=($(echo "${drc_commands[*]}" |grep "^$(echo "$@")"  |if (( ${#prefix} > 0 )); then sed "s/$prefix//"; else cat -; fi;));
  if (( ${#matching_commands[@]} == 0 )) && ! [ $help ]; then
    # if there are no matching command completions
    shopt -s extglob;
    case "$1" in
      @(enable|disable|uninstall|update|updatecode|info))
        shift; if [[ $1 =~ -.* ]]; then shift; fi; #shift and ignore option parameters
        COMPREPLY=($(__drc_get_modules "$@" | grep "^$(echo "$@")" )) ;
    esac;
  else
    COMPREPLY=("${matching_commands[@]}");
  fi;

  IFS="$oldIFS";
}

complete -F _drush_completion d dr drush drush.php
Owen Barton’s picture

@intuited - what version of Ubuntu are you using, and are you using the standard Ubuntu PHP build (5.2.6-3ubuntu4.1)? I am on 8.04 (Jaunty), and have tested on 2 separate machines with the same results, as well as with a php 5.3 build (see above).

Could you please try out my simplified test case at https://bugs.launchpad.net/ubuntu/+source/php5/+bug/322214/comments/4 and and let me know if that works?

intuited’s picture

Hi Owen,
I'm running php5 version 5.2.6-2ubuntu4.2, under Intrepid with the backports repo sourced.
I get the same results you do with the test script: after uncommenting the /php.*/ line both tests fail in the way that you described.
On the upside, this issue will rarely affect my version of the drush-complete script, since it only calls php when rebuilding the command cache.

rsvelko’s picture

@intuited:

totally agrre with the changes you made in v.0.10 .

My thoughts:

1. I want my name in the header :) - if it is not there now. I guess I have forgot to write it...
2. I tested your rework - it is indeed faster - and works.
3. The only thing that didn't work was the autocompletion of modules - nothing happens when I press tab after "drush disable ". THIS is the most important point from the 5 ....

4. Overall - the code is nicer your way - and I am glad that you had a point to start from - you've kept several important parts of my initial code...

5. We need a nice way to rebult the cache when a new module is installed and when new drush commands are installed ... This would require us to hook to some drush internal hook???

Owen Barton’s picture

I managed to fine a work around for the Ubuntu php build issue, and am currently experimenting with a more php oriented approach that I think will make it easier for commands to add their own completions for arguments and options.

Owen Barton’s picture

Assigned: rsvelko » Owen Barton
FileSize
10.06 KB

Here we are. So far I have just added argument completion to the enable/disable commands, but it should be easy to add for others by just implementing hook_drush_complete. I have also added smart --option completion, and made it work with our crazy two word commands (it's tempting to just make all those hyphenated!). It uses drush command parsing, which means that you can put options before commands without causing problems.

Performance is fine on my box so far (0.3s or less), but I have designed it so that if we think we need more performance (after adding more completions) the $complete array is easily cachable - just serialize and store in a tmp directory, named as short md5 of the site root and uri, then add logic to expire every few hours and also after appropriate pm commands. To keep the logic simple I am collecting all completions each time, rather than trying to do lazy loading, which could get quite complex once we have several commands adding to the mix.

Owen Barton’s picture

FileSize
17.76 KB

Here is an improved version. This adds autocompletion for dl/update/info commands (will autocomplete any project name on d.o), as well as watchdog types. It also adds basic caching (needed for the dl/update/info for obvious reasons), and also a layer of optional indirection in the array structure so that multiple commands can share the same completion list. This makes it plenty fast - between 0.1 and 0.2s on my machine.

There is still a fair bit of work to do:
* Need to make the "complete" command hidden from help (probably)
* There is currently a bug in completing commands with spaces - we need to try adding a space on the end when searching perhaps?
* Need to add another cache after the construction of the full project list - this is very expensive to generate and can also be shared between sites (unlike the regular autocompletion caches)
* Need to improve the structure of drush_get_option_help() so we can easily extract the options.
* See if the $complete_arguments conditionals can be safely simplified (whilst still being defensive enough)
* Check that all the --pipe stuff is now removed
* Add doxygen to some of the new functions
* Add completions for other commands

rsvelko’s picture

Clarifying for the audience:
1. Owen has done a mojor push forward (the usage of php - not only of bash ) (respect for that kind of expertise and knowledge of the drush code ) - the right way to do drush auto-completion is by leveraging drush itself which is php based....

The sophisticated coolness we need is acheivable only by using php for the whole work and writing a

bash.completion like that :

(Citation from the recent patch of Owen :)

Index: drush.completion
===================================================================
RCS file: drush.completion
diff -N drush.completion
--- /dev/null 1 Jan 1970 00:00:00 -0000
+++ drush.completion 1 Jan 1970 00:00:00 -0000
@@ -0,0 +1,9 @@
+#!/bin/bash
+_drush_completion() {
+ oldIFS="$IFS";
+ IFS=$'\n';
+ COMPREPLY=( $(drush complete "${COMP_WORDS[@]}" < /dev/null) )
+ IFS="$oldIFS";
+}
+
+complete -F _drush_completion d dr drush drush.php

(End of citation)

@Owen - it is tested by you and works, right?

@drush experts - read the patch and tell us what you think.

@non-experts on drush - patch drush and test if it works ...

I myself tried to test this drush patch with no success:

1. checked out the latest HEAD

2. patch failed
vla@segments:/var/www/vhosts/bin/drush# patch -p0 <437568_0.patch
patching file commands/pm/pm.drush.inc
patching file includes/drush.inc
patching file includes/environment.inc
patching file commands/core/core.drush.inc
patching file drush.php
Hunk #1 FAILED at 156.
1 out of 1 hunk FAILED -- saving rejects to file drush.php.rej
patching file drush.completion

Am I doing sth wrong. I am using the latest HEAD I hope... Please s.o. resolve the failed patch...

PS. Using the bash version for now (#42) . Its purpose was to be the fast solution while we build the slower and better php-based one.

jonhattan’s picture

EDIT: this is a response to #41

@Owen: note there's also available php 5.2.x with readline at dotdeb.org => http://www.dotdeb.org/instructions/ ... For my debian testing I had to specificy the package version to apt-get.

If you wanna try and fix libedit perhaps the best place to start is at http://wanderinghorse.net/computing/editline/

Although libedit/readline is not needed for this issue I think it is of interest in order to run drush in interactive mode (#543550: Drush interactive mode).

intuited’s picture

Note to users of ubuntu and other .deb-based distros:
I haven't found a convenient .deb download for Ubuntu, but the commands

$ sudo aptitude build-dep php5            # install packages required to build php5
                                          #   as configured by debian/ubuntu
$ sudo aptitude install libreadline5-dev  # install the readline development files
$ apt-get source php5      # download and unpack the distro-configured php 5 source
$ cd php*
$ ./configure --with-readline  # modify the distro-specified build configuration
                               # to use the readline library
$ make        # takes 10 minutes or so
$ make test   # another 10 minutes or so
$ sudo make install    # by default this will install php5 to /usr/local;
                       # to change the location pass an option to `./configure`.
                       # `.configure --help` gives a list of options.

should get you a readline-friendly version of php that is otherwise configured as per your distro specifications. Of course it won't get updated by your package manager.

rsvelko’s picture

@ last comment:
question 1: this nice behaviour does not come preconfigured by default?
q2: what features exactly do we get from the above instructions??

intuited’s picture

@rsvelko (and others!)
Sorry, that was really misleading. Those commands will actually get you a version of php, installed in /usr/local/bin, that is compiled with readline, and will run basic command-line apps, but lacks some other functionality and isn't configured to read initialization scripts (eg /etc/php5/cli/php.ini) from the normal location.

What you probably want to do is something like the following, which will actually build the package as per your distribution's intentions (I know that's what I said last time. This time for sure!), but use the standard readline library rather than the semi-functional libedit library. What this means is that it's essentially the same as doing "sudo aptitude install php5", except that the resulting php binary /usr/bin/php will use readline instead of libedit.
This is pretty much the way to go; Debian and the downstream distributions like Ubuntu would allegedly have done this if not for incompatibilities between the licenses used for php (PHP license) and readline (GPL). So I'm not sure if this is actually legal but anyway it does seem to work quite well. It's actually not much more complicated than my earlier mistructions. The main difference is that you need to edit the debian makefile instead of just passing a command-line parameter. Here goes:

##
# First you need to install packages that will be used during the build process and afterwards.

$ sudo aptitude -r install build-essential devscripts
#   build-essential installs the compiler, make tool and other commodities 
#     central to the world of open source software
#   devscripts contains scripts to facilitate the building of debian-ish packages (.debs)

$ sudo aptitude -r build-dep php5     
#   This installs packages (prerequisites) that you'll need in order to install the built php5 packages

$ apt-get source php5
#   this will download the source and the distro-specific build info and patches for php5.


##
# At this point you're ready to start customizing the build as delivered by your distro

$ cd php*
#   this will be a directory with a name of the form "php5-$VERSION"

$ sed 's/--with-libedit/--with-readline/' debian/rules >/tmp/php-debian-rules
$ cp /tmp/php-debian-rules debian/rules
#   The file "debian/rules" governs how the php5 packages will be built by the debian packaging scripts.
#   These last two lines change references to libedit into references to readline.
#   In practice it's probably wiser to do this with an actual editor
#     so you can make sure nothing really weird is going on.
#   In my case, and I think this will be the norm, there was only one instance of the string "--with-libedit":
#     It was in the options to the c compiler to be used when compiling the CLI target.
#   Anyway just go through the debian/rules file and change where it says "--with-libedit" to read "--with-readline".

$ dch --local readline "local build with readline"
#   This just adds a new entry to the php5 debian package's changelog describing what you're doing.  
#   The "readline" parameter will be used to make the new version name.

$ debuild -us -uc 
#   This compiles the code and builds the .deb packages that will be deposited in the parent directory.
#   It takes a while.
#   The -us and -uc options tell it not to sign the package, which you want 
#     because you're not a maintainer who's going to be distributing it.
#   You can also add the '-b' (binary-only) option here, which specifies not to include the source.
#     I didn't bother but it might speed things up, or give smaller .deb files.

$ dpkg -i ../*.deb
#   This will install the packages you've just built in much the same way as running "aptitude install" does, 
#     but without the preliminary step of fetching packages from the repository first (and some other niceties)

##
# At this point, if you didn't get any error messages, you should be good to go.  Try this:
$ /usr/bin/php -v
#  I get:
PHP 5.2.6-2ubuntu4.3readline1 with Suhosin-Patch 0.9.6.2 (cli) (built: Aug 27 2009 22:10:40) 
Copyright (c) 1997-2008 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2008 Zend Technologies
    with Xdebug v2.0.3, Copyright (c) 2002-2007, by Derick Rethans

This is sort of a brute-force method in that you end up installing php stuff that you may not need; it's probably better to pick and choose which debs you actually need to install.

You probably won't need to install the php5-dev package but if you do you will have to install the 'shtool' package:

$ sudo aptitude -r install shtool

The file debian/README.source mentions that if you install the php5-dbg.deb file, you will probably need to install some other packages first, and that you can get a list of likely suspects by doing

$ dh_testdir && egrep '^Package' debian/control | cut '-d ' -f2 | \
	egrep -v '(^php5|dbg|dev|common|pear)$' | tr "\n" "|" | sed 's/|$//' |\
	sed -r 's/([^|]+)(\||$)/ \1 (= ${binary:Version}) \2/g'; echo

But you should already have those installed if you've already installed that package through aptitude. In general if you have dependency problems when running dpkg, it's because you're installing packages that aren't already installed on your system; maybe try `aptitude search '~n^php' |grep '^i'` to see what's actually installed, and then just run dpkg -i on those packages and the ones they depend on.

If something goes wrong, or you would like more background info, I found this guide from the Debian website to be a good reference. Section 7.14 is particularly relevant.

Hope this helps!

rsvelko’s picture

ok, I still do not get it - why would we need to install the better readline lib ?

What would one benefit from the above instructions as a drush user/ autocompletion script developer?

Owen Barton’s picture

@rsvelko - please see my note in #41 describing how Ubuntu and Debian php comes with libedit rather than readline for licensing reasons. This was causing no end of problems with the original completion scripts because of bugs in libedit stdin handling.

I worked around this in my patches my adding "< /dev/null" to the drush call, so this should no longer be an issue and we don't really need to figure out how to recompile and/or distribute a php with readline support for Ubuntu/Debian users for this patch. I am not sure if the "< /dev/null" workaround will work for an interactive drush shell though, so intuited's work may still be beneficial there.

intuited’s picture

Since this topic has now come up in two different issues (this one and #543550: Drush interactive mode), and isn't really directly related to either of them, I've made a separate issue for it: #562888: The readline library, and reposted my last comment on that thread. If people like I can clear out my last comment on this issue, since it's long, mostly if not completely off topic, and now redundant.

moshe weitzman’s picture

Priority: Critical » Normal
moshe weitzman’s picture

Status: Needs review » Needs work

Anyone up for reviving this now that spaces are disallowed in command names. Should simplify the code.

Owen Barton’s picture

I have some more ideas for this too ... among other things, we need a fastpath for super quick cache responses ;)

yhager’s picture

the patch in #48 does not apply to latest HEAD

mariomaric’s picture

Subscribing.

jennifer.chang’s picture

subscribe.

moshe weitzman’s picture

lets revisit this soon now that we have drush cli command, and no spaces in command names.

marvil07’s picture

subscribing

marvil07’s picture

FileSize
8.49 KB

Based in patch in #48, I rerolled it to the actual HEAD, but I tried to avoid change what it does. I mean, it still needs work to update it to the new commands.

moshe weitzman’s picture

FYI, If you use the new `drush cli` shell in drush3, you already get completion of commands. However, if you use a sitealias like `drush @dev pm-status` you can't autocomplete on the alias nor the command which is a bit unfortunate. I'm looking to all you wizzes in this issue to come up with a fix.

Once we get completion of site aliases and commands, we should tackle arguments.

greg.1.anderson’s picture

One thing I was thinking about for drush cli would be to print the prompt something like this:

drush @dev>

This would indicate that the current site was @dev. Another command, such as use @stage would switch the current site to @stage and update the prompt appropriately. Then it wouldn't be necessary to do any special auto-complete handling for commands, just arguments.

I'd imagine that the above would best be implemented in drush cli by setting environment variables; the drush bootstrap would then need to be updated to check the environment variables if there were no options / args that specified the site. The 'cd' command could also be updated to set the environment variable and prompt if you switched to a directory inside some Drupal root, so things would work pretty much as they already do vis-a-vis the current working directory.

Sound good?

moshe weitzman’s picture

Thats awesome, Greg. I think this is still valid for 3.1 (no API change)

greg.1.anderson’s picture

Sounds good. I'll put a patch for #67 in a new issue when it's ready. All of the autocomplete wizzes can continue working on autocomplete for args in this thread.

foripepe’s picture

Component: Code » PM (dl, en, up ...)

Subscribe.

PS. Sorry to change the Compontent to PM, but there isn't Code any more.

colan’s picture

Subscribing.

frob’s picture

subscribe

rsvelko’s picture

so do we abandon this bash script? Now that we have drush cli ?

I hope some ideas from here were used there if we kill this script here.

moshe weitzman’s picture

autocomplete of arguments/options/site_aliases is still wanted.

webflo’s picture

Hi,

started a new project on github for drush autocompletion on zsh. It is in early development. Please test and review ...
https://github.com/webflo/drush_zsh_completion

infojunkie’s picture

subscribe

sch4lly’s picture

sub

colan’s picture

@webflo: Could you move your zsh project to d.o (if this hasn't been done already)? Now that we've got Git working here, it's much easier to work with. Thanks! Looking forward to checking it out / helping with it.

webflo’s picture

@colan I'm planning this I'm still not sure how. whether one or two repositories or different branches. what do you think about this?

colan’s picture

Now that I think about it, I'm not convinced that we need another project for this, just separate issues in this one (project).

As this issue is now long & hairy, perhaps we could simply close it & create new issues for each shell? Or are there still core changes to Drush that still have to happen? If so, keep this issue open for that. Otherwise, I propose the following new issues:

  • Bash autocompletion for Drush
  • Zsh autocompletion for Drush
moshe weitzman’s picture

FYI, the last Example for core-cli --pipe shows how to bring drush command autocomplete to your usual bash shell.

We still want more completion for site aliases, options, and arguments. Also note that we will soon be getting git style 'shell aliases'. We'd like completion on those as well.

boombatower’s picture

subscribe

colan’s picture

New project for the zsh stuff on d.o: http://drupal.org/sandbox/webflo/1113394

RobLoach’s picture

Component: PM (dl, en, up ...) » Core Commands

Like!

chOP’s picture

As someone competent in most things bash and who's a recent convert to drush, I'll take a look at how to provide the features requested by moshe in #81.

Owen Barton’s picture

Note that the bash part is pretty trivial, and mostly already written I think - see drush.completion in the comment 48 patch, rerolled in comment 65. The code for generating the suggestions should live inside PHP, so that we don't have duplicate code for each different shell, and also so we can use a hook to allow command files to generate their own suggestions for possible arguments (where that makes sense). We should also use the new caching system #1172044: Cache command files / Add drush cache API rather than the DIY one in that patch - the command file caching introduced there should also help a lot with latency (which is obviously really important here).

msonnabaum’s picture

I actually started this from scratch (perhaps foolishly) last week with the cache api in mind:

https://gist.github.com/e2231815a99d9395fb76

I've had that in my /etc/bash_completion.d since and it works quite well for all commands and global options. It also caches per bash session which is nice for speed, but not ideal if you're working on multiple sites.

If we're good with continuing with my approach, the next step is command specific autocomplete.

Owen Barton’s picture

Status: Needs work » Needs review
FileSize
21.76 KB

Following on from the code sprint, I have been working on this over the last couple of weeks. The current patch did start from the #48/65 patches, but at this stage is pretty much a complete rewrite, and has gone through several major refactorings. There is a lot of documentation in the patch, so I won't go into too much detail here. This is pretty fully featured in that it can do context sensitive completion of site aliases, options (global and command-specific), commands, arguments, shell aliases and engines. All of this should be working, except that we don't have code yet for each command to return completions for arguments (where possible/sane etc). We do have the hook in place to collect arguments (commandfiles just return an array of strings) and caching system for this in place and it seems to work . I think we can probably add each of those in separate issues.

We started out in Boston thinking of this as a hidden command, but that turned out to be somewhat problematic, because we get conflicts with stuff like --debug producing output which breaks completion and valid commands/hooks on the command line we are parsing returning output. Also, the php parsing to get to a Drush bootstrap still takes quite a bit of time, and unnecessary when we have a completion cache existing (which we would do most of the time). Moving this to a separate include file called before we start the official bootstrap avoided the environment issues and cut about 80% off the run time when using the cache. The cache strategy is documented pretty well in the file, so I won't go through that here.

One gotcha I hit is that we have a hardcoded check in drush_parse_args() (which we use to parse the command being worked on) that short options (-r etc) must always have a value. This is obviously not an issue for completion - the whole point is that the command is not yet complete. This check didn't make sense to me for other reasons, because we don't currently do the same validation on longopts (you can include --root= for instance, with no complaint). It would be good to go through the various places in the bootstrap or specific commands that own these and ensure they have checks, but as far as I can tell an empty shortopt is dealt with in just the same way as an empty longopt, so this shouldn't break stuff in theory.

There is a completion script for BASH included and a pretty reasonable set of initial tests. Have a play, take a look and report back :)

greg.1.anderson’s picture

I only tested lightly, but it works well and if pretty cool.

I tried to find a way to automatically register a completion function in drush core-cli, but I ran into a problem insofar as the complete -F registration function only operates on the list of command names to complete (e.g. "drush") provided to it. I guess we would need to include all of the drush command names and command alias names for this to work right.

moshe weitzman’s picture

@Greg - I'm ready to simply remove core-cli once this lands. I think msonnabaum had some code which let you persistently operate on a given alias.

greg.1.anderson’s picture

The "use" fn currently just sets an environment variable; the same technique could be used w/ "regular" drush if we just checked for same during the bootstrap. cdd / cd could just become an example alias in one of the documentation files. Then I think core-cli can go away. Maybe we do still want a command that will output the bash configuration needed to set up your autocompletion and drush aliases (called core-cli or something else)?

moshe weitzman’s picture

Yes, Mark and I discussed such a drush command. It would validate that your bash_complete is setup by testing for the presence of a 'complete' unix command. If present, it would go ahead and try to symlink the drush's completion script into the bash complete directory. If thats not feasible, we ship with a paragraph in the README.

greg.1.anderson’s picture

Yeah, that would work. From core-cli:

      $is_shell_builtin = (drush_shell_exec("bash -c \"builtin $key\"") == "1");

Of course, if you have bash, you probably have complete (for bash 3.2 and later), but complete is a builtin, so you can't test for it with 'which'.

As an alternative, we already have examples/example.bashrc. Currently this shows how to make a custom bashrc file for use in core-cli, but if core-cli goes away, we could replace it with the completion scripts and instructions on how to use it. For example:

#
# This is a cool drush bashrc file.  To use it:
#
#    drush topic docs-bashrc --pipe | sudo tee /etc/profile.d/drush_completion.sh
#

Then users have a single line to paste into their shell to configure their shell, and we can easily add whatever convenience aliases (e.g. "use") we'd like to the example file.

Well, users will also need to source the configuration script, or log out and log back in again before it will work, which is a little mysterious. If we could run "source" from the setup command [*] then I might think it's worth it, but without that, it might be -more- confusing to have a custom command. I'm a little on the fence about that, though.

[*] If you call "source" via exec, it will run in a subprocess that does not affect the parent shell. :( Even if there is a way around that, the user may have other shell windows open that won't be updated.

Owen Barton’s picture

I think if we have some kind of cli setup command we should offer the option of installing in the users bashrc as well as in the systemwide location. The drush.complete.sh script works fine sourced this way, and I expect a file of useful shell aliases/tricks could work this way too. We would want to ensure we can preg_replace update it in later versions, so perhaps wrap it in start/end block comments. I think keeping some of the shortcuts from core-cli such as cdd and use would be useful for people.

To check for complete, I think we could just check the BASH_VERSION shell environment variables - /etc/profile and .profile use the environment to detect bash, so I think it should be reliable enough for us. We can also check the BASH_COMPLETION* environment to determine systemwide install locations.

drzraf’s picture

1) The bash completion package already has the builtin _have() function which test whether or not a command is available.
Most GNU/Linux distribution provide package functions to integrate with the bash-completion facility.
The completion file "drush" can start with:

#!/bin/bash
have drush || return
_drush() {
    [...]
}
complete -F _drush d dr drush drush.php
# but personally I would only complete "drush", if people need completion they know about aliases

(some examples there)

2) Something is wrong around $set_command_name = $arguments[0].
Eg:

$ drush php-<TAB>
php-eval
php-script

but

$ drush php-s<TAB>
[ nothing ]

Code around line 87 should rather looks like, isn't ?

$a = drush_complete_match($arguments[0], 'command-names')
if (count($a) == 1)
   $completions = array ( $a[0] )
drzraf’s picture

Adding
$completions += drush_complete_match($last_word, 'command-names');
below
drush_complete_match($last_word, 'arguments', $set_command_name);

at the end of the if ($set_command_name) part of the condition
seems to do the trick.

moshe weitzman’s picture

Gave this a spin and it is very close. A couple problems I encountered

  1. I aliased drush to 'dr' but thats not recognized by the script. I see code that tries to handle it but is apparently broken
  2. Would be good to show us how commands can provide argument completion hints. Two good examples are archive-dump and archive-restore. The first takes a site alias and the second takes a filename.
  3. Completion of drush commands works when there are multiple matches but breaks when only one match is left. See what happens when I narrow down to core-topic below:
    ~/htd/d7/sites (7.x *<)$ drush core-
    core-cli             core-execute         core-status
    core-config          core-global-options  core-topic
    core-cron            core-rsync           
    ~/htd/d7/sites (7.x *<)$ drush core-tUndefined index: core-t complete.inc:308

    I think this is the same bug report as #95

Also, pcntl_fork would add a dependency on pcntl which is unfortunate. Note that simpletest is moving away form pcntl. See #771448: Use proc_open() instead of pcntl_fork() in simpletest.

Owen Barton’s picture

Thanks so much drzraf and Moshe for testing this out. I think I have a fix for 2 and #95 in my sandbox, as well as some argument completion code. I am not sure about filename completion - we could generate listings ourselves, but the bash compgen command works pretty well - I haven't worked out if it would be better to just call it directly (and return the results as an array) or return some special key from the hook that has the bash script call it directly with appropriate parameters/globs.

One thing I am wondering about is the hook - should we just use drush_command_invoke_all() which results in COMMANDFILE_COMMAND_complete() function names, or should we go with some code that generates function names in the same pattern as the *_validate hooks (i.e. drush_COMMANDFILE_COMMAND_complete(), but with duplicate words removed). Any thoughts here?

Moshe - were your comments on pcntl_fork intended for the runserver issue?

moshe weitzman’s picture

yes, that pcntl comment was meant for runserver. sorry.

i'm ok with either pattern for hook names.

Owen Barton’s picture

FileSize
31.63 KB

Updated patch includes the following changes:
- Fixed command matching (2 and #95) - "core-t" works as expected.
- Added argument completion examples for archive-dump, core-config and core-topic, as well as more docs on this in general.
- Tested that completion does work for the aliases we have used. I am not sure if it is best to complete on these aliases, or if we should just complete on "drush" by default and add a line to say that people should add "complete -o filenames -F _drush_completion MYALIAS" to their .bashrc if they want completion with an alias?
- Added a system for completion of filenames in arguments - commands can now pass back an optional set of patterns/flags and we use glob() to produce a list of files and/or directories that match. I have added an example for archive-restore. As part of this I have also added '-o filenames' to the complete command options, so that it doesn't add spaces when completing filenames. It is going to take a bit more work to handle quoting files/directories with spaces correctly - we need to research how other scripts manage this - but it seems useful enough as it is.
- Added tests for both regular and filename argument completion. Fixed a couple of tests (looks like we now create an automatic alias for test sites).
- Added a check that drush exists before enabling completion.
- Fixed an issue with overaggressive removal of arguments that have "drush" in them when we are cleaning up our argv (which was breaking core-config arguments), and also documented the argv we expect/return in different situations.
- Fixed a couple of typos in the docs...

Owen Barton’s picture

FileSize
31.73 KB

Adding a test for the "single command" case...

moshe weitzman’s picture

Status: Needs review » Needs work
Issue tags: +awesomesauce

The code is looking very clean. Love it. Found some minor nits below. Will do more testing soon.

+++ b/commands/core/drupal/site_install_6.inc
@@ -18,11 +18,19 @@ function drush_core_site_install_version($profile) {
+
+  // We need to bootstrap the database to be able to check the progress of the ¶
+  // install batch process since we're not duplicating the install process using
+  // drush_batch functions, but calling the process directly.
+  drush_bootstrap(DRUSH_BOOTSTRAP_DRUPAL_DATABASE);
+
   $phpcode = _drush_site_install6_cookies($profile, $cli_cookie). ' $_GET["op"]="start"; include("'. $drupal_root .'/install.php");';
   drush_shell_exec('php -r %s', $phpcode);
 
-  $phpcode = _drush_site_install6_cookies($profile, $cli_cookie). ' $_GET["op"]="do_nojs"; include("'. $drupal_root .'/install.php");';
-  drush_shell_exec('php -r %s', $phpcode);
+  while (!_drush_site_install6_batch_check()) {
+    $phpcode = _drush_site_install6_cookies($profile, $cli_cookie). ' $_GET["op"]="do_nojs"; include("'. $drupal_root .'/install.php");';
+    drush_shell_exec('php -r %s', $phpcode);
+  }
 

How is this related to completion?

+++ b/includes/command.inc
@@ -342,9 +342,8 @@ function drush_parse_args() {
+          // Raising errors for missing option values should be handled by the
+          // bootstrap or specific command, so we no longer do this here.

OK, but we definitely want to soon add the feature to error when an invalid option is passed.

+++ b/includes/complete.inc
@@ -0,0 +1,459 @@
+ * ¶

Lots of leading/trailing spaces here.

+++ b/includes/complete.inc
@@ -0,0 +1,459 @@
+ * For argument completions commandfiles can implement

Add comma after completions

+++ b/includes/complete.inc
@@ -0,0 +1,459 @@
+ * activities to retrieve the list of possible arguments, and also clear the
+ * cache when they know the list of arguments has likely changed.

Direct user to where he can learn more.

+++ b/tests/completeTest.php
@@ -0,0 +1,85 @@
+    // Create an additional alias for D7 site to test.
+    $aliases['complete-dev'] = array(
+      'root' => UNISH_SANDBOX . '/' . $docroot,
+      'uri' => $env,
+    );
+    $contents = $this->file_aliases($aliases);
+    $alias_path =  UNISH_SANDBOX . '/home/.drush/aliases.drushrc.php';
+    file_put_contents($alias_path, $contents);

Why do we mention D7? Can we not use the new site alias that setUpDrupal makes?

We now have proper isolation of tests such that nothing goes to globals paths like /home/.drush. In this case, just use the path UNISH_SANDBOX

+++ b/tests/completeTest.php
@@ -0,0 +1,85 @@
+    // We copy our test command into our dev site, so we have a difference we
+    // can detect for cache correctness. ¶
+    mkdir("$root/sites/$env/modules");
+    copy(dirname(__FILE__) . '/unit.drush.inc', "$root/sites/$env/modules/unit.drush.inc");

State that: "We cannot use --include since complete deliberately avoids drush command dispatch.

+++ b/tests/completeTest.php
@@ -0,0 +1,85 @@
+    mkdir("aardvark");
+    touch('aardwolf.tar.gz');

Code comment please.

+++ b/tests/completeTest.php
@@ -0,0 +1,85 @@
+    $debug_file = tempnam('/tmp', 'complete-debug');

Lets use UNISH_SANDBOX/complete-debug as file path. That gets cleaned up automatically.

+++ b/tests/completeTest.php
@@ -0,0 +1,85 @@
+    $exec = sprintf('%s complete --complete-debug %s 2> %s', UNISH_DRUSH, $command, $debug_file);

Needs unish_escapeshellarg()?

msonnabaum’s picture

I've pushed the site switching code Moshe mentioned in #90 here:

http://drupalcode.org/sandbox/msonnabaum/1076280.git/tree/refs/heads/per...

Super simple. Just site-get, site-set, and site-reset.

drzraf’s picture

About filename completion & co,
it happens when you know that, according to the previous argument.
(it is called $prev in existing completion).
Then you want to complete with filenames, dirnames, hostname, , ... (eg)

Without thinking further the following will _just-work_ (if drush complete can play with returned values) :

Let's store not only the available completion, but also the return value.
COMPREPLY=( $(drush complete "${COMP_WORDS[@]}" < /dev/null) ) ; ret=$?

Inside $(drush complete) we can test for whether filenames, or others ... are needed, then:

[[ $ret -eq 2 ]] && _filedir && return; # filename completion
[[ $ret -eq 3 ]] && _filedir -d && return; # dirname completion
# or even
[[ $ret -eq 4 ]] && _filedir ${COMPREPLY[@]}
# where COMPREPLY[0] may contain  '@(inc|php|module|install)'

But some helpers (which may or may not be needed by the drush completion)
will need to know about the current word (eg: _known_hosts_real "$cur" which completes with hostnames
matching the current argument "$cur", _filedir can take optional file extensions (see above), ...
Roughly, $cur is the bash-completion equivalent to bash ${COMP_WORDS[$COMP_CWORD]},
but it handles the cases where the cursor is not at the end of the line.

There is stuff about whitespaces, quotes and word boundaries inside _init_completion()
which is why it is always used at the beginning of existing completions.
(you may want $cur to be equal to "--myoption=value" or just "value", ...)

_SERVER[] contains every exported variable. But to avoid messing with user environment,
both bash $COMP_* and bash completions (cur prev words cword, ...) are local to the completion function.
I wouldn't advice to export them, but the "-d" flag of php may help here.

Also note that complete -o filenames is not advised
It's better to dynamically set the -o filenames (with the compopt bash built-in) when needed, like in :
[[ $ret -eq 2 ]] && compopt -o filenames && _filedir && return; # filename completion

hope this help
(the question is interesting but I don't know Drupal enough to know how drush may have very specific needs in
terms of filename completion, but personally I would rather try to use bash for filename completion when possible; if the resulting code isn't too ugly)

Also note that the bash "filename" completion is always available in bash with the following bindings: Alt + /
independently of the programmable completion.

jamonation’s picture

subscribing

Owen Barton’s picture

FileSize
35.18 KB

Updated patch should include all of Moshe's fixes, with 2 exceptions
- for "just use the path UNISH_SANDBOX" - it looks like it is already being used to me, so not sure what to change here.
- for "Needs unish_escapeshellarg()?" - we don't need this, the command to complete is normally not passed in escaped, so doing it here would make things less realistic.

I have also done a bunch of work on the file/directory completion, and a much happier with it now. Spaces are now handled (I didn't check other weirdly named files), and we no longer use -o filenames. We also condense full paths into just the basenames where that is possible. I reorganized the arguments array so we don't have the special 'files' array mixed in to all the values. This should also enable future refactoring to handle arguments with different values in the correct order (possibly with repeating arguments on the last one).

We are also now managing insertion of spaces following completions ourselves - this is necessary for correct file/directory completion, and also allows more possibilities down the road - for example if we extent our command array to differentiate "flag" options from options that expect a value, we are able to complete the former with a space, and the latter with an '='.

I also added a cache clear function, and added some tests for this, as well as a few other edge cases that came to mind.

Owen Barton’s picture

FileSize
35.24 KB

Fixes some warnings

Owen Barton’s picture

FileSize
35.64 KB

WIP patch that fixes some more warnings. Having some trouble with the tests though.

Owen Barton’s picture

Been bashing on this for a while and not making much progress - if anyone feels like taking a shot at making the tests pass feel free.

As far as I can tell it is a problem with the @dev alias getting picked up in the test environment (causing it to not find the unit-invoke command we have installed to that site) - the functionality itself (picking up commands in sites) works fine, and it also works if you replicate the test environment and run the same command manually. The $argv being passed in is identical, but I am not very familiar with the alias code so it's a pain to work through it (especially since I can't debug since it only reproduces inside the test environment).

moshe weitzman’s picture

I notice that verifyComplete() makes calls like: drush complete --complete-debug @dev uni. Notice that there is no site alias at the beginning, nor any --root,--uri options. So naturally unit-invoke is not getting picked up because we are not pointing at any drupal site. I think adding --root and --uri to these verifyComplete executions will fix the problem.

I'm not too clear on what the goal of the tests is. Some more high level docs (what, why, ...) would be great.

moshe weitzman’s picture

that last comment is wrong. lets discuss in irc or by phone.

Owen Barton’s picture

Moshe figured out the issue - we need to do some more environment setup for tests (no idea why they were working before though!). Attached patch just blindly fixes the tests, but the plan (patch on it's way) is to abstract out the "early call" functionality (and the bit of code we need from the bootstrap) into something reusable, for other systems that may need very low level control over the Drush environment/argv or bootstrap process (I can imagine locked command SSH key access could use this, for example).

Owen Barton’s picture

FileSize
35.71 KB
Owen Barton’s picture

Status: Needs work » Needs review
FileSize
41.7 KB

Attached patch includes "early" option described above, a simplified argv munging process, abstracting the core environment setup (ETC_PREFIX etc) for tests, a bunch of documentation and tests improvements.

moshe weitzman’s picture

Status: Needs review » Needs work

ok, we're there. feel free to commit this after considering the minor stuff below. lets then open an issue for how to guide users on installing this.

i think `drush5`, `drush6`, `drush7` might be bash aliases to `drush` that we should support

+++ b/drush.php
@@ -27,6 +28,23 @@ exit(drush_main());
+  // Process early global options such as --debug.

lets not use the word early here. initial?

+++ b/includes/complete.inc
@@ -0,0 +1,547 @@
+ * drush [word] : Output aliases, local sites and commands

aliases => shell aliases and what do we mean by 'local'?

+++ b/includes/complete.inc
@@ -0,0 +1,547 @@
+ * Because the purpose of autocompletion is to make command line work efficient,

efficiently

+++ b/includes/complete.inc
@@ -0,0 +1,547 @@
+ * clear the cache (or just the "arguments" cache for their command) when they

typo: they

+++ b/includes/complete.inc
@@ -0,0 +1,547 @@
+ * Determine context (is there a site-alias or command set, and are we trying

We re-use the word 'context' a couple times in a new way in this patch. Maybe say position'?

+++ b/includes/complete.inc
@@ -0,0 +1,547 @@
+  // Everything is cached per-site, except global options.

We might as well cache global options per site too. We almost let these vary per site since we have hook_drush_help_alter()

Owen Barton’s picture

Made alias, global option cache and documentation changes, then committed. Yay!!

Owen Barton’s picture

Status: Needs work » Fixed

Made alias, global option cache and documentation changes, then committed. Yay!!

drzraf’s picture

Is there a way to ensure that tab-completion will NOT run anything or modify the environment in any way ?
Eg: I installed drush_make then:
$ drush make E<tab> # (I want it to complete the filename: "EXAMPLE.make")
Build aborted. [ok] Make new site in the current directory? (y/n):
It means that some work has been attempted on press which is not right (nor safe).

PS:
A space is not appended when a word is completed (because of -o nospace):
$ drush ma<tab>
completes to "drush make" without space appended

Owen Barton’s picture

Thanks for checking this out!

I just committed a fix and a test to ensure complete stops execution early even when there are no valid completions.

The behavior for "drush ma" not adding a space is correct - while "make" is a valid command, so are "make-generate" and "make-test" - the problem was that completion was not listing these additional commands, since a valid command was detected. Note that a space is inserted when a single valid command (or any other word) is reached, so "-o nospace" is not the problem here. I committed a fix and a test for this also.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

greg.1.anderson’s picture

Add "Needs change notification".

drzraf’s picture

Priority: Normal » Major
Status: Closed (fixed) » Active

as per comment 118:
drush scr ins<TAB> The drush command 'drush scr ins' could not be found. [error]

Unix users are worried when pressing TAB brings random failures meaning unknown/unexpected processes are happening under the hood.
(drush version 4.6-dev)

Owen Barton’s picture

I can't reproduce the issue described in #122 (the #118 issue was fixed in #119). Can you please try with the latest 5.x-dev release and confirm if this is still an issue?

greg.1.anderson’s picture

Confirmed that the test in #122 works correctly on my system (no error messages on ) on current master.

drzraf’s picture

Priority: Major » Normal
Status: Active » Closed (fixed)

you're right, I wasn't on the right branch (there are so many :), the next time I will double-check that).
I'm now on master and these errors do not happen here.
let's just mention that:
drush @b<TAB> does not complete the site name
and
drush @beta scr ins<TAB> won't complete the filename (but that's not really a problem for me with Alt+/)
(I would also vote for not offering both the command name and its abbreviated version, if possible; like wd-del and wd-delete)

(sorry for the noise)

moshe weitzman’s picture

Issue tags: -Needs change record

Added Change notice

dwatts3624’s picture

Looks like there's been some great progress here. I'm very interested in this script and would love to join the effort but, even after skimming through the entire thread, it's unclear which code and patches I should be running to get up to the latest version. Has this been moved to a project or somewhere in GitHub? Apologies if I'm completely missing something.

Thanks!

DW

moshe weitzman’s picture

We have bash completion in drush5. see README.

sirtet’s picture

Version: » 7.x-5.x-dev

Can't see anything about autocomplete in README:
https://github.com/drush-ops/drush/blob/5.x/README.md

greg.1.anderson’s picture

This is in the "POST INSTALL" section, item #2. You should upgrade to Drush 6, though; autocomplete is the same in both versions.