Lightweight built in http server for development

This fixes the redirect/logout issue, and changes the default port to 8888. I think moving the 3rd party includes to a separate directory is a good idea (we might be able to auto-bootstrap phpunit into there too).

Getting this to work with multisite is looking a bit tricky - if we set HTTP_HOST, we can get the right sites directory, but then all the site links are broken (i.e. pointing to http://sitename.com, when we need to go to http://localhost:8888). We could bootstrap Drupal ourselves, rather than running it as a cgi, but that makes things quite a bit more complex - I have experimented a bit with doing a drush redispatch or pcntl_fork (which could be super speedy in principle, since the parent could be pre-bootstrapped). Any ideas?

I hadn't seen that - it looks pretty cool too (and weird how they both came to exist within hours of each other!). It looks like it depend on lighttpd - I think one advantage of the httpserver code I am using is that it depends on nothing but a php that was built with --enable-sockets (which is very nearly all of them, as far as I can tell). I think serious developers (who need to play test out .htaccess rules, SSL stuff etc), will probably want to stick with Apache and co. at least some of the time, but I think this style of running sites for development can be really handy for new developers and people who switch sites/environments a lot. Together with sqlite in Drupal 7, this opens the door to a working standalone Drupal development bundle that does not need root access (no system-wide daemons or configuration) and has no dependencies at all other than php itself - a DAMP for the Linux/VPS/OS X world, so to speak. I think something like that has real potential for getting newcomers rolling fast (who knows, we might snag the occasional Django/RoR dev too!).

This looks great - thanks for taking a look at this. I used auto_prepend_file() in Drubuntu (to provide a "global" $conf array) - very sneaky including it here as a php-cgi parameter! I am not sure we need to write out the file for each site though - we could add "$cgi_env['QUERY_STRING'] .= '&runserver_base_url=http://localhost:' . $port"; (or some other php accessible cgi variable of our choosing - ideally one that is not sourced from external data) to the routing code and then provide a static audo_prepended file that contains "$base_url = $_GET['runserver_base_url'];".

Other notes - I like the REMOTE_ADDR improvement, and I also noticed you fixed querystring handling (another todo I forgot to note here) - yay! I think I was using a more specific value than DRUSH_DRUPAL_SITE_ROOT though, which could avoid the basename. I think it is a context called DRUSH_DRUPAL_SITE or something similar (can check environment.inc).

Wow - that's pretty incredible that it runs on windows. Yay!

Updated patch including znerol's fixes, and implementing the $base_url fix via an auto_prepend and a CGI environment variable. Apart from more tests, I think the only remaining todo is to add code to automatically fetch the httpserver.inc files - perhaps in a dedicated vendor drop directory as Mark suggested.

I was wondering about the location of a /lib directory - one possibility is to put it in the users ~/.drush directory, rather than the Drush root or include directory. This would make Drush upgrades much simpler, and also simplify sitewide/distro installations (where the user wouldn't have write permissions to the Drush directory) , but I am concerned it may also be confusing for users having scattered files. We could also have some kind of drushrc style fallback system where we write wherever we have access and check multiple places when trying to include library files - not sure that is really necessary though. Any thoughts?

That's a good point - we could get around the conflicting libs by versioning the library directories - i.e. ~/.drush/lib/Console_Table-1.1.4/, but I agree that we should probably just go with /lib in the Drush root for the time being.

In terms of committing the code, it is against Drupal.org Git rules to commit 3rd party php code libraries - even if they are GPL licensed - See http://drupal.org/node/422996.

Given that there is not really much code in the current command and the way we interact with the php http server is pretty specific, I am not sure there is much benefit in adding an abstraction layer here. I think for real newbies the pure php server is about as easy as it can get - and folks with enough experience to want a ngnix/httpd/lighttpd will be OK dl'ing a separate commandfile for this. Perhaps it could hook into the same command-name as this somehow, or just use a derivative (runserver-ngnix, for example).

For reference, looks like the httpserver code has moved into it's own repo at https://github.com/youngj/httpserver, which is helpful for us.

Actually there is a sneaky way to inject $conf pre-bootstrap I discovered for Drubuntu - I'll try and roll a patch soon that includes this and the /inc stuff.

PHP needs to be compiled with --enable-sockets - I'll add a check for this.

Goodness - look at this: https://github.com/youngj/httpserver/commit/9fa27f503c79308dedd5eb4667b3...

no longer requires php sockets extension
Jesse Young (author)
16 minutes ago

and also this (which might resolve the drupal_http_request_fails issue - although I have a workaround for that already, which we still need to Windows) - https://github.com/youngj/httpserver/commit/fe3924526c653b580e4d9c465a4a...

allow HTTPResponse objects to specify content as a stream, instead of loading the entire response into memory as a string. allows multiple connections to process responses concurrently (although windows will block on php requests)
Jesse Young (author)
about 4 hours ago

Jesse - are you reading this thread somehow, or just trying to prove synchronicity theories ;)

It does still need php 5.3 (pretty reasonable, given what it is doing) - so I will keep the check for that. Patch on it's way.

@adunar - happy you found us - as you can see, we are big fans of your work here :)

@all - have done quite a bit of work on this tonight - very excited about how this is shaping up.

Attempt at summary of the changes below:

- Check for and create (if needed the) lib directory on bootstrap. I also added an option for the lib directory and .git.ignore to cover /lib in the default location.

- Moved table.inc to /lib - this involved quite a bit of refactoring of the table function (it was getting a silly number of nested ifs) so it can use a standard download function it shares with pm release history, and also move over the table.inc file from includes. Thinking about upgrades (and potentially multiple drush versions sharing a systemwide lib) I put table.inc in a versioned subdirectory, and also kept the capital T, so it matches the tarball better.

- I added code to make a git clone of httpserver and checkout a specified revision. I figure git is getting pretty much ubiquitous, and managing upgrades is much easier than with tarballs (and hence we don't need to version the /lib subdirectory for httpserver. Given the pace it is moving, we may want to upgrade fairly frequently until we release Drush 5.0.

- The lib code could probably use some more abstraction if we start adding more libraries, but I didn't want to go overboard given that we only have 2 right now. If we add more we may also want to add a dedicated "drush lib-install" type command, rather than lazy loading as we do now, to make lives easier for distro maintainers and sysadmins who want Drush non-writable.

- I added an attribution for the httpserver library - probably not technically needed since we don't distribute, but I would like to acknowledge Jesse and Envaya's great work here.

- I added arguments for address and port to bind to ($addr was just added to httpserver). I used the "addr:port" format, since I think that is pretty intuitive, is the same as Django runserver, and means you can easily have an optional address (port only) without needing to switch the args the wrong way.

- I also added an option for the php-cgi, since on some systems it might live somewhere else, or not be on the users path etc. You can add params in here too.

- I added an argument to allow you to inject $conf overrides into the running site. This adds the $conf['drupal_http_request_fails'] = FALSE; by default, but you could extend it pretty easily (in a drushrc/alias, not on the command line, since it is a key-value array) to fix up variables you like for development, for example disabling css aggregation.

- I added validation for php 5.3 and findable php-cgi.

- Our class is now in a separate file, since we have a configurable (and hence conditionally included) httpserver.php that it depends on.

Before I forget, I thought it would be really interesting (for Drush, Drupal and the httpserver project!) to see how simpletest runs on this platform - it would be a good way to dig up some edge cases in the 3 projects.

We could also continue the experiment I started with bootstrapping Drupal directly in Drush (not as a CGI) calling menu_execute_active_handler(); and feeding the response and headers into the response object. This should be reasonably possible with Drupal 7 static resets, and may have some interesting performance use cases. When I tried the main problem I had was duplicate headers on the second request, reset wasn't working for whatever reason. Probably many other issues too :)

Yeah - I am having second thoughts about the git checkout also. It shouldn't make any difference for people who checkout Drush via git, I think - git just ignores it and Drush will update it to the right revision (I think you are right a merge would be preferable too). However, I don't think git is quite as ubiquitous as we would like and I think a tarball method may be necessary, especially since the point of this is to minimize dependencies needed to run Drupal. I would kind of like to have some kind of git option too though - if nothing else it makes sending patches upstream so much easier. One idea that would keep the code simple would be to use a git submodule for this (and recommend people use --recursive when git cloning drush), and then fallback to a tarball if the submodule directory is not present.

You mean download/extract the tarball, or fetching files individually? Do you have any thoughts about version numbers in directory names or no (considering updates etc)?

Tarball sounds good to me - we already have the code to do that in the wget package handler - I can just pull it out to general purpose function.

In terms of top level command v.s. core, I don't have any strong opinions. the "core" group is certainly getting pretty crowded, but on the other hand a group of 1 seems a bit odd too.

Updated patch that uses tarballs for both libraries - I think this is the way to go. I abstracted out the code from the wget handler into 2 fairly low level functions, and added a higher level "library getter" function that both libraries can use. Tested each of the download methods (added a flag so curl follows redirects), also dl with wget, non-writable /lib and a few other scenarios.

I didn't reorganize the core command grouping yet - seems like that might be a good thing to explore at our code sprint...anyone got index cards :)

Attached patch resolves conflicts with HEAD, including some bits of refactoring/simplification of the download and tarball functions that made sense as I worked through it. I tested the pm/wget and caching functionality as well as the library fetch function, and both seem to be working as they should. I moved the runserver command to it's own directory - it is a tad unusual as the command name is the same as the commandfile group name - it seems to work though, not sure if it is confusing from a user point of view. I have bumped the httpserver library to 354b9142.

Fixes an issue with --drupal-project-rename for dl.

After some more testing I committed this. Leaving open, since I think this needs some tests (especially the download/tar/lib functions, since they have several consumers.

Fixing categories.

Also, I posted a proof of concept based on the ideas in #5 at #1200738: Command to get started with Drupal with zero configuration.

I have been working on making the Drupal core tests pass, since that is a really exhaustive test for this command (not that we shouldn't have some basic tests ourselves of course). I just committed one fix to runserver that makes cron.php and index.php (when called directly) work correctly. I patched another issue with httpserver which is at https://github.com/youngj/httpserver/issues/1. I need to let all the tests run through again, but it's looking pretty good.

I am still interested in figuring out if we can bootstrap Drupal from Drush (rather than via php-cgi) and call it within DrupalServer->route_request() - probably calling a subclassed HTTPResponse that adds a get_drupal_response() method. The main complexity is passing headers back and forth, and ensuring we have a clean (enough) environment for subsequent requests (we don't want to send double cookie headers etc). This could be via a pctl_fork, or via some kind of command that works through invoke_process, perhaps passing the page and headers via JSON (no need to parse to HTTP and back, since we own both ends). If anyone wants to experiment with this some more, please feel free :)

Note for reference that there are a few major improvements to runserver at #1200738: Command to get started with Drupal with zero configuration (that should help make this more testable) as well as an experimental non php-cgi page delivery approach at #1248030: Alternative(s) to php-cgi with runserver