Replace shell script and wget's with librarian project for cookbooks

Here HERE!

Here is a working example of our vagrant-drupal using librarian.

https://github.com/xforty/vagrant-drupal

Note, librarian doesn't on windows using RubyInstaller. We had to use cygwin. We logged an issue here:

https://github.com/applicationsonline/librarian/issues/34

Otherwise librarian works really well.

@pearcec: @walkah and I are working on a similar project at Myplanet Digital to help us emulate acquia's servers for local development, among other things. will definitely release when we can, but here are some of our thoughts just to toss them out there:

- Using a Bundler Gemfile to ensure gem's are all at the correct version. Will also help you control bug reports, as anyone outside your gem specs will have had to intentionally alter versions :)

- 100% cred to @walkah on this side of things, but we're taking the tack that our vagrant env should use the same cookbooks and configs that we use to configure integration and testing servers (the ones that emulate acquia). As such, we're minimizing how much we rely on "special" cookbooks (a la vagrant_main) with tons of code custom to the vagrant environment. So if we're using capistrano to deploy to our integration envs, we're also using it to deploy to vagrant. This ensures that we're repeating that same process as much as possible, and if something is improved in one place, it's improved elsewhere (DRY-like mentality, I suppose). So the take-home is that vagrant does the setup op the environment, but deployment itself uses other tools (although vagrant can help set up the CONFIG files from templates for those deployment tools).

- Using a config.ini file and the inifile gem to parse it. The idea is that, while we encourage people to peak into the Vagrantfile, no one should need to edit it directly for the commonly tweaked stuff. So things that would go in here would be github username and project name, so that the repo can be cloned to the right place for capistrano to deploy it.

- We're aiming to make the box workable in both an install-profile and site-as-repo context. So we should be able to point it to a git repo that cap will set up to deploy, and it will detect whether the repo is one or the other, and deploy accordingly (a deploy:drush_make task runs before if it's an install profile, etc). There should also be an option to pull in db-dumps from a remote location (say, a public dropbox url), so that teams can go that route if they need to

- Experimenting with booting a windows VM as well on the multi-VM host-only network, for cross-browser testing

- Getting squid cache working in the VM for when d.o isn't reliable (happened just today and slowed us down). Maybe having a rake task to allow devs to easily ping d.o maybe 20 times to ensure it's consistently available before running vagrant destroy, so people have an easy way to check whether they should hold off destroying their VM (and it's cache).

- Getting dnsmasq running inside the VM, so the dev can just point to that local nameserver, and any domains needed can be available without having to edit the host file

- Potentially using something like soloist to get all the host config down to one step (rvm, bundler, maybe squid on host, etc)
https://github.com/mkocher/soloist

So in theory, the setup could get down to something as simple as:
$ git clone blah
$ gem install soloist
$ # edit config.ini
$ soloist
$ vagrant up
# dev environment is good to go :)

Any feedback would be totally appreciated. We'll figure out sharing as soon as we have a minimum viable product that works :)

@pearcec: @walkah and I are working on a similar project at Myplanet Digital to help us emulate acquia's servers for local development, among other things. will definitely release when we can, but here are some of our thoughts just to toss them out there:

- Using a Bundler Gemfile to ensure gem's are all at the correct version. Will also help you control bug reports, as anyone outside your gem specs will have had to intentionally alter versions :)

We too are interested in pinning version. I am not sure bundler is the correct approach. Wouldn't you need to do the following?

bundle exec vagrant

We are looking at using rvm and gemsets to pin gem. Not sure how this works or if it is possible. But I like the idea. If not we will likely fall back to bundler.

https://github.com/xforty/vagrant-drupal/issues/7

- 100% cred to @walkah on this side of things, but we're taking the tack that our vagrant env should use the same cookbooks and configs that we use to configure integration and testing servers (the ones that emulate acquia). As such, we're minimizing how much we rely on "special" cookbooks (a la vagrant_main) with tons of code custom to the vagrant environment. So if we're using capistrano to deploy to our integration envs, we're also using it to deploy to vagrant. This ensures that we're repeating that same process as much as possible, and if something is improved in one place, it's improved elsewhere (DRY-like mentality, I suppose). So the take-home is that vagrant does the setup op the environment, but deployment itself uses other tools (although vagrant can help set up the CONFIG files from templates for those deployment tools).

I couldn't agree with you more on this. I wish someone had a capistrano process for deploying with drush makefiles. I haven't found one yet. Have you?

- Using a config.ini file and the inifile gem to parse it. The idea is that, while we encourage people to peak into the Vagrantfile, no one should need to edit it directly for the commonly tweaked stuff. So things that would go in here would be github username and project name, so that the repo can be cloned to the right place for capistrano to deploy it.

That is an interesting idea. We more or less took the approach we expect people to clone our project them push it to their own repo. At which point they are using that version. IF they want to upgrade they can but it might cause problems with version compatibility.

Perhaps as vagrant settles into 1.0 and the concepts around vagrant-drupal project mature that could be useful.

- We're aiming to make the box workable in both an install-profile and site-as-repo context. So we should be able to point it to a git repo that cap will set up to deploy, and it will detect whether the repo is one or the other, and deploy accordingly (a deploy:drush_make task runs before if it's an install profile, etc). There should also be an option to pull in db-dumps from a remote location (say, a public dropbox url), so that teams can go that route if they need to

I think we conquered handling context by not trying to automate what are calling a "run time decision". Get the environment up so you can get to work. We have new site starts and we are using drush site-archive (limited success). And just heavily document use of drush command line tools. Such as site-install and site-aliases.

- Experimenting with booting a windows VM as well on the multi-VM host-only network, for cross-browser testing

Here is our experience thus far:

https://github.com/xforty/vagrant-drupal/wiki/HowTo:--install-on-windows

- Getting dnsmasq running inside the VM, so the dev can just point to that local nameserver, and any domains needed can be available without having to edit the host file

- Potentially using something like soloist to get all the host config down to one step (rvm, bundler, maybe squid on host, etc)
https://github.com/mkocher/soloist

So in theory, the setup could get down to something as simple as:
$ git clone blah
$ gem install soloist
$ # edit config.ini
$ soloist
$ vagrant up
# dev environment is good to go :)

Any feedback would be totally appreciated. We'll figure out sharing as soon as we have a minimum viable product that works :)

Please post what you have. A marketplace of ideas on this will be great.

Oh hey, one last thing (more on-topic :P )

Librarian would seem to solve the issue of apache-licensed cookbooks being stored on d.o GPLv2 infra

@pearcec Oh man Christian, I honestly have no excuse for NOT replying. Sorry about that man -- I just came back to post a link to what we have internally so far, and I see that I'd left you hanging.

Anyhow, what we've got so far is here:
https://github.com/myplanetdigital/ariadne

Firstly, I really want to apologize for not working more closely. I've been working on this internally, and as I've said elsewhere, we're scratching our own itch at the moment, and it might be pretty niche. We're totally interested in merging efforts and working more closely once we've hashed a minimum viable product out for internal use. I understand that people operating in cowboy fashion like this isn't usually a good thing, but we have a need to move at a decent clip :)

Also, in regards to you capistrano-drupal question, this conversation is going on on github:
https://github.com/previousnext/capistrano-drupal/issues/5

I'll catch up on the rest of your post and reply appropriately!

We too are interested in pinning version. I am not sure bundler is the correct approach. Wouldn't you need to do the following?
bundle exec vagrant
We are looking at using rvm and gemsets to pin gem. Not sure how this works or if it is possible. But I like the idea. If not we will likely fall back to bundler.

As I'm just learning, using gemsets and bundler are a bit redundant, and gemset are built more as generic sandboxes than as a good way to pin versions. That's definitely bundlers territory. There are a few ways to negate the "bundle exec" bit, and we're trying to figure out the best way now.

• RVM binstubs
• zsh bundler plugin (could probably put it in .rvmrc
• A simpler function in rvmrc to negate the need for bundle exec in the project dir with the Gemfile

That is an interesting idea. We more or less took the approach we expect people to clone our project them push it to their own repo. At which point they are using that version. IF they want to upgrade they can but it might cause problems with version compatibility.

Really want to keep lock-in between releases to an absolute minimum. Ideally, the config file should be the only thing that needs editing. Also central is the idea that someone can fairly easily pull out the cookbooks used in the vagrant env and merge it into their own infrastructure rather easily. I spent quite some time trying to use the pennyworth continuous delivery sandbox, and realized that even if I did pull out the pieces I needed, it would be near-impossible to efficiently keep up with changes in the project so that they benefitted my own setup as well, if that makes sense. This was before the maintainer was using librarian, and so all the cookbooks were just kinda dropped into that project. It was really confusing, and I really don't want to put others in that situation where they detach themselves from future functionality.

I think we conquered handling context by not trying to automate what are calling a "run time decision".

I think I might be convinced that this is the right approach... Perhaps auto-detect is a poor approach. I guess first thing is I'd like to get cap working for deploys to the VM, and then we'll figure out if it still makes sense to automate -- maybe it won't :)

Experimenting with booting a windows VM..

Sorry, that was rather vague on my part. We're experimenting with running windows in a second VM, so that those of us on Macs (which is pretty much all of us internally), can boot up that second VM and view a page served by the linux server VM. Right now, this requires a big windows ISO, but hoping to give this a spin, so that it actually boots with vagrant as well: https://github.com/jedi4ever/veewee/tree/master/templates/windows-7-prem...

Anyhow, I know it f&$#-ton of info, but let me know if you have any thoughts. I'm trying to keep the readme up-to-date. I'm realizing that the big gaping hole is the project overview and goals at the top, but aside from that, if you know what it's supposed to be doing, it's hopefully decent enough. Definitely give me shit if it's out of order or makes too many assumptions on reader background :)

One last thought on this quote:

That is an interesting idea. We more or less took the approach we expect people to clone our project them push it to their own repo. At which point they are using that version. IF they want to upgrade they can but it might cause problems with version compatibility.

Internally, we're really trying to encourage a mentality where people destroy their vagrant boxes at the end of the day and rebuild each morning. The idea is that config's might be tweaked, or a feature might be added, and so you want the freshest without thinking. Just rebuild as you're making coffee in the morning.

So I imagine we'll also make use of drush-archive to compress the site into a tarball in a shared directory (which will persist between rebuilds), and then after rebuild, just explode it into a full site. So hopefully something like this could help make the whole destroy-rebuild thing as painless as possible :)

(Or maybe drush-archive wouldn't be needed if we're using capistrano... hmmm... *ponders*)

Anyhow, another reason to rebuild every day: if using the same cookbooks that are used in production, it helps put any changes through their paces. But there's a fine line, because you don't want to be guinea-pigging your changes on developers -- that's totally not cool. In practice, I need to be better at doing testing as I commit to the project, as too often I was causing regressions during development of v1 of our vagrant env, and so others were understandably reluctant to destroy and rebuild. Everyone need to have faith that when they blow their environment away, it'll come back brand-spankin new, plus maybe some improvements.

So yeah, I've just got really dead simple tests running with the "vagrant kick" gem, but I'd like to dig into full BDD testing at some point, so I can be more confident that I'm not breaking things as I work. There's some interesting stuff out there.

https://github.com/Atalanta/cucumber-chef
http://www.jedi.be/blog/2011/12/15/bdd-testing-with-vagrant/

or berkshelf?

True. Berkshelf is a little ahead, but librarian is a much better community player. Librarian a general, "bundler-builder" library for building dependency resolution systems. So librarian-chef, librarian-salt & librarian-puppet exist, and it will likely gain more steam because writing dependency resolution tools is hard.

Berkshelf developers, on the other hand, seem to eschew the idea of splitting itself out into smaller pieces that would be useful to others. I think Librarian has more legs, if you're evaluating long-term.

For example, the opscode guys (maintainers of chef) have built a test harness, test-kitchen, for testing infrastructure (cookbooks). They just today merged in a feature that allows multiple provisioners. (https://github.com/opscode/test-kitchen/pull/128) Since it's much more consistent to have a single stack of tools for testing either puppet or chef or whatever, I think the trend will eventually be toward favouring librarian, since it can be used in the toolchain for testing all provisioners.

Anyhow, just my two cents :)

-----------

Also, since I see there's talk above about pinning versions: https://github.com/fgrehm/vundler

I'm moving toward using the vagrant installer, creating all extra functionality as generalized plugins, and resolving the plugins via vundler. It's currently very simpler, but we're talking about incorporating bundler.