Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.Our 50 testbots for Drupalcon Portland were quite a stretch, and pointed out how much more we could automate this process.
We need to do a number of things manually when creating a testbot and we could be doing all those things much better, even when building from a predefined AMI at AWS.
Our current process for spinning up a pile of testbots is to create multiple spot requests on AWS, then configure them manually and connect them to qa.drupal.org manually. It's painful for no good reason.
So we could do these things:
- Automatically set the hostname + /etc/hosts using AWS techniques. (The hostname has to be set up right before puppet can work). This can be done using AWS's metadata stuff,
- Automatically run puppet
- Puppet could check out a named release of PIFR provided in the userdata
- At that point, we still need 1) a testbot configured on the qa side (with an appropriate URL) 2) The key from qa set (using drush) on the testbot, 3) the testbot number from qa set using drush into the title on the testbot.
I don't really know how to do the last item: Mass-creating testbots on qa, and then automatically configuring the data from them on the actual testbots.
We could possibly create a pifr drush command (or web service) which would allow the mass creation, then take action based on that data. Since the qa testbot data would include the URL of the new testbot, the manipulation of the testbot with the key and title would then be possible...
| Comment | File | Size | Author |
|---|---|---|---|
| #1 | 20130525_155742.jpg | 1.3 MB | rfay |
| #1 | 20130525_155826.jpg | 1.43 MB | rfay |
Comments
Comment #1
rfayWhiteboard notes from our discussion at Portland Saturday Sprint.
In the conversation were Nick_vh, wamilton, rcross, with jthorson and rfay
Comment #2
rfayOur decision was that there are two tasks to be undertaken here.
1. @Nick_vh: Build a new xmlrpc service with will allow the testbot to request its identity (testbot id + client key) from qa.d.o dispatcher. It will identify itself using a key provided on the testbot by other means. Why don't we call the service "testbot_register": testbot_register(myhostname.example.com, primary_ip_address, authkey) will result in (testbot_id, client_key). If the hostname and IP provided already exist, give the key for them that already exists (although this would require a new column I think…). The service on the QA side with 1) Create a new testbot record using the hostname/url provided, 2) request-enable the testbot, 3) Return the (testbot_id, client_key). 3) Send an email to maintainers with the testbot build information.
2. rfay: Build a script that will 1) Let the testbot build proper /etc/hostname and /etc/hosts on boot, set the hostname 2) Run puppetd (which must do what is required by Nick_vh's xmlrpc
3. rfay: Make puppet load the database if it is not already populated.
I went ahead and did 2 and 3 this week - did quite a number of commits on drupaltestbot and drupaltestbot-puppet. We can now spin up any number of testbots automatically, and the only remaining thing is to reduce the config time associated with each. (At this point, the config consists just of putting a key on the testbot and creating the testbot config on qa.
In your court, @nick_vh - Happy to help any way. Instructions on the simplified routine are at https://drupal.org/node/1748618 - feel free to edit and improve.
Comment #3
Nick_vhAck! Let's get this going :)
Comment #4
rfayHey, while you're at it, could you make a deregistration command, so we could spin them down just as fast? We could then spin up 100 and then bring them down a couple of hours later...
Comment #5
isntall CreditAttribution: isntall at Drupal Association commentedComment #6
MixologicThis is definitley solved. Testbots are automatically provisioned now. :)