"Conduit" and "Worker" represent potential candidates for the next generation of qa.d.o and the testbot infrastructure. Based on the software behind Jimmy's ReviewDriven site, they provide a number of potential benefits over the existing infrastructure; and form the basis of a framework which should be more understandable, maintainable, and extensible than the existing solution.

We'd like an environment where we can centralize development efforts, as well as to facilitate demonstrations of the architecture and its potential features to the greater Drupal community.

We would be interested in the creation of a dev site to host the "conduit" (i.e. qa.d.o) portion of the architecture, and a VM which would represent the "worker" (i.e. testbot) side of things. The worker piece includes a daemonized process, and communication between the two leverages a 'Services' rest interface, so it would be preferable to have the worker as a standalone VM as opposed to another site on the dev environment.

The worker component doesn't need access from the internet, but the ability for developers to browse to the worker site would be beneficial. Both sites should be built with secure/restricted access, as well as isolated from communicating with other key infrastructure; to help guard against any potential security concerns introduced during development. Feel free to ping me for more details behind any of these requirements.

The worker component also requires installation of the inotify extension.

Jimmy has drush make files available for both conduit and worker servers, which can be provided once the infrastructure is ready for site build.

Comments

boombatower’s picture

boombatower CreditAttribution: boombatower commented

bump. can we get this going?

boombatower’s picture

boombatower CreditAttribution: boombatower commented

Any update, last I heard that was work being done on a new VM. Any sort of time-frame or updates? If it's going to be a while perhaps we should use a temporary old VM or AWS or something?

jthorson’s picture

jthorson CreditAttribution: jthorson commented

We're going to be contacting OSL regarding a new VM for worker, and seeing if there's room on dev for the conduit server.

jthorson’s picture

jthorson CreditAttribution: jthorson commented

Also related: #1677432: A drupal.org development site for pift rewrite for d7 and conduit

jthorson’s picture

jthorson CreditAttribution: jthorson commented

I'll put in the OSU request to get our limits increased, to create RAM/CPU space for the worker server. Once they're increased, I can create a new VM on ganeti.

We still need to chase a vhost (on dev?) for the conduit side of things.

jthorson’s picture

jthorson CreditAttribution: jthorson commented

Worker VMs have been created on OSUOSL, and Worker sites installed at drupalworkerdev2.osusol.test and drupalworker3.osuosl.test.

We still need a D7 dev site for Conduit, after which this can be closed.

nnewton’s picture

nnewton CreditAttribution: nnewton commented

One confusing thing. I heard today that the D7 site needed a VM. Is there any reason this couldn't go on stagingvm or devvm?

-N

jthorson’s picture

jthorson CreditAttribution: jthorson commented

The workers needed a seperate VM, as they're running a deamon process. This is done.

I think it makes sense for the 'conduit' piece (think qa.d.o) to go on dev ... we'll be making REST calls between a D7 d.o dev site and the conduit server, but being on the same host shouldn't really affect things.

drumm’s picture

drumm
he/him
Location NY, US
CreditAttribution: drumm commented
Status: Active » Postponed

Dev QA sites now seem to be working, #1227778: Set up dev sites for qa.drupal.org. General info on our dev sites is at http://drupal.org/node/1018084.

Once that's confirmed good, I'll make a dev site for this. For D7, you can blow away all the files in the dev site and replace them with D7 and the new modules. The DB is populated, so you can test an upgrade, or just drop all the tables & start over.

When you are ready to move toward production, bzr+ssh://util.drupal.org/bzr/qa.drupal.org-7/ is ready for additional modules and such. I'll be making sure that gets deployed for staging #1559658: Get staging sites for subsites on Drupal 7 working.

jthorson’s picture

jthorson CreditAttribution: jthorson commented
Status: Postponed » Active

This will have absolutely zero in common with the existing qa.d.o ... it's a ground-up redesign and rebuild. The only reason I mention qa.d.o is to give people a frame of reference to explain what the 'Conduit' project is.

The plan is that this would exist in parallel with the existing solution; so that qa.d.o can provide a stable testing layer for the D8 push while this is in development/initial deployment. Thus, there is no upgrade path that needs to be verified.

All we need is a directory and vhost entry. Building it as qa.d.o and then wiping it out sounds like a whole bunch of extra/unnecessary work; when instead we could probably get away with running:

drush make --yes https://raw.github.com/boombatower/drupalorg_qa/7.x-1.x/drupal.make

I'd also hope we can quit bumping this to postponed. It's been over a month since it was opened, and the only progress we've made (ie. the worker VMs) is because I sent the OSU quota request and built the VMs myself. I've got under 3 weeks remaining that I can put towards working on this, and it's pretty crucial for my core convo in Munich that we've had time to work out any kinks. I'm not really looking to saddle the infra team with much in the way of getting this running ... I really just wanted advice as to where it should go. Point us at a directory, give us a virtualhost entry, and we'll take it from there (i.e. Jimmy and I can do the install and any related troubleshooting that's needed).

jthorson’s picture

jthorson CreditAttribution: jthorson commented

From an .htaccess authentication perspective, I believe the worker session logic may be able to work around it ... and if not, this is likely something we should add.

In any case, it looks like there are only two paths that the workers will communicate with:

i) item/claim
ii) item/result

There will be more in the future for the d.o side of the interaction, but that code has not been written yet.

boombatower’s picture

boombatower CreditAttribution: boombatower commented

It would be better if we add a rule to the .htaccess file instead of having to make an exception on the worker. The rules jthorson described should be all that is needed, but they will be prefixed with api/1.0/ etc where the version may change. A rule like api/* would probably be better.

drumm’s picture

drumm
he/him
Location NY, US
CreditAttribution: drumm commented

Dev sites for QA on D7 are working now too. I wanted to get D6 working as a prerequisite, we should have them all working consistently for both 6 and 7. This one is named http://conduit-qa_7.redesign.devdrupal.org/.

All dev sites now allow passwordless connections from our IP ranges and for /sites/default/files and /api.

boombatower’s picture

boombatower CreditAttribution: boombatower commented
Status: Active » Fixed

Seems like this is completed. I got the workers configured and pointed against http://conduit-qa_7.redesign.devdrupal.org/.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.