shortly, we'll have the newest version of the project issue tracker running on drupal.org, and issue followups will then be true comments. this provides a great opportunity for integration with the Drupaltestbed project, to include patch test results directly in the comments where the patches are attached.

i think it's best to start with a very simple implementation -- the test results are appended to the body of the comment itself. later we can do something fancier, like putting pass/fail graphics right next to the file itself, but this will be an excellent start.

should be relatively simple to implement by adding an XML-RPC callback to project issue which allows editing of comments. the call can simply be a user/pass to validate editing rights, then an array of data that the callback function uses to perform the edit on the comment in question.

Comments

Rok Žlender’s picture

Rok Žlender commented

Great true comments.
We will also be able to act on new followup/comment by triggering a call to test server to test a patch. Same way now the email parsing code does.

robin monks’s picture

robin monks commented

I see three steps:

1) Projectissue sends new issue or comment to testing
2) Testing processes
3) Testing sends results back to projectissue

Which means we'll need both sides to act as both a XML-RPC client, and server.

Robin

hunmonk’s picture

hunmonk commented

setting up project issue w/ an XML-RPC server component is very straightforward, as the testing site simply calls the server methods as needed.

sending the posted comment via XML-RPC is a bit more complex in terms of setup. do we do it on-the-fly, or via cron? since i want a generally usable solution, it's not acceptable to hack in direct calls to the testing site, so do we make the calls happen with a hook? do we drop the calls into a special .inc file?

thoughts please :)

hunmonk’s picture

hunmonk commented

actually, after further thought, i'd rather only put the server-side stuff in project issue module.

to pass the issue node/comment info from d.o to the test site, i believe the cleanest method would be another module, which would work like this:

  1. it would keep track of the last issue nodes/comments it's processed.
  2. on a cron run, it would pull all issue nodes/comments since the last processed, and send those to the test site via XML-RPC. while cron doesn't provide real-time processing, it ensures that the issues/comments are fully formed (i don't know of a reliable way to make sure you have the fully formed object during an insert cycle, as it's still in process)
  3. test site receives the data, processes it, and calls the XML-RPC method in project issue as necessary to update the issue nodes/comments
dww’s picture

dww
we/he/they
commented

I'm not thrilled about an XML-RPC interface that allows remote sites (even if they authenticate with user/pass) to edit comments at will. I'd prefer a system where there was another module that maintained its own table for test results, XML-RPC was used for the test server to insert data into this server module's table, and this server module would then use the presence of data in the table associated with a given patch to render additional text and/or graphics next to that patch when the comment is displayed.

That said, this all sounds very cool, and it'll be a killer feature on drupal.org when it's ready. ;) There are also a bunch of interesting possibilities for this kind of thing outside the realm of d.o...

dww’s picture

dww
we/he/they
commented

hunmonk thinks my approach is too complicated and wants more reasons behind what i'm talking about... instead of explaining in IRC, i'll just type them up here:

A) I'd rather the original comment was preserved, as written, in the DB. That's part of the whole philosophy of how Drupal handles data. The user's original data is sacred, but it can be filtered and altered in various ways between the DB and being rendered on the screen.

B) I'm assuming these are regression tests, and therefore, the underlying code can change, causing a patch to no longer apply. If you directly edit the original comment, instead of maintaining your own test metadata, you've now got a vastly more complicated problem of editing the test results to modify them from "applied cleanly, all tests passed" to "fails to apply".

C) I'd prefer modifications of issues to be as structured as possible. From a distributed system perspective, each entity that operates on someone else's data should do so in as limited a way as possible. This protects you from bugs and/or malice. So, if the test server can freely edit all comments on drupal.org, we now have to be excruciatingly careful about all aspects of the test server and its security. If all it can do is update a table with records about test results, the worst that can happen if someone compromises the test server is bogus test results. Aside from malice and security compromises, a more structured, limited design from the beginning will protect d.o's issue data from bugs in the test server.

D) I don't care about an extra JOIN when rendering an issue. If the DB server can't handle it, we can (and should) investigate caching, but we shouldn't hold-back functionality or design decisions solely because we're afraid of d.o's DB infrastructure.

hunmonk’s picture

hunmonk commented

@dww: nicely explained. i'm convinced. i take it this means you'll be taking the lead on writing that portion of the functionality?? ::)

hunmonk’s picture

hunmonk commented

following a heavy round of strategic design w/ dww, i've completed a working model for this feature. i won't go into detail here about it's functionality, as i'll be releasing it as a project in the next few days. in short, there are a pair of modules, one sits on the server with project module, one on the file testing server (they can be the same server, but they don't have to be). file data and test results are passed back and forth via XML-RPC.

although i'm sure we'll want more involved logic for selecting/re-selecting patches for testing in the future, i've simply started out by sending all posted files via cron, and they are tested only once.

you can see the displayed results in action at http://project.drupal.org/node/180968

you can play with this yourself to see how it works, just be aware that after you add a file, the site has to pick it up via a cron run for testing, so you might want to hit http://project.drupal.org/cron.php after you post your file(s) to speed things up a bit. for now, just post text files please, and if you want them to be marked as failing the file tests, simply include the word 'fail' somewhere in the file.

hunmonk’s picture

hunmonk commented

fyi, i've added a simple mailer to the codebase. users may now opt-in at project/issues/subscribe-mail to receive email notification when one of their patches fails testing.

hunmonk’s picture

hunmonk commented

i've thoroughly commented the modules involved here, and also abstracted the testing functions to support calling rok's test functions and receiving the replies. i think we're getting close to a release here.

hunmonk’s picture

hunmonk commented

added update/delete support for users/issues/followups, implemented a simple regex to check for valid patch files to pass to the test server, and added more robust error handling for the XML-RPC calls. the only thing missing now for a basic deployment is hooking in rok's testing stuff, which should be fairly easy to do. i have the code deployed on http://project.drupal.org, so i suggest that we use that environment to hammer out integration with test.drupal.org ASAP.

since the code is now fairly mature and well tested, i've created a project for it, so let's continue any further discussions about this in the issue queue of the new project :)

http://drupal.org/project/project_issue_file_test

hunmonk’s picture

hunmonk commented

ifac is nearing deployment -- no more major code changes are coming. the pift suite has been fully updated and tested, and is working beautifully on project.drupal.org. the only missing piece is to tie it into the testing code rok has. this is out of my hands, as i'll need rok to do most of that.

i await further instructions :)

hunmonk’s picture

hunmonk commented

the pift architecture has been successfully deployed on testing.drupal.org. right now it's just running it's default dummy tests, and submitting it's results to project.drupal.org, for the Drupal project only. things will start working for real as soon as we get the real tests hooked in.

Amazon’s picture

Amazon commented

Chad, should this issue be moved to fixed or closed?

Kieran

hunmonk’s picture

hunmonk commented
Status: Active » Fixed

sure, we can mark fixed. it works, it just needs to be turned on when we're ready.

Anonymous’s picture

Anonymous (not verified) commented
Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for two weeks with no activity.