Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.Comprehensive test suites allow developers to refractor, patch, and produce new features without needing to worry about who's toes they're stepping on. Drupal has an awesome qa group and people that have the uniquely twisted mind for these activities but they may not have the statistics they need to direct their effort. Using automated tools that we can rerun often you will create the statistics that will allow the testers and new code contributors to focus on the modules that need the most help.
Deliverables:
A script that will check out the cvs version of each of the top 40 drupal 7 modules and generate statistics about how many assertions are present in the test suite compared to how many lines of code there are in the module. Since we don't actually know what the normal ratio is for drupal yet your work will be the foundation of goal setting for new projects.
Contact:
Chuck vose (vosechu)
| Comment | File | Size | Author |
|---|---|---|---|
| #27 | test results gci.txt | 8.66 KB | vosechu |
Comments
Comment #1
dalinI like this.
Comment #2
cwgordon7 CreditAttribution: cwgordon7 commentedThis sounds nice. I wonder whether there would be some better metrics we could use for gaging code coverage - I know the code coverage module integrates with xdebug and actually calculates the ratio of code "covered" by the test vs. the code not covered. This approach has advantages in it's simplicity, however, and could end up being very useful. I think we should move forward with this proposal. The one thing that this is missing is a list of resources - links to pages about CVS and about SimpleTest assertions would be particularly useful. Remember, your target audience is a new participant, who may not know what these things are! Other than that, great task idea, this can be marked RTBC.
Comment #3
vosechu CreditAttribution: vosechu commentedRegarding the coverage testing from xdebug, I think it would be fun but does that push this into a harder task category? If not I think that Phing could be just the thing for this (fourth link).
Resources:
Getting started with SimpleTest
http://drupal.org/node/291740
SimpleTest project page
http://drupal.org/project/simpletest
SimpleTest mainpage (you don't need to download this to get started with SimpleTest in D7 but it is a good resource)
http://www.simpletest.org/
Automatic code coverage analysis with Phing
http://devzone.zend.com/article/2955
Comment #4
cwgordon7 CreditAttribution: cwgordon7 commentedI don't understand what you're suggesting with Phing. Our test suite is not written in Phing, so that won't work. We do have a Drupal code coverage module, however it would be more complicated to set up a script to automatically run that. Instead, a rudimentary analysis of number of assertions, as originally suggested, would be simpler, though perhaps less accurate at gaging the true state of test coverage in the top modules. Let's leave it with the original suggestion of counting the lines of code in .test files vs. the number of lines of code in the module. Note that a list of the top Drupal modules can be found out http://drupal.org/project/usage.
Comment #5
vosechu CreditAttribution: vosechu commentedThanks cwgordon7, I agree. The phing suggestion was merely because Phing has a nice interface for xdebug's code-coverage output but I agree that for a starter project this is already hard enough.
Comment #6
cwgordon7 CreditAttribution: cwgordon7 commentedLooks good, then, let's leave the original proposal as is.
Comment #7
cwgordon7 CreditAttribution: cwgordon7 commentedHi vosechu,
Please make an account on the google application site and apply to be a mentor for Drupal, so I can add this task. It needs a mentor listed on the google system. You can apply here: http://socghop.appspot.com/gci/mentor/request/google/gci2010/drupal. Thanks!
Comment #8
Anonymous (not verified) CreditAttribution: Anonymous commentedHi vosechu,
Could you also estimate the time, in hours, that it would take a student to complete this task once it is claimed? This estimate is required to input the task into the google system.
Thanks!
Comment #9
vosechu CreditAttribution: vosechu commentedAssuming they're only working like 2 hours a night this might stretch out to 10 days (20 hours). We'll see if I'm wildly off on that though.
Comment #10
dawehnerhttp://www.google-melange.com/gci/task/show/google/gci2010/drupal/t12904...
Comment #11
webchickComment #12
aantn CreditAttribution: aantn commentedHey everyone,
I took the Google Code-In task. I'm storing the code for this in a GitHub repository. https://github.com/aantn/Drupal-Test-Statistics/blob/master/drupal_cover...
I've also attached the most recent version. Please review.
Comment #13
aantn CreditAttribution: aantn commentedI'm having some trouble attaching the file. Please just use the version in git.
Comment #14
vosechu CreditAttribution: vosechu commentedI tried to run but ran into a problem loading argparse, since I'm not a python person I don't know what python's package manager is but if I have some time tonight I'll try to figure it out.
It would be really preferable to be able to just query the top 40, I know the scrape isn't that clean but the module list does change and I would like to be able to actually run it on all the modules when I have the cpu time. I accept that this does technically cover what was requested though, so as long as someone can get past the argparse error and do a review I think this is just a nice to have.
I also see that you used CVS checkouts which I really like for internal use but I think that the average user is going to want to use the latest release. Do other people thing this is valuable or is it beyond the scope of the project? I know Drush will download the latest release of something but I don't know if that really satisfies the problem at hand since the public at large isn't going to have Drush and I understand that the logic is fairly complex to figure out what the latest release is.
I've also come to realize that my code review is going to be lacking since I don't know Python idioms which is a little sad. I feel like you would have gotten a more meaningful review had you done it in PHP but then you'd have to have done it in PHP which could be contrary to learning good code. :P So my apologies that I won't have a better review of the Python code.
Side note: I really like that you did global stats as well. Also the line about modules which have no tests.
Comment #15
vosechu CreditAttribution: vosechu commentedFigured out the package manager thing, if you wouldn't mind putting a note about using easy_install to get argparse that would be cool. i'm running through the suite now and will comment more after it finishes.
Comment #16
vosechu CreditAttribution: vosechu commentedSo it looks like something is off about the CVS checkout because I'm fairly confident that Views both has tests and has some files. Looking at the checkout directory though Views is lacking files which contributed to it saying that it had no tests at all. Seems like there are a lot of modules that this is happening to. Looking at the CVS checkout code I'm not seeing anything immediately wrong with the command though. Can you take a look at this?
For reference (though I suspect you already found it) here is the Views CVS checkout instructions: http://drupal.org/node/38878/cvs-instructions/HEAD
Comment #17
cwgordon7 CreditAttribution: cwgordon7 commentedvosechu, I think the latest development (HEAD) version is fine, it's both easier to do and more relevant to developers, and test statistics don't matter much to regular users who would prefer stable release tarballs. We can just leave that as is.
I think this is looking good, after the feedback from vosechu is addressed this should be close to finished. We should try to get someone really good with python to look at it, though, if we can.
Nice work, keep it up!
Comment #18
aantn CreditAttribution: aantn commentedI pushed a new version which adds an appropriate error message if argparse isn't installed.
The problem that Chuck raised doesn't seem to be specific to the views module - views works fine over here, but I'm missing one or two other modules instead. I'm working on finding a fix.
Comment #19
aantn CreditAttribution: aantn commentedAlso, as cwgordon7 said, I think that it makes sense to use HEAD. If you want to know what the standard ratio of asserts/code is right NOW, then you don't want to look at a release that occurred three months ago.
Comment #20
dawehnerBut HEAD doesn't make sense always. Many maintainers only work in the branches DRUPAL-6--1 and completely ignore other branches.
Comment #21
aantn CreditAttribution: aantn commentedChuck, the views module really doesn't have any tests.
Can you please delete all of the checkout directories and try again? If you interrupted the checkout in the middle, then something might have been corrupted.
I also made the following changes:
1) Fixed the checkout for the Zen module (which needs to be handled specially because it's a theme)
2) Removed the rules module, because it has no HEAD
Comment #22
aantn CreditAttribution: aantn commenteddereine, I'm not that familiar with Drupal's development conventions or with CVS, but wouldn't most code be committed to HEAD, even if it's also tagged for a specific version (or pulled into a stable branch)?
Comment #23
dalinYeah I personally never use HEAD on my modules. For example
http://drupalcode.org/viewvc/drupal/contributions/modules/cd_sunlight/
There are four branches:
DRUPAL-5
DRUPAL-6--1
DRUPAL-6--2
DRUPAL-6--3
DRUPAL-7--1
The current stable release for Drupal 6 is using the DRUPAL-6--3 branch. But a module may use HEAD instead of branches.
So I think you'll need to do this in a couple steps:
1) Determine the module's currently recommended release for Drupal 6 (and your code should be able to easily switch to Drupal 7 as well) from the project page:
http://drupal.org/project/cd_sunlight
2) From the release's "Notes" page you can determine what CVS tag the release was generated from:
http://drupal.org/node/966318
3) From that tag you may just be able to reverse engineer the name of the branch (strip the fourth dash and everything following). If that branch exists, then use it. Otherwise the project is probably just using HEAD.
Comment #24
mr.baileysPrevious comments are correct and relying on HEAD to be the most recent version for a module is flaky. You can take a look at HEAD branch and how to use it effectively to get an idea of the different strategies contributors can use with regards to using HEAD.
I think dalin's approach (using the recommended release) is a good one, but instead of deriving this from the project page you can use the release information available as xml on drupal.org (for example http://updates.drupal.org/release-history/views/7.x).
Comment #25
webchickIf you're in an existing Drupal installation, drush dl [projectname] automatically gets you the right version. If we're only doing the top 40, it might be easier to script this by hand.
Comment #26
aantn CreditAttribution: aantn commentedPlease check the latest version.
Comment #27
vosechu CreditAttribution: vosechu commentedThis looks awesome now and I appreciate the hooks you added in to ensure that local developers can test their own modules without needing CVS. You're really a fantastic programmer. This is absolutely worth the 4 points. If you want to take it farther there are things we can do but if you would like to leave it with us that's fine too. I'm going to mark the task as accepted.
Attached is the preliminary results of the test run I just did. Fantastic work aantn!
Comment #28
webchickNice job, aatn!
To wrap some context around a few of these numbers:
This one shows so astronomically high because it's also testing stuff in /includes (and occasionally system.module as well)
These are ok; those tests are now in core with better than 1.0 ratio :)
Those are also "ok"; we don't currently have a way to test JavaScript.
Toolbar, though, should definitely have at least some test coverage. That must've totally gotten missed this release. D'oh.
vosechu, maybe a useful follow-up GCI task would be to use http://drupal.org/project/code_coverage or some similar xdebug hack to determine test coverage in terms of the actual lines of code hit during test runs (it can produce output similar to http://acquia.com/files/test-results/modules_aggregator_aggregator.parse...). The last time http://acquia.com/files/test-results/index.html was calculated was August 2009, and it'd be interesting to see how we've changed since then.
Comment #29
aantn CreditAttribution: aantn commentedChuck, thanks for mentoring.
I'd love to work on the script more. Perhaps I could do a follow-up task or two?
Comment #30
vosechu CreditAttribution: vosechu commentedThat sounds like a great idea aantn, I'll come up with some followups but if you have some ideas about what you would like to work on let me know over email or irc.
I totally agree that getting actual code coverage analysis would be fun and shouldn't be too hard. If you're feeling really ambitious we should work towards something in PHP so that we can wrap it into PIFR or some automated process like the usage stats but I don't even know where to begin with that. I'll write more tasks about this tonight and you can tell me which you like. But feel free to propose ideas and I'll formalize them.