Closed (fixed)
Project:
Project Issue File Review
Version:
6.x-2.x-dev
Component:
User interface
Priority:
Normal
Category:
Feature request
Assigned:
Unassigned
Reporter:
Created:
16 Nov 2011 at 18:14 UTC
Updated:
10 Mar 2012 at 08:40 UTC
I find the "Review log" extremely difficult to read.
On two separate projects I get different results locally than from qa.d.o. I've tried for 10 minutes to find out the hash codes of the versions of Drupal and all involved contribs that were used to run the test, but I've failed.
It would be very helpful to have a table with these hash codes in one place, so that we can recreate the exact configuration locally and hopefully be able to track down the issues.
Comments
Comment #1
rfayI don't believe that checkout is ever done on the basis of hash codes, but always on a branch or tag.
So you'll see in the log "git checkout... 7.x-1.x" or "git checkout ... 7.x-1.0-rc1"
The branch identifiers will be the same if you've done a pull; the tag identifiers should always be the same.
Does that make sense?
Comment #2
salvisYes, sort of, but it's a terrible pain to extract that information. I'm taking http://qa.drupal.org/pifr/test/193554 (forum_access 7.x-1.x) as an example.
It says:
[09:48:03] Command [git clone 'git://git.drupal.org/project/drupal.git' 'checkout' --reference /var/cache/git/reference 2>&1] succeeded[09:48:03] Command [git --work-tree='checkout' --git-dir='checkout/.git' checkout '7.x' 2>&1] succeeded[09:48:03] Main branch [7.x] checkout [complete].Ok, so I guess I have a copy of Drupal as it presented itself at 09:48:03 in some timezone on some date. No hash is given, I don't really know what I have here.
Immediately after the quoted line above I get
[09:48:08] Command [git clone 'git://git.drupal.org/project/acl.git'So we're turning to acl now, which is a dependency of forum_access.
Some 30 lines down it says
[09:48:08] Dependency branch [7.x-1.0-rc1] checkout [complete].So, that's the last tag on the 7.x-1.x branch. What if I had required the 7.x-2.x branch? Or the -dev version?
Immediately afterwards I get:
[09:48:13] Command [git clone 'git://git.drupal.org/project/chain_menu_access.git'some 20 lines down:
HEAD is now at 682fd5e... #1186208 by mrfelton, chx: Fix callback inheritance by removing the empty(['page callback']). (Theand another 10 lines down:
[09:48:13] Dependency branch [7.x-1.0-beta4] checkout [complete].This is nuts! 682fd5e/7.x-1.0-beta4 is not the latest tag — it was followed by 7.x-1.0 which I used to produce a release on August 8! See http://drupalcode.org/project/chain_menu_access.git/shortlog/refs/heads/...
Why is the bot not using the latest, recommended release?
On with
[09:48:18] Command [git clone 'git://git.drupal.org/project/forum_access.git'No "HEAD is now at ..." line this time!
[09:48:18] Dependency branch [7.x-1.x] checkout [complete].After the experience above, I don't trust this thing anymore. We have no idea what it's testing, only that it did it at 09:48:18...
So, in conclusion, I would like to have a human-readable table with Drupal, the subject under test, and all the dependencies, giving the tags and hashes of each that were used for running the test. Without that information it's difficult to exactly reproduce a test.
BTW, did you notice how painful reading this comment is? And I left out 95% of the lines in the Review log.
Comment #3
jthorson commentedWell ... a couple of points ...
1) The review log was initially designed as a debug tool for troubleshooting, but outside of the testing infrastructure maintainers, I suspect that *very few* people have ever attempted to go line-by-line through the log themselves (as direct end-user consumption was not its original intent).
2) Initial support for contrib projects with dependencies was launched *just slightly* over a month ago, and the feature is still in its infancy ... there will be most certainly be bugs to uncover and additional efficiencies which could be gained.
Your comments are valid, and your suggestions are good ones ... though to be totally honest, the tone of your last comment comes across a little on the critical side, which detracts from the intrinsic value of the ideas contained within.
As for 'trust', this is just another tool made available for the community to leverage as seen fit ... if it doesn't meet your needs, then you may need to fall back to manually executing run-tests.sh for now ... in the meantime, suggestions for improvement are definitely welcome; and patches even more so. :)
I'm moving the issue over to the 'Project Issue File Review' issue queue, as the PIFR project is where the changes required to enable this feature request would be implemented.
Comment #4
rfayYeah, the reality is that until a month ago, projects with dependencies were not even possible to test! We are still so proud of that accomplishment (which took years) that we haven't started to process the ways to better communicate what dependencies are chosen and how to present debug information.
That said, we're glad that you have become one of the few who knows how to read the debug log!
Comment #5
salvisI'm sorry, this was not intended. I've stated in more than one place how great this tool is, and I'll gladly say it here again. About 200 hours of work have gone into my Forum Access tests, and being able to execute these automatically is a major turning point! I've been hoping to see this for a long time!
It makes it all the more frustrating that out of the two projects with thorough tests, both behave differently on qa.d.o than on my local machine. Not knowing which version of the source is being used is a major obstacle to analyzing the discrepancies.
Originally, I wrote I'd have to come back to this issue in two days, but just before hitting [Save] I bit the bullet, copied the log out into my word processor, applied various colors, and finally distilled it down to the very problems. Combined with the experience of Randy being able to extract the relevant line on multiple occasions, I was pretty frustrated (sorry to use that word a second time) at 3am, about having such a great tool and being incapable of using it effectively.
Comment #6
jthorson commentedCertainly understood ... and don't get me wrong, having a detailed report is always prefereable to a non-descript 'the log sucks' ... thanks for taking the time for such an in-depth analysis! :)
Comment #7
jthorson commentedSituation may be improved with the introduction of #1347594: Remove verbose logging on successful 'exec' calls in the next PIFR release.
Comment #8
salvisThank you for working on this!