Rework the SimpleTest results interface and clean-up of backend code

cwgordon7 - April 23, 2008 - 04:15
Project:Drupal
Version:7.x-dev
Component:simpletest.module
Category:task
Priority:normal
Assigned:boombatower
Status:patch (code needs review)
Description

There can be a lot of messages in a SimpleTest Test Case, and these messages can be hard to sort through. The attached patch adds a javascript-based filtering form which lets you filter by message type (pass, fail, and/or exception) as well as the group that the pass/fail belongs to. Filtering is done on a per-test-case basis.

This patch also has the side effect of making it more awesome than it already was.

AttachmentSize
simpletest_message_filtering_02.patch6.47 KB

#1

cwgordon7 - April 23, 2008 - 04:50

As per comments by dmitrig01, removed some stupid code, and made the search filter as-you-type.

AttachmentSize
simpletest_message_filtering_03.patch6.69 KB

#2

kourge - April 23, 2008 - 05:54

I reviewed the JavaScript code over IRC with cwgordon7 and posted the revised version of the main function.

#3

Dries - April 25, 2008 - 18:58

I guess the reviewed Javascript needs to be integrated and the patch rerolled?

#4

cwgordon7 - April 26, 2008 - 20:16
Status:patch (code needs review)» patch (code needs work)

Yes, that is correct.

#5

cwgordon7 - April 27, 2008 - 20:08
Status:patch (code needs work)» patch (code needs review)

Ok, here is a new patch with reworked javascript and increased ui awesomeness.

AttachmentSize
simpletest_message_filtering_04.patch7.71 KB

#6

Dries - April 28, 2008 - 17:53

I'm not sure the filtering should happen on a per-group basis. I'd be inclined to make it a global toggle.

In general, I find the output to be quite messy. I'm not, at all, a big fan of the nested fieldsets. I find it very tedious to open and close them. I'd much rather have a flat page or single-level fieldsets with global filter options.

All in all, I think we're on the right track with this filter mechanism, but I'm not convinced we should sprinkle every fieldset with it.

#7

Dries - April 28, 2008 - 17:54

On a related not, it is not clear why we put square brackets around the category names.

#8

boombatower - April 28, 2008 - 20:48

@Dries: Agree with the brackets. All that was added at the sprint, I'm not sure who added it either. :)

#9

cwgordon7 - April 28, 2008 - 22:23

The brackets weren't "added"- when the $group stuff was created, they were moved over for consistency with the old messages of form "[browser] GET to node/135". I think they should probably be removed.
The SimpleTest administration framework does need some work. But let's tackle this one problem at a time. Plan of attack:

1) Filtering form

Same as this patch, except with global toggle as suggested by Dries.

2) Minimal assertion information

For an overview, we should simply give minimal assertion information, such as maybe "pass/fail/exception" and the group type. It is possible we could arrange these into several columns so they would all fit on one page.

The cool part would be that these assertion-overviews would be anchors, and when you click on them, they expand to show more info about the assertion, including the assertion details, the line # at which the assertion occurred, and perhaps a snapshot of the current page in the SimpleTest browser for functional tests.

3) Move stuff out of fieldsets

We need a better format for viewing assertions than fieldsets. Ideally, a jQuery-tabbed interface would be nice, with one tab for each test, but that seems very far beyond my javascript capabilities at the moment.

Basically, just make the interface more awesome.

#10

boombatower - April 29, 2008 - 03:04

@cwgordon7: Wasn't trying to be condescending. When I was referring to adding I was talking about how it was separated out of message.

#11

cwgordon7 - April 29, 2008 - 22:02

Oh, got it. Ok.

#12

Dries - April 30, 2008 - 07:02

Gordon, Jimmy -- looks like a solid plan! Great.

#13

boombatower - May 16, 2008 - 06:53
Status:patch (code needs review)» patch (code needs work)

I think there are outstanding issues raised by Dries. Correct me if that is wrong.

#14

cwgordon7 - May 21, 2008 - 00:12
Status:patch (code needs work)» patch (code needs review)

Indeed, therefore a new patch. The new patch:

1) Fixes some stupid javascript.

2) Makes it a global toggle.

3) Removes the fieldsets if there are no results shown within them.

4) Removes the current working directory from the file results (so all paths are relative to the drupal install).

I know this solution isn't perfect— I'd love to have a completely awesomified simpletest ui— but I think it is a good start. Reviews welcome!

AttachmentSize
simpletest_message_filtering_05.patch7.6 KB

#15

cwgordon7 - May 21, 2008 - 00:27

A preview of the awesomeness

AttachmentSize
simpletest_awesomeness_00023.png166.45 KB

#16

kourge - May 21, 2008 - 05:12

I tidied up the JavaScript a bit, so here's a rerolled patch - it's almost exactly like the previous one, simpletest_message_filtering_05.patch, with the only difference being the JavaScript.

AttachmentSize
simpletest_message_filtering_06.patch7.98 KB

#17

Dries - May 21, 2008 - 09:45

I think this is starting to look good.

I'm confused though, what is 'Filter by category'? From the looks, I'd expect this to match any pattern. From the text, it sounds like it would only match categories. Is there a reason we don't match everything?

#18

cwgordon7 - May 22, 2008 - 22:43

Matching everything gives a lot of false positives, so I left it out. It's easily possible though, if that's decided as the right way to do it.

#19

cwgordon7 - June 12, 2008 - 18:30

..bump? Should I add descriptions to the filtering?

#20

Dries - June 12, 2008 - 18:53

I'd like to see us try and improve the usability a bit. As I said, I was reasonably confused by the label of the form. I'd recommend that we simply match everything.

#21

cwgordon7 - June 12, 2008 - 21:15

Ok, it now matches everything.

AttachmentSize
simpletest_message_filtering_07.patch8.2 KB

#22

pwolanin - June 13, 2008 - 19:57
Status:patch (code needs review)» patch (code needs work)

generally looks good.

however, Passes should be deselected by default

Also, a Group that has all passes should still show the basic info even when Passes is deselected, e.g.:

Book
1 test case complete:
97 passes, 0 fails and 0 exceptions.

#23

catch - June 13, 2008 - 19:58

I think having the pass output like that is important - otherwise people will think some tests didn't run at all (and with no failures you'd get no feedback whatsoever). Reassuring.

#24

cwgordon7 - June 13, 2008 - 20:00
Title:Add a filtering form to SimpleTest module messages» Rework the SimpleTest web user interface

Ok, yeah I'm changing this into a general UI-reworking issue.

#25

boombatower - June 16, 2008 - 21:54

Couple changes I think would be nice, otherwise looking nice:

  • Adding a fieldset around the filter options. That would be cleaner looking and help separate them from the results. Also I think it should be collapsed by default since many times looking at all the assertions is what is useful.
  • Can the event on the filter input field be changed from after every character typed to after a user stops typing (I know that isn't an event, but not sure how difficult it would be to implement a delay after last typed character?) or onblur or capture enter key press. As it is currently it is very slow when very many assertions are displayed on the page and it attempts to sort after every keystroke.
  • The simpletest.test needs to be updated since the [ ] were removed from the assertion types. (+1 for that change)
  • Having the default assertion type be set to the test name is somewhat long and odd. I think a better solution would be to set it to "Test" or something since the assertions are already in a fieldset with the tests name would be better.

#26

boombatower - June 16, 2008 - 22:00

This patch changes simpletest.test to work with new patch except for:

<?php
$this
->assertAssertion($this->pass, 'Other', 'Pass');
$this->assertAssertion($this->fail, 'Other', 'Fail');
?>

the "Other" types since that is being discussed and not yet confirmed either way.

AttachmentSize
simpletest.test.patch1.24 KB

#27

cwgordon7 - June 16, 2008 - 22:23

I was actually thinking of postponing this until the Batch API/DrupalTest/UI issue was sorted out.

#28

cwgordon7 - June 16, 2008 - 22:25
Status:patch (code needs work)» postponed

..

#29

cwgordon7 - June 24, 2008 - 22:28
Status:postponed» active

With that issue worked out, I'll now start from scratch with this...

#30

cwgordon7 - July 4, 2008 - 07:02
Status:active» patch (code needs review)

Here's an awesome patch.

AttachmentSize
awesome_ui.patch24.32 KB

#31

Dries - July 4, 2008 - 07:38

* I think we should get rid of all the green status messages. I know, that's a separate issue, or maybe it isn't as it could be Javascripted out.

* I like the 'Run more tests' link. It cleans up the UI and let me focus on the test results of this run.

* The filters don't seem to work though. It doesn't show the passed tests ... See screenshot.

AttachmentSize
testing.png497.2 KB

#32

Damien Tournoud - July 4, 2008 - 08:10

While we are at it, I would really like a "Rerun the same tests" button on the result page.

#33

boombatower - July 4, 2008 - 15:44

@Damien Tournoud: That would be great.

@Dries: Are you talking about the directory created messages?

#34

boombatower - July 4, 2008 - 15:45
Status:patch (code needs review)» patch (code needs work)

Based on #31 I believe this needs work, and #32 would be nice to incorporate.

#35

catch - July 4, 2008 - 16:05

Regarding the directory created messages, there's a generic patch for those, changing the dsm to a watchdog (which I'd support): #257537: file_check_directory is verbose on success. - Might as well fix across the board as special case it for tests.

#36

moshe weitzman - July 16, 2008 - 06:03

The directory issue hs been committed elsewhere. Just need to addres Dries' other points and Damien's as well.

#37

cwgordon7 - July 26, 2008 - 05:36
Status:patch (code needs work)» patch (code needs review)

Addressed the concerns in #30 and #31. This patch now provides a "Re-run same tests" button, and an awesomified user interface! :)

AttachmentSize
awesome_ui_02.patch27.19 KB

#38

cwgordon7 - July 26, 2008 - 05:58

Re-rolled as per discussion with dmitrig01 on IRC.

Also attached is a screenshot of the awesomeness. :)

AttachmentSize
awesome_ui.png79.28 KB
awesome_ui_03.patch27.63 KB

#39

boombatower - July 26, 2008 - 06:42

I much prefer the tables, but I like the bolding of non-framework assertions.

The re-run tests button is good, and I like that the entire list of tests isn't display, but a link to them instead.

I would strongly vote for tables to remain.

As long as this patch continues to provide a re-run tests button #286946: Remember the tests that were selected on the test results page is a duplicate.

#40

cwgordon7 - July 26, 2008 - 07:36

I would strongly vote for tables to remain.

I would strongly vote for the tables to go. They're ugly, they overflow off to the right of my screen, they're a rainbow of too-bright colors, the tableheader is annoying, and it makes me scroll down way too far.

But if enough people think we should keep the old ugly tables, I'll reroll.

#41

cwgordon7 - July 26, 2008 - 07:39

Oh, and forgot to mention, they take forever for firefox to load/parse when there are ~70 tables each with ~20-500 rows, and sometimes (though rarely) it crashes and/or refuses to parse any more table rows below a certain point. IE6 just crashes, and IE7 crashes half the time. Safari and Opera seem fine, but the tables are still just as ugly.

</myargumentsagainstuglytables>

#42

webchick - July 26, 2008 - 17:19

Going into this not having read the rest of the issue, nor reviewed the patch (will in a moment) but only clicked on the screenshot @#38 (this simulating a typical end-user), here are my thoughts:

  • Site title matches is bold. Ahhh! Scary! Does that mean it REALLY passed? Does it mean it failed? But if it failed, why is it green and not red?
  • Also, why does it list a bunch of sub-things related to it, which other tests don't?
  • Why is "Found assertion" such a big deal that it's bolded too?
  • Where do I find fails/exceptions in a huge bulleted list like this? Will they be grouped together at the top? Interspersed between?

Reviewing the patch now...

#43

webchick - July 26, 2008 - 17:29
Status:patch (code needs review)» patch (code needs work)

Er.

When I run this patch, check off a test and then click the "Run tests" button, I get a blank page with just the word 'HERE'.

#44

webchick - July 26, 2008 - 17:31

Hm. After removing + print "HERE"; from the patch, the results page still doesn't show me any results. I tested with Filter's tests, which have failures at the moment.

#45

boombatower - July 26, 2008 - 20:40

They're ugly, they overflow off to the right of my screen, they're a rainbow of too-bright colors, the tableheader is annoying, and it makes me scroll down way too far.

The same colors are used in new patch...and the same number of assertions are displayed with no way to collapse them....so how is the scrolling an issue?

As for the crashing when parsing the same number of assertions are displaying with new method...very similar data to parse. Could be related to system running it? I don't ever have firefox crash, but it does take a few moments longer to parse, but isn't that to be expected?

Table is a good format for displaying large amounts of data, columns give you a lot of power. As seen in similar applications throughout the computer world I think a table is the best choice. If the main argument against it is cosmetics that is not really a big deal, especially since bulleted list isn't any better (opinion). I think testing is mainly concerned with results (data) and table displays the data in an organized format.

#46

cwgordon7 - July 26, 2008 - 21:06

When I run this patch, check off a test and then click the "Run tests" button, I get a blank page with just the word 'HERE'.

UGH. Sorry... (but this is coming from the $this->banana person? ;) )

the results page still doesn't show me any results. I tested with Filter's tests, which have failures at the moment.

Odd. I'll look into it...

The same colors are used in new patch...and the same number of assertions are displayed with no way to collapse them....so how is the scrolling an issue?

Um, there is a way to collapse them. And the assertions now fit onto one compact line, rather than overflowing onto 2 or 3 lines in a table row.

Rerolling the patch after I figure out the results page problem.

#47

boombatower - July 27, 2008 - 04:46

Still not seeing all this overflow and such with tables. :) The main change appears to be the removal of much of the debug information such as file, line number, etc. Things that I think are very important.

#48

cwgordon7 - July 27, 2008 - 04:57

@#47 - Um, no. That information is still there. :P Did you even try the patch / look at the screenshot?

#49

cwgordon7 - July 27, 2008 - 05:13
Status:patch (code needs work)» patch (code needs review)

New patch features:

  • Darker colors for easier readibility.
  • Proper exception handling.
  • The standard watchdog icons for easier readibility.
  • Does not randomly shout "HERE" when you try to run tests.
  • Actually shows results.
AttachmentSize
awesome_ui_04.patch28.72 KB

#50

cwgordon7 - July 27, 2008 - 06:47

I actually decided I don't like the icons. New patch removes them.

Also, some javascript bugs, new patch removes those too.

AttachmentSize
awesome_ui_05.patch32.28 KB

#51

cwgordon7 - July 27, 2008 - 06:48

Oh, and also, filter tests had some weird html tags in them, now escaped for better UI.

#52

webchick - July 27, 2008 - 18:16
Status:patch (code needs review)» patch (code needs work)

Hrm.

1. I was just about to post that "No, it's still broken cos it's only showing me the number of test passes/fails" before I accidentally clicked on "Filter: 163 passes, 12 fails, and 0 exceptions" and saw that it was expandable. So we need some sort of UI thing to indicate "this will expand when you click it and there will be more stuff below it" since right now it looks like just any old bulleted list item. Maybe a "+" icon, or a ">" icon like we have on menu items?

2. I get that you're trying to compress the output, but it's annoying that there were test failures and I had to click twice to see what they were. I would prefer if test failures were auto-expanded so that I could immediately see them in front of me. This would get real tedious if I was truly doing test-driven development which would involve re-loading this page 50 times trying to figure out where my tests were still failing.

3. I still don't know why some of these are bolded and some aren't. It looks like a bug. I have a feeling it's on purpose, but if it is, it's not clear why "Parse simple www-string but not the end-of-sentence period." and below is bold, but "GET to http://localhost/head/node/1, response is 7660 bytes." and above is not. Are the bold ones assertions from within filter.test itself? If so, we should probably preface each line with some sort of indicator, like what filename the assertion was found in?

4. When I click away from this page and go do other stuff, then come back to the testing page because I want to test, it shows me the collapsed test results rather than the test selection form. That's very odd, and not what I would expect. We should clear the cookie or whatever's maintaining state if admin/build/testing is hit directly.

5. I love love love the "Re-run same tests" button. :D

6. When I unchecked the "Fails" button, suddenly the entire output of this page went away. That's not what I would expect. I would expect the "Filter: 163 passes, 12 fails, and 0 exceptions" and the "Core filters: 68 passes, 12 fails, and 0 exceptions" to stay there, but for the individual failure rows to disappear, so that only the passes remain. OTOH, when I uncheck "Passes" the page does do what I expect, event though I realize it's taken "Filter administration functionality: 95 passes, 0 fails, and 0 exceptions" out entirely. I guess the logic would be, "If unchecking one of the options results in an empty list, remove its parent. Otherwise, keep the parent in place, with the unchecked children removed."

I've tried to annotate a screenshot so this feedback is a bit more clear.

AttachmentSize
simpletest-ui.png195.8 KB

#53

boombatower - July 28, 2008 - 02:03

I'll try and be frank. I'm not trying to start a war, but from the screenshot (haven't gotten around to running patch) there is no line number or file. Not sure, but I would prefer if my comments weren't taken as an offense, as they seem to be. There are several things that I really like about the patch as I have mentioned above.

I'll take a look at the patch in more detail in a bit.

#54

cwgordon7 - July 28, 2008 - 02:16

No offense taken. :)

As you can see from the screenshot (http://drupal.org/files/issues/awesome_ui.png), the line, function, and file are still displayed. Not until you ask for more information, true - but I prefer not being overwhelmed with information, and would rather face that extra click to see the extra information on the assertion I'm concentrating on than have a harder-to-sort-through interface.

#55

webchick - July 28, 2008 - 02:21

Oh, sheesh. Again, I didn't realize those sub-items were clickable and revealed more data. We really need a different UI construct. :\

#56

cwgordon7 - July 28, 2008 - 03:15

Agreed. I'll collapsibilize it and make it look like a link.

#57

kourge - July 28, 2008 - 03:45

One thing about the patch: I'm now sure why exactly there are two functions, Drupal.behaviors.simpleTestResultsCollapse and Drupal.batchUpdate.simpleTestResultsCollapse, with exactly the same code.

If the same function needs to be under two namespaces, you can simply to this:

Drupal.behaviors.simpleTestResultsCollapse = Drupal.batchUpdate.simpleTestResultsCollapse = function() {
  $('li > span.simpletest-overview').click(function() {
    $(this).siblings().children('ul').toggle().toggleClass('simpletest-hidden');
  }).each(function() {
    $(this).siblings().children('ul').hide().addClass('simpletest-hidden');
  });
};

As an added bonus, I chained .each with .click. As far as I know, chaining isn't just pretty syntactic sugar; it should also reduce perf hits because you don't select the same set of elements twice or more times. (The last time I checked, jQuery doesn't cache recent element selections.)

I apologize for not being able to reroll this patch myself. I'm currently packing up for Canada, and packing mes bagages is a pain :(
Whistler should be fun, though.

#58

cwgordon7 - July 28, 2008 - 07:04
Status:patch (code needs work)» patch (code needs review)

New patch addresses #'s 52 through 57.

AttachmentSize
awesome_ui_06.patch34.91 KB

#59

Wim Leers - July 28, 2008 - 14:46

Subscribing.

I agree that the file names and line numbers should be hidden by default. That annoys me endlessly in simpletest for D6. On a sidenote, I think we also need a better interface for selecting tests. At least if the D7 UI is the same as D6's. I have to scroll way too much.

I'll try to find the time for a more in-depth review later.

#60

drewish - July 28, 2008 - 16:33

subscribing

#61

boombatower - July 28, 2008 - 20:58
Status:patch (code needs review)» patch (code needs work)

@Wim Leers: not the same interface for selecting tests. I felt the same way and change it a while back.

@cwgordon7: Ok the clicking is much more reasonable. Sorry about that the whole misunderstanding grew out of the screenshot. I agree with webchick that the interface isn't easy to figure out though.

#58:
1) When the test is running it provides links which interrupt the tests and cause them to fail when clicked. (intended?)
2) Nothing shows on the results page. Even when I check "Passes"

#62

cwgordon7 - July 28, 2008 - 21:14

#61 - 1) Interesting. Not intended. Can't seem to reproduce... can you describe what you did more specifically?

#61 - 2) Interesting. Also can't seem to reproduce. Do you mean an entirely blank page, or just no results?

#63

boombatower - July 28, 2008 - 22:47

1) On the page doing the batch process the results now form links, but when clicked they take me to the results page, which has problems as in #2, and the tests never finish running.
2) Page displays with checkboxes and (from picture) intended interface, but not assertions display. (cache issue with javascript maybe?)

#64

cwgordon7 - July 28, 2008 - 23:00

#63 - 1) There are no actual links (<a> tags) on that page, only pseudo-links that collapse/expand the results set.
#63 - 2) I have a feeling this could be solved when we solve #1?

Can anyone else confirm these issues? I cannot.

#65

cwgordon7 - July 29, 2008 - 05:32

Re-rolled for increased general awesomeness. Does not address #'s 61 and 63 as I cannot seem to reproduce.

AttachmentSize
awesome_ui_07.patch43.27 KB

#66

webchick - July 30, 2008 - 07:06

I tried several different devious ways to screw up SimpleTest and could not reproduce the items highlighted by Jimmy, either.

I can confirm that this new version addresses all of my previous concerns. A couple other things I found during this pass through:

1. The contrast between the green and red is poor, and I need to squint to tell the difference. The #2 Web Content Accessibility Guidelie is to NOT use colour as the only way of communicating information, for a variety of reasons. I suggested appending (pass) (FAIL) and (EXCEPTION) to the test results to meet this requirement.

2. There's a "check all" box at the *bottom* of the form, which I gather is for convenience, but breaks core standards. I'd remove it, leaving the one at the top like we have on admin/content/node and similar.

3. One kind of annoying bug is if you expand out the test results during the test run, the next test group that runs will sproing the thing back to the default collapsed view. But Charlie's informed me this requires changes to progress.js to fix, which is out of scope for this patch. Would be awesome to get it in later though... and maybe even the checkboxes on this screen, too! :)

#67

cwgordon7 - July 30, 2008 - 07:10
Status:patch (code needs work)» patch (code needs review)

"Charlie's informed me this requires changes to progress.js to fix" = "Charlie is guessing that in order to do this he has to go into progress.js, but it's dark and scary there." ;)

So #1 and #2 fixed. If someone else who's handier with javascript than me wants to tackle #3, by all means, go ahead. :)

New patch here for review.

AttachmentSize
awesome_ui_08.patch43.04 KB

#68

webchick - July 30, 2008 - 07:19
Status:patch (code needs review)» patch (reviewed & tested by the community)

Ok, that addresses all of my major points.

Tentatively marking RTBC.

#69

boombatower - July 30, 2008 - 19:56

I'd like to take a look at it. Will do in a bit.

#70

webchick - July 30, 2008 - 20:29
Status:patch (reviewed & tested by the community)» patch (code needs review)

CNR for Jimmy. :)

#71

boombatower - July 31, 2008 - 00:35
Status:patch (code needs review)» patch (code needs work)

I think the interface is much improved, but there a few small/trivial items that are inconsistent or I think should be changed.

  • "Re-run same tests" -> "Re-run tests" as "same" is somewhat redundant. - changed
  • Status of assertion should be more prominent. As in previous table layout it was easy to scan column and see the assertion's status. I moved it to the front of message. -changed
  • Consistent case on status. The pass was lower and the other two messages (fail, exception) were uppercase. That is somewhat inconsistent. If you are trying to emphasize that should be done in a different manor I think. Since you can also filter on the status it isn't necessary to make the non-pass messages different. - changed

The code has a number of very long lines of code that are hard to read. Also some places where the same condition is executed 5 times in the same set of code.

Specifically this line of code was written 5 times and uses embedded conditions in the shorthand format. Not easy to read.

<?php
($results[$class]['#fail'] ? 'simpletest-fail expanded' : ($results[$class]['#exception'] ? 'simpletest-exception expanded' : 'simpletest-pass collapsed'))
?>

I rewrote code to assign the result to a variable and use a switch statement to evaluate the condition. Fewer lines isn't always better, I think readability is more important.

Same issue with exception status. The extra long lines with embedded conditions are in a number of places and it would be preferable if the code could be simplified for readability.

In simpletest.test the function assertAssertion($message, $type) lost the $status attribute which somewhat removes the purpose of the test. The status message is displayed and could be parsed as before. (I'll leave that to you)

When the filter has removed all possible assertion from an item it would be preferable if an empty message displayed.

I have yet to review other areas, but I think these need to be addressed first. This is a major change and I think the code should be reviewed thoroughly.

AttachmentSize
awesome_ui.patch42 KB

#72

boombatower - August 28, 2008 - 21:05
Assigned to:cwgordon7» boombatower

Since this seems to have become neglected I've decided to finish it up.

First off is a re-roll which I am currently working on.

#73

boombatower - August 28, 2008 - 21:17
Status:patch (code needs work)» patch (code needs review)

After a bit of wrangling and cleanup I have a re-roll.

I will be checking it over again, run all tests against it, and post conclusions. In the mean time reviews are welcome.

AttachmentSize
awesome_ui.patch40.18 KB

#74

boombatower - August 28, 2008 - 21:30

Issue that seems to be present is the file display in not always correct.

For instance from the block test:

[PASS] Box successfully created.

    * Group: Block functionality
    * File: drupal_web_test_case.php
    * Function: testBox
    * Line: 285

The function is correct, but the line and file refer to the assertion call not the actual function.

#75

boombatower - August 28, 2008 - 22:00

Found a fix for that problem, was issue with backtrace traversing logic.

Working on removing some interested "debug" output.

#76

boombatower - August 28, 2008 - 22:31

Important changes (doesn't include general clean-up items):

  • Fixed backtrace traversing.
  • Remove custom_caller due to improved backtrace traversing.
  • Change 'caller' field to 'function' in db for consistency with use.
  • Lots of additional comments.

Working on further clean-up or related interface code.

AttachmentSize
awesome_ui.patch43.49 KB

#77

boombatower - August 29, 2008 - 00:32

Large number of changes locally, I have to go now, but I will patch later.

So don't make new patches wait for mine which has a large number of patches.

#78

boombatower - August 29, 2008 - 19:03

Important changes (doesn't include general clean-up items):

  • Re-moved rerun tests dependency on db results. It shouldn't be necessary and results no longer exists due to other patch.
  • Improved re-run tests to handle more then one test class.
  • Split up results and select tests code making it much much cleaner and easier to read.
  • Removed lots of left over code.
  • More comment goodness. (Lots!)
AttachmentSize
awesome_ui.patch50.35 KB

#79

boombatower - August 29, 2008 - 19:10

A few minor changes, I think this one is ready for thorough review.

AttachmentSize
awesome_ui.patch51.85 KB

#80

boombatower - August 29, 2008 - 19:11
Title:Rework the SimpleTest web user interface» Rework the SimpleTest web user interface and clean-up on backend code

#81

boombatower - August 29, 2008 - 19:12
Title:Rework the SimpleTest web user interface and clean-up on backend code» Rework the SimpleTest web user interface and clean-up of backend code

#82

cwgordon7 - August 30, 2008 - 08:48

The $custom_caller parameter has been lost, why? I think it is necessary in order to map exceptions to the actual function in which the php error occurred, rather than ->error()?

#83

boombatower - August 30, 2008 - 23:22

Hmm...I think I fixed the traceback logic so that it works properly and doesn't need that anymore.

If anyone sees any problems let me know.

#84

boombatower - September 4, 2008 - 00:27

still applies with offsets.

#85

Damien Tournoud - September 4, 2008 - 08:03

Applied the patch, reinstalled the environment.

Ran tests without any issue. The new interface is neat.

But: the checkbox on top of the test selection page has no apparent effect, and the "Re-run tests" button has no effect (it returns to the test selection page).

Two feature requests:

* I hate having to click to expand a test group when I selected only one test group. Could we make a special case for this.
* The expand/collapse status is lost when the results are refreshed during the test.

#86

Damien Tournoud - September 4, 2008 - 08:04
Status:patch (code needs review)» patch (code needs work)

I guess this is a small CNW.

#87

boombatower - September 4, 2008 - 17:39
Status:patch (code needs work)» patch (code needs review)

"But: the checkbox on top of the test selection page has no apparent effect, and the "Re-run tests" button has no effect (it returns to the test selection page)."

May need to clear your browser cache as the javascript files may not be updated, checkbox works for me.
Re-run tests works great for me...hmmm.

"I hate having to click to expand a test group when I selected only one test group. Could we make a special case for this." - not sure what you mean by that. (possibly follow up issue)
"The expand/collapse status is lost when the results are refreshed during the test." - is an issue to be solved later.

#88

boombatower - September 4, 2008 - 17:41

Had the clear test results code commented out.

AttachmentSize
awesome_ui.patch50.33 KB

#89

Dries - September 5, 2008 - 10:21

The new UI looks like a great improvement, unfortunately, I'm struggling with my test environment. See #304328: SimpleTest requires max_execution_time > 30 seconds.

My one comment is that people might not understand 'Clean environment'. I know I didn't when I first saw that button. It might be better to make that a 'Clean environment' checkbox with a little bit more explanation. By default, it should be checked.

#90

boombatower - September 5, 2008 - 17:13

Clean environment is done everytime the tests are run. The button is intended for use when the tests crash and you need to manually clean the environment.

The button used to have a fieldset around it explaining its use. As of the patch I started work on the fieldset was removed and the description was placed at the top of the page.

Should I move it back to the fieldset way?

#91

catch - September 5, 2008 - 17:33

Could we move 'clean environment' to cron instead of having it in the ui?

#92

boombatower - September 5, 2008 - 17:39

Let me clarify its use.

The equivalent of clean environment is run after the test suite has been run. So everytime you select some tests to run, when they finish the environment should be clean.

If the tests crash, usually when in development, the environment may not be cleaned up. So it is really a developer tool, but I guess SimpleTest itself is. The button just makes it easier to clean up the mess, since otherwise you have to manually do it all, which gets old really fast.

#93

boombatower - September 5, 2008 - 17:44

It appears that this patch causes certain tests to timeout. Like "Site-wide contact form." and possibly "drupal_http_request" for starters. Which may be [304328] and webchicks issue on #296310: TestingParty08: drupal_http_request redirects need a test

#94

boombatower - September 5, 2008 - 17:54

It appears that the contact one works by itself, just not in the batch I ran it with..hmm

But drupal_http_request dies with:

PHP Notice: Undefined variable: function in /home/jimmy/software/php/drupal-7/modules/simpletest/drupal_web_test_case.php on line 105, referer: http://drupal-7.dev.loc/batch?op=start&id=136
error
PHP Fatal error: Uncaught exception 'PDOException' with message 'INSERT INTO {simpletest} (test_id, test_class, status, message, function, line, file, message_group) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6, :db_insert_placeholder_7) - \nArray\n(\n [:db_insert_placeholder_0] => 7\n [:db_insert_placeholder_1] => DrupalHTTPRequestTestCase\n [:db_insert_placeholder_2] => exception\n [:db_insert_placeholder_3] => Undefined property: stdClass::$code\n [:db_insert_placeholder_4] => \n [:db_insert_placeholder_5] => \n [:db_insert_placeholder_6] => \n [:db_insert_placeholder_7] => Notice\n)\nSQLSTATE[23000]: Integrity constraint violation: 1048 Column 'function' cannot be null' in /home/jimmy/software/php/drupal-7/includes/database/database.inc:365\nStack trace:\n#0 /home/jimmy/software/php/drupal-7/includes/database/mysql/query.inc(29): DatabaseConnection->query('INSERT INTO in /home/jimmy/software/php/drupal-7/includes/database/database.inc on line 365, referer: http://drupal-7.dev.loc/batch?op=start&id=136

Appears to be issue with getAssertionCall...working on it.

#95

boombatower - September 5, 2008 - 18:07

This patch adds:

<?php
$trace
['function'] != 'error'
?>

to the trace crawler.

Otherwise it gets mad when notices and stuff are recorded.

Getting notices, which is what caused issue:

[EXCEPTION] Undefined property: stdClass::$code

    * Group: Notice
    * File: drupal_web_test_case.php
    * Function: error
    * Line: 333

AttachmentSize
awesome_ui.patch50.37 KB

#96

catch - September 5, 2008 - 20:20

boombatower, I know cleanup is only for if tests abort, I still think it could be done on cron though. I'd be fine with that happening in a followup patch, but it just occurred to me it'd be nicer to remove it altogether than worry about help texts.

#97

Dries - September 5, 2008 - 21:58

It's not 100% clear. So, will running some test automatically clear all my previous environments or just the last one? Personally, I don't want to worry about these questions. Thanks for answering.

#98

boombatower - September 5, 2008 - 22:04

Basically each run of the test suite cleans up after itself. If the literal clean environments code was run that it might interfere with a different batch running at the same time...something that would be used on testing slaves for t.d.o.

So clean environments is used for cleaning up left overs when tests crash during the development cycle.

What text would make it more clear? I would prefer to get other feedback since you have an outside perspective, not having written the code.

#99

catch - September 5, 2008 - 23:03

Makes sense that we'd want to avoid running clean environment when another test might be running, so cron is no good. How about moving it to devel.module though?

#100

boombatower - September 6, 2008 - 02:54

Cron to devel module? sure. The button, I would prefer is stays in core. Just may need better explanation which I can add.

#101

catch - September 6, 2008 - 11:07

No I meant move the button to devel module, I personally wipe my HEAD database every few days anyway, which is my main use for simpletests, if I was using simpletest for a site, I'd also be using devel, so I don't think we need it there.

#102

Damien Tournoud - September 6, 2008 - 11:49

What about reverting to the old interface on that matter? The fieldset:

Clean test environment

Remove tables with the prefix "simpletest" and temporary directories that are left over from tests that crashed.

... makes complete sense to me. We could simply add that this should not be done while other tests are running.

#103

catch - September 6, 2008 - 12:38
Status:patch (code needs review)» patch (code needs work)

That's fine with me too, and it's a very small issue in an otherwise lovely patch. Marking to needs work for that bit.

#104

boombatower - September 6, 2008 - 21:25
Status:patch (code needs work)» patch (code needs review)

#102: That is what I suggesting.

Added the fieldset back with a bit more description as to its intent.

AttachmentSize
awesome_ui.patch50.16 KB

#105

Rob Loach - September 7, 2008 - 07:17

It would be nice if the tests that failed still had their checkboxes toggled on so that you could just click "Run tests" to re-run all the previously failed tests.

#106

boombatower - September 7, 2008 - 07:37

Hmm...that would be interesting. How about we make a separate issue for that as this patch is already very large and holding up several other issues. Possibly a button that says re-run failed tests...or a combo with pass/fail/both and then a re-run button?

If someone could run the test suite (I'll hopefully get around to it) and compare the results (as there appears to be fails/warnings in core last I checked).

#107

Damien Tournoud - September 7, 2008 - 11:10
Status:patch (code needs review)» patch (code needs work)

Ok, I'm worried about this patch: it has became huge and hard to review, and it mixes rework of backend code (that we need) and of the UI (that could wait).

About the backend: for another patch (#304924: Extend drupal_error_handler to manage exceptions), I need to keep the $custom_caller facility.

About the UI: the progress report is completely unusable on my laptop. Firefox hangs for several seconds each time a test ends, so I can't do anything else in the browser while I run tests. I think we should fallback to the old progress report. On the other hand, I really like the new display of test results. It is very pretty and usable.

Could we split the work on the two, produce a patch for the backend code in a separate issue to get this committed as quickly as possible?

#108

Damien Tournoud - September 7, 2008 - 12:24

Based on #107, I submitted separately a backend cleaning patch: #305077: Rework simpletest backend.

#109

Rob Loach - September 7, 2008 - 16:06

Boombatower at #106: sounds good! #305150: Keep running tests checked

#110

boombatower - September 7, 2008 - 19:06

#107: We can look at the changes made by this patch, but while cleaning up the backend it is very easy to cleanup UI along with it. Especially since new UI needs new backend code anyway. Separating the two will only create more work and postpone this patch.

As for the custom caller stuff that might need to come back, I removed it since it isn't used anywhere and if the code works properly I don't see a use for it.

#111

boombatower - September 9, 2008 - 03:22
Status:patch (code needs work)» postponed

Will re-roll upon #305077: Rework simpletest backend getting committed.

#112

webchick - September 10, 2008 - 04:16
Status:postponed» patch (code needs work)

Ok, #305077: Rework simpletest backend is in. Back to code needs work.

#113

boombatower - September 10, 2008 - 04:49

I'll try and get to this tomorrow. I have class all day, but should have plenty of time after that.