SiteEffect: Automated frontend regression testing

SiteEffect has a build in crawler, but adding relevant D8 urls manually is tedious. On the other hand adding all urls automatically would lead to an explosion of urls, e.g. with all watchdog messages. The idea was to get the router item behind every url and only add one url per router item. I got the basics to work at the dev days after talking to dasjo and klausi. I now finished the integration into siteffect. Without any other then the default core modules enabled I'm now capturing 223 urls.

I have tested the patch to remove the misc icons (#1356602-19: [META] Clean up icons in misc) which got reverted because it removed the 'configure-dark.png' which was still used on the "Manage Display" page. SiteEffect detects the missing icon as well, but of course fully automated.

Tables as you can tell are not being handled correctly by siteeffect right now as I didn't look into it yet. Most important thing is though that changes are being detected and you can easily check through the SiteEffect UI that images are missing. The siteeffect url overview below shows the three failed urls, which are all missing the icon. Note that only 'article' is being checked as all 'page' urls go back to a menu router item that is covered through 'article'.

This is not ready at all to run fully automated on a remote server somewhere, but this will be my main focus in the near future. Up until then I can always run tests locally if anyone is interested. It would be good to test other patches, especially those changing the themes or frontend.

Thanks for the feedback. I'll take a look at admin_menu, I assumed they do some hard coding to e.g. show all urls for the content type route, will see. Right now I'm handing all new urls from SiteEffect to a tiny custom D8 service, which then resolves the url to the route and sents it back. SiteEffect checks if the route isn't yet present within the urls and then adds it. BUT SiteEffect doesn't care about the route itself, it just sees it as a unique url identifier, therefore on the D8 site we can just append e.g. the nid to the route string. This would actually be quiet nice as you can then decide per site which urls you want to track. That would be the SiteEffect contrib module then.

Another problem I see is dummy content / config though. Testing everything without nodes, views, languages, placed blocks etc. probably doesn't make too much sense. Do we have some installation profile for that? Do we need to come up with one? In that case it would be good to come up with a list of needed dummy content and data.

Wim just send me a link http://www.phpied.com/css-diff/ which creates gifs based on phantomjs and imagemagick. @TechNikh gifs are also the main output of your solution, right?

The problem that I'm having with gifs are kind of the same that I'm having with pixel diffs. I still can't really tell where the src of the change is, e.g. can anyone tell me what happened here: http://www.phpied.com/files/diffcss/tmp/dif.gif
Did the width change, margin, padding? On which element... and it get's a lot worse when you have more then one change on the page.The other problem I see with gifs is that you have to check all pages manually if they throw any error at all.

@znerol @TechNikh Maybe you can open separate issue for your tools and link them to the overall issue #2099579: Discuss automated visual regression testing as I open this issue here in particular to post updates on SiteEffect and this is already starting to get out of hand. I therefore changed the title of the issue.

I'll have the next two weeks to just work on siteeffect, so there will hopefully some major progress.

Sorry, for the late response, I wanted to try a couple of things but didn't get to it yet. There will be an API for siteeffect at some point, but this will take some time. Besides that, siteeffect records so called "snapshots" which consist of A) a full sized screenshot and B) a DOM like representation of the page with key values. This means that simply handing siteeffect a screenshot will not be sufficient as siteeffect does a lot more then "just" compare images. What siteeffect therefore needs to work properly are callable urls.

@fabianx I didn't quiet get the data format / the meaning of the data in comment #17. SE uses webdrivers to collect all the data, in particular I'm using the "nodejs wd" module. But if I get it right then your patch makes it possible to save e.g. a node deletion confirm page to a static html file??? If that file contains the rendered html with the original script and css file references then it would be really easy for me to process these. I just need to open the files through the webdriver and can record all the data I need. If that really worked then we could make major progress here!!!

Handling content changes like the timestamps you mentioned is one of SE main focus areas. SE creates the following view

You can then mark/flag these changes (layout, text, pixel) and SE will then ignore these changes in future testrun. This is possible because SE doesn't just work on images, but on mapped elements. Right now you can only define ignored elements through CSS selectors globally, so not per test, but this will change soon. I'm also working on automatic generation of the shortest unique CSS selectors for elements, then you can just flag elements through the UI. Need a day or two to get this fully working, as mentioned, siteeffect is still under heavy development. The real potential though lays in detecting where changes are actually coming from. As another example one of the screenshots from the twig testruns:

SE gets that the search field, forms block etc. shifted down because of the changed heading and therefore doesn't mark it as a change / error. This isn't really necessary for core development as hard regression testing might suffice, but it will be really useful for real projects.

But back to topic. Would you be able to sent me some of the dumped test data? I could try to get your patch up and running, but it will - as always - take me couple of hours, time I just don't have right now. This way I could easily test if this could work in general.

Wow, this is fantastic and closes the gap I had no clue how to deal with. Looks like I'm talking to the right person here :)

The amount of urls is huge, at the moment siteeffect won't be able to handle 7000+ urls and it will take foreever to run all tests for the urls. Something like this isn't really the scenario I have been planning on to deal with. I would need to implement a simple routine to check if anything even changed at all, something like checksums on the PNGs and DOM files, that way I could save a lot of computations. This would be basically a preprocessing step. Besides that I need to reuse the webdriver session as right now for every tests the browser will be killed, caches cleared, etc. If I reuse a browser session it should speed up things a lot. It's all in the siteeffect backlog, but there are other more critical issues I have to work on before that.
We could look into more basic pixel-diff approaches, but these will also take quiet some time to run as they need to load the pages, save the screenshot and do a full pixel diff.

I have tested the first 100 urls and it all works, which is really great news. I "just" need to scale it. So this might take some time, sorry! If only I had more time... the usual. I'll think about how we could make some more progress here quickly. Maybe you can set up wraith (phantom + imagemagick) and filter the urls which are 100% identically and siteeffect then handles the rest of the urls which differ for whatever reason. Thoughts?

Maybe you could spin up a second instance with the twig patch, I can give siteeffect two different urls for master/dev version of a page. We could then test if this works with a couple of the know failing urls. Just to make sure it all works in general. Just an idea.

I will sent you the code for getting the route of a url, maybe you can make use of it.

I implemented a PNG CRC checksum and session reuse in a nightly session, so we could give it another try. The urls seem to have changed tough, am just getting an access denied.

Thanks! The session and checksum improvements seem to do the trick. I captured 1000 urls in under 30min with two processes running in a VM. Roughly the same for the diff phase. As this is kind of an extreme scenario I think it is quiet ok, a decent server will be able to do much more then that. My UI will burn though if I put 7000+ urls in their though, but I could try to do it batch wise for now. What would be super useful would be another directory with the output of the patched test runs. Then I can compare them directly and switch forth and back as need. Not sure when I would have time to look into it though, maybe next week at some point.

I'm pretty sure SiteEffect (or any other visual regression test) would have caught this. I spend quiet some time to make all the dates deterministic, some tests now still fail for whatever reason, but it looks like 98% of the tests are fine with the hack. It's a really evil hack, as I even need to replace the usage of native PHP Datetime(). This will never go into core, so we need to solve the problem of how a reroll of the patch can somehow be automated. Right now I'm using PHPStrom to refactor and replace function calls with my own testing mockup functions.
With Fabianx's approach of writing all test pages to files I now get more then 11.000 urls. I need to take a day and put it all together, working with the testbot is always time consuming. But I'm convinced that this will work! If I got this working I will test the issue mentioned in the comment above and the initial issue of twig autoescape and post the results.

@YesCT thanks for linking the issue. I think I need to switch strategies, getting the testbot output to work will take way more time then expected. I tried to reroll the working patch from 2 weeks ago against head and I got tones of merge conflicts. I have now tried to replace function strings through regex and sed, but the outcome seems to differ from my patch. Besides that there are still issues with UUIDs not being deterministic between runs. Lot more work is needed here.

Therefore I think in the meantime I'll go back to the approach of testing all kinds of urls of a fresh installation with all modules enabled. That already worked at some point, I only need to see how I can automate the whole process, but that shouldn't be to difficult on AWS or digital ocean.

platform.sh seems to be a good option to easily spin up new installations with specific patches. I'll have some time tomorrow to take a closer look. Testing RTL shouldn't be a problem.