JavaScript screen recording plug-in

We have thrown this idea around several times and everything that is available requires some sort of client side capture device (software). Even firefox plug-ins that perform some of these captures of OS depended and the plug-in makes them browser depent.

I don't think there is anyway to get around dependencies.

I have designed the module with those two plug-ins in mind since they provide some of the most useful data. They are outside the scope of the GSoC project, but depending on how things go I will look in to writing them. (I'm going to maintain this module after GSoC)

It would be much preferred if there was someone with experience in these areas as it would be nice if the suite could as some point do things like overlap feedback messages with video or sync separate audio and video files.

Thanks for this post eigenator!

So I have researched this a little and spoken with a few actionscript/flash specialists they told me capturing a audio is relatively simple. There is acctually a module that partly does this : http://drupal.org/project/riffly and some adobe documentation to go along : http://livedocs.adobe.com/fms/2/docs/wwhelp/wwhimpl/common/html/wwhelp.h....

Screen capturing seems to be a whole other issue, although it is definitely amazing if we could implant it, I am unsure of the audio value without screen capturing. It would be great if in the analyses setup were we can point the usability tester to the point of the video/audio capture where its going wrong.

Adobe Connect is doing screen capture, so it should be possible. However we have to think about the participant, we really dont want to hassle the user with installing anything. There is already very little incentive for the participant to do a study, setting requirements will only push it back further. Some people have flash installed (97% ish?) so I doubt that will be an issue.

I pinged Aaron Winborn on this issue and send a few mails to dutch flex specialists, I hope to get back on this issue in 2 or 3 days.

This would be great for someone who knows what their doing to complete these plug-ins. The API is ready and they could start at any time. Obviously I can help were needed or write the plug-ins myself time granted.

Sadly I won't be attending, so someone else will have to take over that.

IRT to screen capture, it's possible to mostly capture all of a user's interactions with the page through javascript and even 'replay' a lot of them in a time shift.

For example, the location of the mouse cursor on the page can be tracked, as can the visible portion of the page in the browser, the browser width and any typing or clicking the user does.

'Recording' all this data would not be trivial however, although could be done to be browser independant (albeit a lot of cross-browser issues, no doubt) and not require any installs. 'Replaying' the data is likely to be more troublesom since I don't think the browser lets javascript easily scroll the page. Some trickery could possibly work around that.

Just an idea for now...

limit = 10; counter = 0; $('body').bind('mousemove', function(e) { if (counter > limit) { d = new Date(); console.log(e.pageX, e.pageY, d.getTime()); counter = 0; } counter++ });

So I wrote up this javascript that should do it. Ok kidding, dmitrig01 gave me feedback on this issue and intermediately gave me code to work with. When I ran it in firebug it seem to be logging it quite good, although the time stamp seems to be a bit confusing.

Jimmy is this something that you could easily turn into a plug-in?

Yes, mouse tracking was one of the original plug-ins that we were looking for and this approach would seem fine.

As Bevan said the hard part will be re-playing the data. From prior experiments it is possible to scroll the page with JavaScript, but I don't think you can move the mouse. So the mouse tracking data would need to be rendered in another manor. (Or perhaps with a plug-in for firefox that would only need to be installed on the UTS engineer's machine - purely theoretical, no idea if that is possible)

I was just pointed to : http://www.robotreplay.com/ Can you take a slight look at how they do it? The do a lot with javascript.

Wow! Robot Replay is great. This video works and explains it quickly and simply; The video on their homepage is broken.

The magic is in the javascript which is uncompressed and open for viewing See robotRecorder.prototype.record = function() { around line 1200.

An integration module for drupal has been on d.o for a couple of months/ I updated it to drupal6 but am having problems getting it working on drupal.geek.nz. AFAICT the problems are to do with robotreplay.com, not the drupal module.

We should consider integrating this into the UTS instead of, or as well as, building our own version.

Nitobi seem open-source friendly and say that a free version of robotreplay will always be available.

Bevan, thanks for pointing us to the module, sometimes the amount of usability related modules out there still scares me.

I am leaning towards building our own version, as we can do 'things' with the play back and don't have to rely on a third party to do it (ie syncing with audio feedback)

It won't require the user to setup an account on RobotReplay? And BIG sites usually want a bit more control over their logging, it's always hard to go with a third-party's 15kb.

But this choice will require us to do some work.

I'm up for writing some stuff like this, but we need to determine if the project is complete for GSoC requirements. If so I would like to release a stable 1.0 and then begin work on some more advanced data collection plug-ins and analysis plug-ins.

Just an idea on replays:

Why not use an icon you move around the screen, instead of requiring a firefox plugin? It would seem to me all you need is a visual indicator to show where the mouse was, and not necessarily a cursor.

Does javascript have the ability to perform a 'click' at a specific coordinate? or will playback be achieved some other way?

I agree with your comments, however it's worth considering that creating this from scratch is a lot of work, cross-browser debugging and head-banging. Don't under-estimate cross-browser javascript. I know this from experience. Multiply your time estimates for js work by 10, not 2 or 3 as usual; Seriously. This applies even with jQuery. Don't even try without it.

On the other hand integrating an existing service is (relative to the above) trivial.

As a compromise, using robotreplay as a guide, starting point, base to extend upon or even to collaborate could be a massive time-saver. There are no license or copyright details in the the magical js file or on the website (except for the website content), and it is unencrypted. IMO this suggests it would be okay to use it and that nitobi might well be interested in collaboration or even open-sourcing parts of their service.

I think we should approach them to discuss these open-source and collaboration possibilities.

We should also see what alternatives to robotreplay are out there. 20 minutes on google turned up ClickTale which appears to the current King of the domain and TapeFailure which appears to be more advanced than RobotReplay but less advanced than ClickTale. It has been shut down in order to relaunch as VisTrac. I found this review / comparison of the three.

I also found these resources which Jimmy will probably find useful:

• http://css.dzone.com/tips/jquery-tracking-the-position-o
• http://www.lupaworld.com/89596/viewspace_32961.html
• http://www.mcs.vuw.ac.nz/cgi-bin/info2www?(elisp)Mouse+Tracking

> Why not use an icon you move around the screen, instead of requiring a firefox plugin? It would seem to me all you need is a visual indicator to show where the mouse was, and not necessarily a cursor.

Assuming you're talking about replay, and not record, this is how RobotReplay seems to work I think.

These are the sorts of questions and problems that have already been solved by RobotReplay etc.

The icon for cursor is great...while I was discussing it in person I came to that conclusion as well. I'll take a look at the libraries mentioned.

Great search, Bevan.

Clicktale definitely looks amazing, I am unsure about all of it's functionality but most of seems to work fine, I just installed it.

We could indeed, setup a relation to find out whether Robotreplay is willing to cooperate, although I don't have really high hopes, as they released this as a service and probably run their company similarly. You want me to send an e-mail, Bevan do you want to?

We have to more thoroughly consider the scale of this type of logging, since 15/kb doesn't make much difference on a small site, but if you're running something like drupal.org or bigger, you usually work with a lot more data UI wise, and performance wise 15/kb makes a difference.

I'll be working later this week more on the UI, Bevan could you please post your ideas on the workflow in the workflow issue? Bevan, make sure in your blog post that you mention that we are just starting on work on the UI, that the process will be much lighter and easier for the user in a few weeks.

Please open a new issue, for this feature request and explain a bit further. I do believe that, we should focus on avoiding anything that requires the user to do stuff. There is lots of usability software that already does this and as we are primary testing within the browser, we should take advantage of that.

Looks like the feasible plug-in is the JavaScript screen recorder and is to be part of GSoC 2009 if possible.

List on the project page, http://groups.drupal.org/node/21985.

Considering that robot replay's script is over 1800 lines of JavaScript goodness this is not something that can be accomplished in ~10 days as I suspected. Recording all events is by no means trivial.

The best open source (even jQuery based) library I can find is http://visionmedia.github.com/jquery-event-playback/

It looks fairly complete as you can tell from the demo, but lacks a few important things. No keyboard/typing support and does not record page scroll events.

I can attempt to improve the script or write one from scratch, but it will require longer then 10 days.

Interestingly enough a search for "javascript screen recording" results in this issue.

I would like Bojhan's thoughts.

Some interesting alternatives http://screencastle.com and the open version http://www.skoffer.com. It doesn't seem to work on linux and thus is not truly a solution.

Seems to be true of all the services described at http://www.hotscripts.com/blog/online-screen-recording-services/.

Except http://www.screentoaster.com/ loads and seems to start, but has an issue stopping. It detects my mic and tv tuner card.

Another option altogether is to use something like Java which is cross platform, or even develop our own Java recorder.

tags

If the javascript plug-in is pursued it could record scrolling and keyboard actions as well. In the same fashion that robot replay does.

I think one of our concerns is not to alienate a particular OS.

Here is another JavaScript screen recorder:

http://drupal.org/project/userfly

Video demo:

http://userfly.com/screencast

Subscribing

You can also try to use Screen VidShot.It's really cool screen recorder.