Server-based Screenshots

@bronco
I'm interested... services like this one and with shrinktheweb moving towards preview pages, makes it difficult to manipulate generated images.

I took a look at CutyCapt, seems very promising. My next project, will require a screenshot of every node to a "page" content type to be stored using imagecache/imagefield.

broncomania, I am also interested in your code, "or your way" of using CutyCapt, instead of relying on paid or limited services.

I also agree with you, as to why in the world, "would you use paid services", when the equivalent or better code is open source, thus you should be able to DIY, and contribute back to the community.

If I don't hear from you I will try it myself, or look into khtml2png, also as an alternate method, too.

Definitely would like to hear more about this?

Implementing cutycapt on my site took about 5 minutes... I've probably spent hours and hours investigating and trying out the free services that do the same thing but save you load.

Installation on linux is easy if you follow the advice on the cutycapt website.

Using it with Drupal just modified zubrag's script at http://www.zubrag.com/scripts/website-thumbnail-generator.php to create a simple function:

function cutycapt($url) {
$screenshots_folder = file_directory_path() . '/cutycapt/';
$cache_expire_time = 10080;
$image_type = 'png';
if (isset($url) && valid_url($url)) {
  $website_url = $url;
}
else {
  drupal_set_message(t('Missing URL to grab website screenshot.' . $url), 'error');
  return;
}
$website_url_md5 = md5($website_url);
$cached_filename = $screenshots_folder . $website_url_md5 . '.' . $image_type;
if (!file_exists($cached_filename) 
|| filemtime ($cached_filename) + $cache_expire_time * 60 < time() ) {
@exec('xvfb-run --server-args="-screen 0, 1024x768x24" cutycapt ' . escapeshellarg('--url=' . $website_url) . ' ' . escapeshellarg('--out=' . $cached_filename));
}
if (!file_exists($cached_filename)) {
drupal_set_message(t('Thumbnail Generation Error. Thumbnail not created.'), 'error');
return;
}
else {
return $cached_filename;
}
}

I can't vouch for the code though, I'm not much of a programmer. Basically it just returns the path to the source grabbed image, which you can then play with on imagecache to your heart's content. I set them to expire once a week old, but obviously that's only the source image, so your resized versions in imagecache would stick around for longer.

Sorry for being late on my reply. I was jammed up with a project, still looking at a couple of months b4 I can free myself up.

I'm so eager to dive into this project and try lightsurge's code and hopefully wss_cutycapt will be a great powerhouse.

broncomania, maybe you should open a alpha module page for your module or a new discussion thread, so you can get feedback and cooperation. I wouldn't know how to go about doing this, though; just thought about mentioning it. If you move this thread, please let me know. Thanks.

If I don't speak to guys, have a Merry Christmas and a Happy New year!

broncomania,

have you a working plugin of CutyCapt? Can you post CutyCapt plugin?

My focus is on D7 version of this module becouse the d6 works but it is not 100% completa

+1 subscribe

I tried kthml2png on ubuntu 11.10, and finally gave up. Way too many dependencies on KDE 3, which 11.10 doesn't support. Works fine on 10.04, but is still more complicated to install than cutycapt.

cutycapt, on the other hand, was a breeze to install with synaptic.

subbing.

I've spent a few hours researching cutycapt and broncomania's code proposal in sandbox. Does anyone have any experience with trying to implement something like this?

It looks like the cutycapt project was created and then abandoned back in 2010/2011. No updates have been made since then, and it relies on software that is not readily available. I found some obscure instructions on installing QT4.4.0 that "might" work and I found one, single link to an old repository with QT code in the archives.

I'm not sure this would still work, and I saw many people complain about it not supporting Flash and having quite a few bugs that were never resolved. This is the same issue that I ran across when building ShrinkTheWeb, because none of the opensource or even commercially available code to capture screenshots would yield a high enough success rate. That is why I ended up building my own system from scratch way back in 2007/2008 and have had to update it every few years to keep up with changes.

At any rate, I have asked the owners of this project to transfer ownership to me, and I was looking through the queue at issues that we might tackle first. This one presented an interesting alternative to using an online service, like mine, but I'm not sure that it is feasible.

I'd like to ask anyone who has tried this to offer their feedback and thoughts.

Sincerely,

Brandon

I used cutycapt for a project a while back and it worked really well (apart from niggles with flash content, which I wasn't very bothered about, I'm probably even less bothered now), but that was 4 years ago, like you say it looks a bit left behind. I actually have a feeling I got flash working in the end, but I may be remembering wrong. All I wanted was to do really was make links to educational web resources in a resource directory look more interesting.

You might have more success reviewing it by installing Debian Lenny or a distro from that time?

I didn't mind the labels on thumbnails with shrinktheweb, but I didn't like the preview pages that the the images linked to, although I can see why you would do that to make the service viable. Doing it that way it seemed like I was encouraging users to click links to content, but on clicking it, they went somewhere else, a sort of hijack of the initial user intention.

It was that and the fact that third party integrations at the time seemed to be shutting down as fast as I was finishing adding them, so I was getting fed up, I'm glad to see yours is still around.

PhantomJS looks like the most viable DIY solution for web screenshots now.

Hi lightsurge,

"You might have more success reviewing it by installing Debian Lenny or a distro from that time?"

Whatever we decide to implement must be cross-platform compatible, because it is unlikely that someone will have the exact distribution we develop on. So it is looking more like cutycapt is a lost cause.

"PhantomJS looks like the most viable DIY solution for web screenshots now."

This is something that I will look into and consider. It would have to meet a lot of criteria for us to spend money developing for it:

1. have a high capture success rate (few unresolved, critical bugs)
2. support Flash (even though Flash may be going away in the next few years, there are millions of sites using it currently)
3. must have a very active community (to ensure we don't spend thousands only to have it "break" down-the-road with nobody fixing issues or keeping it updated with web technologies) *
4. must be cross-platform compatible (to keep down support costs and also avoid lots of disappointed users)
5. must be reasonable in resource consumption

There may be other criteria but those are the main ones.

*To put it into perspective, ShrinkTheWeb has to spend in excess of $100,000 per year just on research-and-development to keep up with web technologies to keep a high capture success rate. For our little company, that is a lot of money. So we definitely don't want to take on something else that requires a lot to maintain --especially if we will have no way to earn the money to pay for that maintenance.

"I didn't mind the labels on thumbnails with shrinktheweb, but I didn't like the preview pages that the the images linked to"

ShrinkTheWeb has always had a policy to never force watermarks on thumbnails. In 2013, we added the ability for "Custom System Messages" upgraded users to put their own watermark on thumbnails but not one we force on them.

The preview verification page lasted for about a year before we decided to switch back to the original way. Since a couple of the larger competitors, at the time, had switched to a preview page to ensure that users were being redirected to the site they expected (for "safety" and "malicious abuse prevention"); we thought that was a reasonable idea and followed suit. It was the only time we took after a competitor and it ended up being a big mistake. Even though the idea of "verification" was meant for good, there was not one person who saw it that way. Everyone felt that users were being "hijacked" (as you say), and it also complicated our integrations. So we made the tough decision to switch back and it has been back to normal for several years now. Just a little detail for background's sake. ;)

"It was that and the fact that third party integrations at the time seemed to be shutting down as fast as I was finishing adding them, so I was getting fed up, I'm glad to see yours is still around."

Very true. Many website screenshot services have come and gone nowhere (or disappeared) in the last several years. The problem is that when/IF a service becomes popular, that is usually it's death knell. It is expensive to properly run a truly scalable screenshot service and it is a struggle, if not run by a very well-funded company. In our case, ShrinkTheWeb was not well-funded, but we were fortunate enough to have enough paying customers to support all users -including Free ones. ShrinkTheWeb is a small company with BIG company expenses. So I feel very lucky to have so many loyal supporters.

##########

At any rate, I think that offering a server-based option that does not rely on a 3rd-party would greatly benefit the community. So I will consider it. However, offering that option does hurt my business and it's already tough to keep the lights on when we have to spend so much to keep up with demand (and targeted attacks).

I don't know how people would feel about it, but I think I might require a linkback to my service in exchange for providing a completely free, server-based solution that doesn't require any effort on the part of Drupal users. Personally, I think it is not asking much considering the effort we would be putting in. I don't know. Just a thought.

Cheers,

Brandon

Just a quick update: I've downloaded the various branches of this project and have begun reviewing and comparing code to become familiar with what already exists. At the same time, I've noticed that this was requested for the 6.x branch, which is no longer -technically- supported (as even Drupal will no longer support 6.x in the near future). So, I'm updating the title appropriately and updating the version as well.