Cache PDF file in server

No. What you're referring would be to develop some kind of 'cache' of PDF files. This could be done, but it would require tracking the revision date of the node and the date of generation of the latest PDF, to invalidate the 'cached' copy in case there has been any changes to the node since the last time that the PDF was generated.

This would save some seconds of PDF generation time, but would waste a lot of space in the hard disk. Since in most situations the only limit is the hard disk space, this feature is probably not useful to the vast majority.

Makes a lot of sense to me, Super G. Not a 'cache', but an archive of snapshots, in PDF, in parallel to the DB contents generating the nodes, which is dynamic and inherently lacks snapshots in the typical hostnig MySQL situation. I agree with you: disk space is cheap and responsive relative to server CPU in the typical shared hosting evironment, particularly with Drupal which is so DB query intensive.

I think that generating a PDF of all nodes would both slow down the site too much and take up too much space. Unless, of course your users would really like to download all nodes as PDF.

I see these options:
Option 1: Saving all nodes as PDF
- Pros:
Faster for all users
- Cons:
Slower site as each node editing action forces a PDF generation
Huge disk space waste as PDFs are stored for all nodes, including nodes not accessed in several weeks/months

Option 2: PDF 'cache'
- Pros:
Faster for most users
Smaller disk space usage
Ability to configure cache size to leave only the top accessed PDFs
- Cons:
No speed-up for the first user to ask for the PDF of a node modified recently

You don't understand..

Your solution makes your server LESS performant. Unless of course, you rarely create/edit nodes, and all your users all downloading several PDFs all the time. Think about it, and you'll probably reach the same conclusions.

Hi SuperG. I am interested in this small module which trigger the generation of PDF when the node is saved. I am currently implementing a site where the content is mostly static, eventually it should be possible to modify your trigger such that it will allow an administrattor to request the generation of PDF for specific node types. If you can share your module I will look into extending it to allow generation of content "by request" of administrator (or other user with sufficient permission rights).

If the node are rarely edited, it is the better solution.
For me, by example, it is the case.

Why don't provide this functionality as on option for advertised user.
for example we could choose between
1- normal mode
2- generated on creation/update of the node
3- generated on the fly for the first time and by update later

It just need to prevent the administrator on what he's doing.

I don't understand your opinion jcnventura an administrator is not a baby.

No.. I'm not a baby but I'm going to act like one on this issue, because you are and you need to learn some manners.

Understand two things:

1. You're using software for FREE that is designed by me in my free time.
2. I decide what I do in my free time.

So, if you want this one, this feature now costs you 5000€ (five K euros). Normally, I'm willing to accept patches that other people do and keep on maintaining those for free. In this one case, even if that happens, I won't accept it. You can, of course, fork the module, and keep on maintaining it yourself.. It would actually create more free time for me, so I would welcome it.

João

Oups, maybe it is my English, so I want to rectify.

When I say "an administrator is not a baby", it is not for you, but for the people who use "your" module.
I wanted to post a patch and I thinked that you were opposite to this. that's why i have said "I didn't understand your opinion".

If you've been hurted by what i have written, sorry it wasn't my intention.
I'm a developper, it was to make advance "Your" project which is used by many people on Drupal.

5000 euros lol

Peace and THX A LOT FOR YOUR WORK !!!

Thanks for clearing it up.. Yes, it really annoyed me at the time, as the phrase only made sense to me as an insult.

The main reason why I don't provide such an option is not that I am fundamentally against it, as I think it would be extremely useful and increase performance in sites that use the PDF functionality. It's the lack of time to do it, and the fact no-one has done it for me. Of course, if someone did sponsor this it would probably move up in the list of stuff to do in my free time :)

It's easy to add a node API hook and have the module create a PDF file each time the module is edited/created. Doing that however, would slow down the node creation/editing process that in extreme cases it may lead to PHP timeouts with unknown consequences (the worst being loss of the node contents). Also, Some kind of interface must be provided to enable the user to re-create all the nodes when a newer/better version of the PDF tool is released. So, on this option, the best would be to use the Job Queue to schedule the creation of these PDFs during the next cron run (of course, if a user were to ask for the PDF before the cron execution, then he would get a delay, and the cron job would now be irrelevant).

As this could eventually lead to several gigabytes of wasted space on the server, I would prefer the cache option, whereas the first user to access a PDF would get the delay, but all the others would just download the cached copy... This would allow the PDF 'store' to be configurable and all the usuallly-used PDFs would be instantly available. The problem with this is that handling a file cache is something that would probably benefit from a third-party module, but the one available (http://drupal.org/project/fastpath_fscache) doesn't seem to be actively maintained on a regular basis, and is used by almost no-one.

João

Subscribing. Thanks

subscribe

I read a couple of threads about the caching issue and all point to this thread now. Has there been any progress on the caching of PDF files? João mentioned in some of the other issues that Drupal cannot handle delivery of cached PDF files. Is that still the case or might core updates have fixed this?

I can see this as a very useful feature, but I do see the performance concerns as well. In my case I only want very few PDFs to be cached, not all of them. I created a view to pull together various pages (like a print basket) which then can be printed into one PDF file. It works fine, but the delay while the PDF is created is a concern. The link to the PDF version is send out in an email, recipients click on it and see the PDF (instead of individual pages with a print to PDF button on it). The few seconds it takes to load might signal to them that the page is not working correctly and they might close the tab/browser. Ideally, I would be able to cache this file to inprove the load time, but I am not concerned about all the other pages that do not need cached versions and work just fine with generation on the fly.

It seems that various people were interested in variations of the caching option. Should we set a different status and see if this can be moved forward?

Thanks, J.

A status change would only force me to set it back.. If you write a file caching module or tell me of an existing module that does file caching, I would be willing to use it to store the generated PDF files and to look for them there before generating them.

@Super G,

Hi,

I am interested in your module that could expand print.module; "a module you wrote to save a PDF output to a file on the server instead of saving it to the visitor's computer."

Could you upload it, so we could try to add it to print.module ?

Thank you

Leon

adding those lines juste after $pdf = stream_get_contents($pipes[1]); (line 386), makes it run for wkhtmltopdf.

$pdfoutput = $pdf;
$filepath = "sites/default/files/print/" . $filename .".pdf";
$fp = fopen($filepath, "w");
fwrite($fp, $pdfoutput);
fclose($fp);

@chipway-drupal: thank you. In D7, I changed your lines (in print_pdf.pages.inc) as follow:

global $user;
$pdfoutput = $pdf;
$filepath = "public://fileviewer/". $user->uid . "/". $filename;	
$fp = fopen($filepath, "w");
file_unmanaged_save_data($pdfoutput, $filepath, FILE_EXISTS_REPLACE);
fclose($fp);

so that PDFs are saved to specified folders, that can be deleted at user deletion (Rules can create such folders upon the first login of a new user, and then delete them upon user deletion - at least this is the way I'm doing it, but please point me to a more functional method if this one is naif).

PS. If anybody is interested... I'm writing the generated PDFs to these /sites/default/files/fileviewer/[user-ID]/ folders because I'm using Fileviewer to have PDFs rendered in an html5 PDF viewer - instead of having them immediately available for direct download, or to screen. To explain further: I imagined that the PDF version of a node has to be stored and rewritten both by Print and Fileviewer in a "shared" folder (the mentioned /sites/default/files/fileviewer/[user-ID]/ folder), so that Fileviewer is able to always load the right PDF (there is always only one available) and convert it to PNGs, for direct HTML5 rendering. This way, no third party software i.e. Adobe Acrobat or Flash is needed to access a beautifully readable PDF document made in Drupal.

At the moment I am using File Field Sources to manually load the stored PDF to be rendered in the PDF viewer (we are always talking about a single PDF per node), at node save... but the idea is to have Rules doing this job, at node save (I don't know if this is possible... it should be matter of controlling fields content at node save - but point me, possibly).

And I would like to say hello to jcnventura: many thanks for your time!

Hi miro marion,
Could you tell if this could be applied to D6?
Thanks!
Rosamunda

Oh, I didn´t realize that #18 seems to be applicable to 6.x :)

But I dont get one thing, those lines shouldn´t be added to the .module file?
And how can I tell in wich nodetypes is this applicable?
Thanks!

Hi again,
I´ve tried modifying print_pdf.pages.inc according to #18 and nothing happens. There´s no pdf created. (I´ve run cron and still nothing)

This is what I have now:

  if (is_resource($process)) {
    fwrite($pipes[0], $html);
    fclose($pipes[0]);

    $pdf = stream_get_contents($pipes[1]); <==== LINE 354
$pdfoutput = $pdf; <==== NEW LINE ADDED
$filepath = "sites/default/files/print/" . $filename .".pdf"; <==== NEW LINE ADDED
$fp = fopen($filepath, "w"); <==== NEW LINE ADDED
fwrite($fp, $pdfoutput); <==== NEW LINE ADDED
fclose($fp); <==== NEW LINE ADDED
    fclose($pipes[1]);

I´ve installed (and I´m using to generate the pdf´s): wkhtmltopdf-0.9.9-static-i386.tar
I´m using 6.x-1.12 version of Print.

Now, I´ve updated print to the latest stable 6.x version, and still won´t do anything.

bump?

I'd like to add, that I think this feature could be very useful.

I'd love the ability to save a PDF (displaying a node via the print--node-type.tpl.php template) to the file system.

Hello jcnventura,
Would this module http://drupal.org/project/filecache/ be the one you are looking for to store the pdf files?

Regds

@lionic, no. That module moves existing drupal caches "tables" to files.

What I want is a file cache management library so that I don't have to create one from scratch (stuff like checking if the file in the cache is valid and serve it or create a new one and store it there, and a cron job to clean files older than the configured file cache lifetime, etc.).

But I think a simple one can be created, using hook_node_* to delete the cached PDF file when the underlying node is updated, and having no cache lifetime at all. The file would be created if needed and stored in the 'cache' directory and served to the user from there. Subsequent calls would just serve that file, instead of generating it on the spot. I already did some work to prepare that on the 2.x, by using the node's last modified date instead of the current time on the timestamp information.

@jcnventura
Cool! Can i help in any way?

I'll announce here when I create and merge that patch. Feel free to test it to your your heart's content at that time.

If you want to create the patch yourself, that would be even better.

Sounds like Boost with its file cache management. Boost can't escape its own cache handling though because its cached pages must be directly served by web server. But in print module use case I consider using Drupal cache API good enough for storage and expiration handling. Upon request for PDF it can be checked if page is newer than cached PDF, amongst security and other checks.

@ogi: I respect a lot your point of view in this, but Drupal's cache API is I believe completely table-based. And PDFs can easily go to the 1Mb file size area, where table storage is not the best solution. I'd prefer something that actually stored the PDF file as a file. Or am I mistaken?

The large PDF file size is a strong argument and none of Drupal 7 cache handlers can handle this sanely :-(

Even File Cache is not perfect here due to its serialization of cached data. With Drupal 7 cache API a possible hack is to have special case for cache objects by print module: 1) file name of cached content (and possibly other data) is passed to cache_set, 2) file name or file handle (amongst other data) is returned by cache_get and 3) upon deletion of such specially handled cache objects the cached file is deleted too. 3) is unique possibility in File Cache compared to other cache handlers.

(Disclaimer: I'm the maintainer of File Cache)

Hi, I've rolled a patch that can cache generated PDFs instead of regenerating them every time. The caching is enabled/disabled via the admin interface (disabled by default) with some options to control file lifetime and maximum cache size.

Files are linked to the node but the sha256 hash of the generated HTML, not the node id + modification time. They are stored by default in private://print_pdf_cache/ if the private scheme is available, otherwise the temporary scheme is used. I have tested with wkhtmltopdf and dompdf; I was unable to get mPDF and tcpdf working on my system. However, I believe I made the necessary modifications for those two plugins to work as expected. The cache is cleared when using the performance / clear all caches button.

Feedback is very welcome, please do not hesitate to ask questions or give suggestions.

@kienan: really nice patch.. Thanks a lot.

I'll probably use only part of this. I'd prefer to start with a simpler version, without cache size management. Limiting the cache size is something that should be done by cron, and through something that accounted for the file usage, to avoid deleting a popular file.

You're welcome, and thank you!

The cache size management should be easy to remove. I agree that it's not the ideal solution since it clobbers the oldest file(s) right now. Another improvement that can be easily made is extending the lifetime of a file in the cache when there is a cache hit on it.

In many ways I would like to some form of cache size management available early on, so that sites generating a large number of pdfs don't run over their disk space too much - a little bit less admin work is nice.

I'm looking forward to when you have a chance to include parts of this in a release :)

http://drupalcode.org/project/print.git/commit/81966f6

I've committed a simpler version of the cache mechanism from #34. The main differences being:
1) it caches only content pages;
2) the cache key is currently the node ID, not a SHA hash of the source HTML;
3) it invalidates the cached file on node update or node delete;
4) that it caches in print_pdf_generate_path() and not in print_pdf_generate_html();
5) there's no cache table for now. The module already has a table that it updates when a PDF is generated, that can be used to implement a proper cache aging algorithm.

thanks!

would you accept a patch that used the print_pdf_page_counter table to expire PDFs after a configurable amount of time?
i'm concerned that other parts of the page might be changing (blocks, etc) and that node update wouldn't catch them all.

The blocks, etc. shouldn't be part of the content, but yes I do see the problem in my approach when that happens. Honestly, however, if you have stuff that changes often and is not part of the node content, and you want that in the PDF, I don't see how caching will help much.

Expiring the cache after a configurable amount of time would be an option. It needs to be run on cron, though.. I don't think that caching and doing file deletes should be in the same function.

It looks like it would be a bit more complicated to clear the cache using cron once every X minutes. If it's set to less than the cron run, it will be cleared slower than expected.

For clearing the cache using cron, I think the best procedure is to wipe the cache by implementing hook_cron_queue_info (or just hook_cron), and then using a cron module eg. Elysia Cron to optionally fine-grain the timing control of cron runs on print_pdf. This has the advantage of being extremely easy to implement in this module, and allowing the administrator a large degree of flexibility; however, it increases the optional dependencies.

Thoughts?

kienan,

the lifetime for a PDF cached object should be days, hours, not minutes.. That was one of the reasons that I didn't use your 'hash' mechanism to decide on cache hits. If you have pages that are built from 'scratch' often (like a views page), it makes no sense to cache them at all. I realize that dynamic blocks will not get cached, and that at the moment the cache mechanism is all or nothing, but there needs to be a compromise here: either disable PDF caching or find a way to live without your content getting updated often.

However, as I said in #28, the last thing I want is to start writing a full-fledged cache module just for print.

Hi jcnventura,

I misunderstood the messages about the print_pdf_page_counter and the cron suggestion. I clearly had something different in my head after reading them. I agree completely the lifetime should be in days, or hours, and not minutes. I hadn't considered the frequently changing blocks / view content when I wrote the original patch, so it makes absolute sense to only worry about the node content for simplicity (and yes, that is a small drawback, but one I can live with).

Sorry for the mis-communication, and thanks for being patient and accepting :)

Here's what I've now seeing, as the solution being discussed, having re-read the previous message more carefully.

- print_pdf_page_counter has the timestamp updated with the request time of the pdf for each node / path.
- there can be an admin set variable lifetime (eg. 1 day or whatever)
- whenever cron is run, all the pdfs are checked to see if the last access is older the the lifetime at which point they will be deleted
- the node_update and node_delete are still there to delete out-of-date files immediately when a node is altered/deleted.

I'll probably play around with it today and see if I can do something.

kienan,

Your points describe the algorithm I was considering pretty well.

It does mean that for a heavily PDF-access site, files in the cache may never expire at all, and they'll just accumulate there. Honestly, disk space is so cheap nowadays, that I think any proper site fitting that profile should not be concerned about it.

Here's a possible patch for using cron to clean up the cache based on the time since the last access on print_pdf_page_counter and a user set lifetime variable.

Apologies in advance if it is more appropriate to make a new issue for this patch rather than to continue posting here.

Edit:

Note : I couldn't seem to get it to work consistently with hook_cron_queue_info(), which would be more appropriate on a site with many cached PDFs.

Nice..

http://drupalcode.org/project/print.git/commit/acb2fc2

And with this, the PDF caching mechanism is finished according to my plans. Any bugs/feature requests related to PDF caching should now be reported in separate issues.