First: Thank you. With your help I've got successfull all my issues solved I had with Video.js, because of storing the videos in private filesystem. Firefox, IE9, iPhone had more or less issues with no resumeable downloads for videostreaming.

Another issue, that you maybe could adress with your great module could the downloadung of pictures (that I also store in private space because of access control).

The issue is, that drupal is not returning status "403 not modified", when the file not changed since last request, what causes loading every picture everytime the articles get accessed.

As my site is a fotoblog, where I store pictures in high quality, the load is high. The pages gets more than one time accessed by the same user, what causes high (useless) traffic, whats a problem espezially for mobile devices over cellular network.

It would be nice, if you could take a look at this. Thanks.

Comments

KarstenS’s picture

Title: May you add support for returning 403 status when files in private filesystem not changed? » May you add support for returning 304 status when files in private filesystem not changed?
KarstenS’s picture

StatusFileSize
new499 bytes

With help of this thread, I've got it into your module. Maybe this could be interesting for others too.

One thing of the linked thread, that not worked, so I've left it out:

if(!empty($file))

Maybe you take a look. I really don't know it is important or not.

A patch for your module attached.

sinasalek’s picture

It's probably better to use file's modification time instead of query the database
Does this work for you on all major browsers?

KarstenS’s picture

It's probably better to use file's modification time instead of query the database

As I've wrote: The code is not mine. I've just done some modifications, like turning out stuff, that your code already do later and set the right string variables into it.

And removed a little part, that not worked, without seeing the reason why it did not work): if(!empty($file))

During implementation I've use firefox with proxomitron in the middle, to watch the requests and answers. As I was ready, I got with no file in cache answer 200 and with unchanged file in cache 304.

After that, I've cleared the browser cache of IE10, FireFox 19 and Chrome 24 and took a look at the network meter sidebar gadget. I've load all my photogaleries in all 3 browsers. I had high load as expected. Every file got downloaded.

The second turn with every file in browsercache. Same Galleries. Nearly no load.

After that, I've checked with iPhone (iOS 6.12) . Result: Its using the browsercache too as I wanted to get.

Yes. Its working.

But one thing I didn't undestood. To check for not modified, the if condition should be

if($modified == $ifs)

and not

if($modified < $ifs)

Am I wrong?

EDIT: Yes. You are right. Something ist wrong. One thing I've not tested before was uploading a modified file. After that it was still using the old one :(

I hope I find some time to try the way you've suggested.

EDIT2: I've found a working solution, that really detects changed files and returns 304 when not changed. Only filetime seemed not to work, why ever. So I do also check the etag too.

New code attached next post.

KarstenS’s picture

StatusFileSize
new3.25 KB
KarstenS’s picture

StatusFileSize
new3.59 KB

I've added a new file as I've found some things that really could not work as it should. Actually I can test it in 9 or 10 hours. I also asked a question within the code, as my problem yesterday was, that my "get time from file" returned the value of the old file (was the timestamp cached?).

Did you notice, that the "If file exists" at the beginning of your module works with case sensitive file names, while the download itself use case intensive file names?

KarstenS’s picture

StatusFileSize
new3.58 KB

Uhh. What an issue I've build in in the No.2 archive.

Corrected it in the new build.

KarstenS’s picture

StatusFileSize
new3.46 KB

My final thing after testing.

Its working with time only now. I had in the if contition a = instead a ==. Fixed

To be sure, the ETag check is still in it. Maybe there is a file replaced by a file with same timestamp. So the ETag should make it more sequre I think.

Here is my final version. I you agree it is fine, maybe we suggest it for DP8 core?

KarstenS’s picture

StatusFileSize
new3.5 KB

Ok. Now really my last for now. After iPhone generated

Notice: Undefined index: HTTP_IF_NONE_MATCH in _resumable_download_download()

in the drupal log when loading a page with a HTML5 video, I check for "IF_NONE_MATCH" before reading it and when not set, it only uses timestamp for modification check.

sinasalek’s picture

Great, would you please provide a patch?

KarstenS’s picture

Status: Active » Needs review
StatusFileSize
new3.33 KB

Hope I've done the settings here right and its shows correct as patch.

Before I created the patch, I've made code and comments a little bit nicer to read (and corrected some writing errors).

deanflory’s picture

So, I take it this patch breaks the project page's "NOTICE : This module does not need any patch to work" rule and I should apply it?

KarstenS’s picture

The notice is not wrong. The module is working without a patch.

This patch do

1) add a new feature to the module

2) and is actually for testing and not for productive usage

When there will be no bugs found, it hopefully will get into a new release of the module.

Till now, it works for me without any issue.

sinasalek’s picture

Issue summary: View changes
Status: Needs review » Needs work

tx KarstenS, your patch will be included in the next release.
I'll take care of the minor issues and testing myself

KarstenS’s picture

I was running in issues with large files (HTML5 video streaming).

Depending on the device I was watching the video, it took more and less long to hang up the whole webserver (had to restart it local on the device it is installed).

I've talked to the guy that made the server package I use and after he took a look at it, he told me it could be caused by delivering file data through PHP and I should use (my server runs lighttpd) X-Sendfile instead.

Ive worked on it using X-Sendfile and X-Sendfile2 (code is not cleaned up for this), but this would work on lighttpd only and sometimes some pictures get not transferred and I don't know why.

lightsurge’s picture

I'm not sure this is a good idea. From my understanding, if you return 304 for a private file, you're not only saying that it's okay to use a file that's in the client's cache, you're potentially saying it's okay for a forward proxy to deliver up a file it's cached... when of course it might not be okay, because it might be different client.

KarstenS’s picture

It works this way.

1. The first time a browser requests a file, it does it the normal way. The server responds the file with a tag in the answer header.

2. Next time the browser requests that file, it knows the file is already in the local cache and changes its request to "Send me that file, if its not modified since...".

Now there are two possibilities.

The file has been modified so the server do the same answer like in number one.

Or the server responds 304: "I do not send you that file becaus its equal to yours" and close the connection.

What proxies do is has nothing to do with this one.

What 304 do: It heavily reduces network traffic by not sending files to the same client over and over.

lightsurge’s picture

re #17

My worry was that the module would return 304 if the file is not modified before file_access() checks were made, so potentially in a case where two people are sharing the same browser, if one has access to a file, the other may also be able to access it via a 304, because the file is in cache, or worse still, two people using completely different machines could get access via a 304 because a forward proxy has it in cache, and the code doesn't check for file_access before responding 304.

I took a look at the code the other day though and it looked like the access check is made before responding with 304, so I think all is fine.

sinasalek’s picture

Status: Needs work » Fixed

NOTICE: I only applied the patches, i have not been able to test it, if someone confirm that everything works properly i can create a new release
https://www.drupal.org/project/resumable_download/releases/7.x-1.x-dev

KarstenS’s picture

I switched to a Windows System and Apache some years ago. I'm using another plugin here as I didn't wanted to rewrite the XSendfile code for Apache again.

But I had no issues with the 304 part till the end.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.