broken image paths on upgrade (or migrate) if using private files setting

i just added a 5.x-1.6 version of the image module. please give that a try, if it doesn't fix it re-open this issue.

are you using mysql 4.0?

Seeing similar issues here. MySQL is 5.0.45.

Log claims to be regenerating derivative images. Any image load gives page not found errors. Worked in drupal-5.2 with image-5.x-1.0. Fails to load images after upgrade to 5.3 and upgrade of all modules to latest on drupal site. Using private files with similar URLs to those of original poster. Actual images are still in the same place.

look at the files table in your database. for my Drupal 4.7.x sites, the filepath field showed relative paths, e.g. images/image.jpg. to work with Drupal 5.x, , i've needed to change the paths to be absolute, e.g. /drupal/file/system/path/images/image.jpg.

i, too, was not able to see existing images after upgrading to Drupal 5.3. when i created a new image i noticed that it stored the absolute path mentioned above. running a mysql query to update all existing images[1] seemed to fix the problem for me, but i'm not clear if this is the desired way for the paths to be stored. (nor whether this is specific to the Image module or something related to Drupal's file handling in general. the absolute paths in the filepath field also work for non-image files.)

i've tested this on a site with Drupal 5.2 and Image 5.x-1.5, and a site with Drupal 5.3 and Image 5.x-1.6. both sites are using private files, which i suspect is a factor.

[1] e.g. update files set filepath=concat('/drupal/file/system/path/',filepath);

I've been investigating this issue with an upgrade from 4.7 to 5 using 5.x-1.5 and 5.x-1.6, both without success. Lines 94 & 95 of image.install are supposed to check for duplicate records in the file table, but after disabling both lines:

//        $ret[] = update_sql("DELETE FROM {files} WHERE fid = ". (int)$file->fid);
//        $ret[] = update_sql("DELETE FROM {file_revisions} WHERE fid = ". (int)$file->fid);

...the upgrade worked without a hitch, all image paths being retained. Hope this helps...

The Secure Pages module was what was biting me. When doing the upgrade via a https connection, it is failing to allow update.php to run in https mode and is instead switching back to a node. That was preventing the updates from running for many of the rest of the modules. Going back through and enabling the modules one by one got all images back fine. Now to decide what to do about secure pages.

subscribing

@mediafrenzy -- it's easy enough to "correct" by using a db query like the one i mentioned above (just replace '/drupal/file/system/path/' with your actual file system path). but, before i'd advocate that fix, we should figure out if that's the "right" way as far as Drupal is concerned.

again, this seems to be a bigger issue than just the way it pertains to the Image module: should files.filepath contain an absolute path? it seems to me that this changed between 4.7.x and 5.x, but i can't confirm that yet. adding a new image supports that hypothesis (an absolute path is written to files.filepath), but i haven't had a chance to test w/ any other file types yet.

if we could see what Drupal does by default w/ new files of various types, preferably using a few different modules, and test against public/private file settings, it seems like we could make some solid conclusions.

OK, an update when working from a much cleaner installation...

when i upgrade Drupal from 4.7.7 to 5.3, then update the Image module to 5.x-1.6 (running update.php after each step, of course), i find that files.filepath is updated for the original images, but not the derivatives (sounds like what mediafrenzy was describing above). that is, files.filepath has an absolute path for images where filename='_original', but a relative path for all other images.

so, a (MySQL) query like this is working for me (assuming Image's "default Image path" has been left at its original value: 'images'):
update files set filepath=concat('/drupal/file/system/path/',filepath) where filepath like 'images/%';

Just adding that I'm also having trouble updating a site to 5.x-1.6. Images are not displayed and the site is still running Drupal 5.2. I'm pretty sure the key is the private files setting.

When I switched back to 5.x-1.4, images are displayed again.

Drupal 5.2

MySQL 5.0.27
PHP 5.2.3
Apache 1.3.37

The SQL is OK, you'll need to substitute your own path though.

mediafrenzy, can you confirm if this issue has been fixed? may the ticket be closed?

I've just migrated from 4.6.6 to 5.6.
I needed to do the following:

- Using phpMyAdmin, export (in SQL format) the results of:
SELECT fid, nid, filename AS image_size
FROM files
WHERE
filename = "thumbnail" OR filename = "_original"

- Change the resulting SQL INSERT so that the results above may be reimported into the image table

Re-opening this since I think it's the same issue.

I just did an upgrade from 5.x-1.x (// $Id: image.module,v 1.209.2.11 2007/03/05 03:54:04 walkah Exp $) to 5.x-1.6 and saw a bunch of files table updates (which ran smoothly). I ended up with paths that looked like image/pic.thumbnail.jpg instead of files/image/pic.thumbnail.jpg - this resulted in broken images using private file system, although the originals worked.

Ran this to tidy it up in case it helps anyone:

UPDATE files SET filepath = REPLACE(filepath, 'images', 'files/images') WHERE filepath REGEXP '^images';

This is still an issue, and should be fixed in the upgrade path.

Also, this is pretty much the very definition of critical. ;)

Just confirming I had this issue too on Drupal 5.9 with Image 5.x-1.9 (upgraded from 5.x-1.x-dev), and using private files. I resolved it by updating all the files.filepath db entries starting with "image/" to be full absolute paths.

Newly uploaded images (to their own node) seem to be added just fine and display without modification, but they don't show in Preview (though if you Preview and then Submit they show fine too).

Cont'd from http://drupal.org/node/299961 which turns out to be exactly this iissue albeit with Inage 5.x-1.9

The story so far : Drupal 4.7->5.10 using the handbook method of a test upgrade to a copy of my site at mysite.com/test (subdomained as test.mysite.com). Private filesystem. Result : ~1,000 images not accessible.

Drupal 5's use of Realpath() appears to be the culprit.

I tried this on both the live site and the subdomain test site

<p>Realpath for test.tonysleep.co.uk/files :&nbsp;
<?php 
 echo realpath('files');
?>
</p>

realpath returned the same path for both the live site and the subdomained site, ie '/home/halftone/public_html/files', which clearly isn't going to work for files in test/files

I can only get this to work by setting the file-system path to an absolute server path : /home/halftone/public_html/test/files for the subdomain at /test. Which fixes the problem inasmuch as I can now upload new images and they show up as images. But previews still don't work. They are created in images/temp however and moved when the node is saved.

Also it does leave ~1000 images in the files table with the wrong filepaths (ie 'image/abcd.jpeg' instead of '/home/halftone/public_html/test/files', but that's fixable via SQL.

However I'm not convinced I should be using the absolute server path. It makes any test site very un-portable to have the server path hardcoded into the files table. Besides, files (zips, PDF's etc work fine as they are, with relative filepaths of 'files/example.zip'. I can download OK without changing them. But 'images/example.jpg' don't get served.

Does this point to broken coding in Image module or files.inc, where realpath lives? I don't really want to change all the filepaths to absolute if I'm fixing a symptom rather than the cause and creating a problem that will bite me at some subequent upgrade.

Relative paths for the filesystem have always been OK previously. And the Image module upgrade just copied the old relative path across. Surely it should nag the user that an absolute path is now required, if it is?

What's the status of this issue after 10months please?

Any advice welcome, except possibly 'stay on 4.7'...'

(Ref. #31, adjusting version number)

Further to #32 I have now done a fresh install on a spare domain, Drupal 5.10 + Image 5.x-1.9, to try and figure what's going on. Files method is set to private.

- With fille system path set to 'files' Image works fine except previews still do not appear. The field 'filepath' in the files table for new images then contains 'files/images/example.jpg'.

That is not what 4.7 did. 4.7 stored simply 'images/example.jpg' in filepath, when system path was set to 'files'. The upgrade to 5 should adjust these paths to 'files/images/example.jpg but does not.

- If I set the filesystem to /home/ohhffsco/public_html/files then filepaths for new images in the files table contains '/home/ohffsco/public_html/files/images/example.jog'. Again this works aside from the fact that previews don't display.

Interestingly, images with the filepath'file/image/example.jog' also work in image galleries etc but not if embedded in stories, pages etc

- So I have used the SQL query at #24 and set the filesystem paths= 'files' and files/temp' and all appears to be working correctly so far even in my sudomained test site. Image_assist works OK too.

The realtpath() stuff doesn't appear to be an issue as I thought from the logs and reading other stuff on Drupal. URL's in the HTML are correct and the images files get served,. Just an artifact of wrong filepaths I guess.

The basic evil seems to be that the upgrade from 4.7->5n should have run that query @#24 to adjust filepaths to files/images then none of us would have this trouble. The only issue I now have is why previews don't work. It may possibly be that I upload pre-sized, so no server resize is involved. 4.7 used to preview them though. Anyone know?

If your original size is smaller than your preview size then image won't create a duplicate "preview" image.
That's by design.

Thanks Hetta. The images I was uploading are the exact same size as my preview size (520px). Copies do get made into 'images/temp' but no preview is possiblle.

I'lve now tried with a larger 1280px image and behaviour is the same - no preview shows, but image is uploaded and viewable after saving node.

I don't feel this is fixed. The fact that Image doesn't create duplicate "preview" images when the original is small enough is good, but not at all the problems this issue is trying to address.

The two current problems being addressed in this issue, as I see them, are:

1) When upgrading from earlier versions of Image and/or Drupal, either the files.filepath db entries that start with "image/" need to be updated to be prefixed with the file system path (be that 'files/' or a full absolute path), or better, the issue keeping relative paths from working resolved.

2) Previews of newly uploaded image nodes do not show the image preview, or thumbnail, or any sort of image, until you actually fully Submit the new image node (which appears to be related to same permissions issue of number 1, but can be split into a separate issue if needed).

Number 1 can be worked around with manual sql updates to your db to get a working site (not that one should expect average users to have to be doing that; or at least some instructions if they should).

Number 2 is an ongoing issue, making it difficult for contributors to upload new image content on an ongoing basis, and has no work-around at this time, and is a significant functionality issue (imo).

Some trouble shooting info for the Preview issue.

Steps to reproduce and what appears to be happening.

1) Create content -> Image
2) Selected "Test-Pic.jpg" to upload.
3) Clicked Preview.
4) Preview page comes up, starts to display boxes for preview and thumbnail images, but then reverts to alt text, with no images.
5) Checking (path)/files/images/temp/ the files Test-Pic.jpg Test-Pic.preview.jpg Test-Pic.thumbnail.jpg have all been created perfectly.
6) The urls the preview page are trying to display are http://domain.com/system/files/images/temp/Test-Pic.preview.jpg and http://domain.com/system/files/images/temp/Test-Pic.thumbnail.jpg
7) Trying to view the above urls directly gives a Drupal "Page not found" page, which I believe is what it will give when trying to view private files that don't have permissions.
8) A search of files.filepath shows zero matches for LIKE "%Test-Pic%" (whereas searches for existing images do return results).
9) Click Submit, and the node and images are working as they should.
10) The above search in 8 now returns the Test-Pic entries.

My guess is that during previewing, entries need to be added to the files db to allow the various images/temp/ files to be displayed, then removed/changed when the same is done to the temp files.
I suspect everything else is probably working as it should, as per previewing and submitting images go.

Just a suggestion of a quick fix that worked for me (assuming the problem I was seeing was actually the same - previews and thumbnails disappeared after upgrading from 5.1 and old image to 5.10 and 1.9 with private filesystem).

Go to administration|site-configuration|image

change thumbnail width to something new
save
change thumbnail width back to what it was
save

All derivative images get regenerated and thumbnails etc. reappear as if by magic ;)

Ah that sounds like an easier work around for the disappearing existing images. Well more work for the server, but certainly something easier an average user could do rather than going in and doing direct SQL. Probably in recreating all the images, it rewrites them correctly to the files db. I haven't tried it myself since already did the SQL tweaking.

But that probably doesn't help the preview of newly uploaded images.

I found #215792: Preview won´t show up when filesystem is set to private and although that is marked as a duplicate of this one find if confusing to mix the two problems up as they appear separate IMHO.

Anyhow, that said using the latest 6.x dev version (2009-07-13) this bug (the previewing part when creating image nodes) persists.

There is a not very elegant workaround one can do: create a system/files/images directory in your web root and then symlinking the temp directory from image private data there. This is a slight security issue but given images should be removed from there immediately they are posted if everything works well I can live with this as a temporary workaround.

Related to #504494: Broken path to images when creating new image node/image import.?

I'm confirming this.

I think the problem is that the filepaths are not stored relative to the filesystem directory in the database. So this may be a more serious problem, but I don't know... IMHO, we should be able to move the files/ directory, and specify a new position for it in the drupal interface, and everything should work (that's not what happened to me, but the idea is that **everything** filepath related should be done relative to the filesystem path that we specify in the interface under Site Configuration.)

And the snippet of sql code in comment 20 didn't help me. I ran it (adjusting as needed) and it did make changes, but nothing helped. I tried many different combinations of urls.

I had everything working locally, and uploaded my site, preserving relative file structure, etc. Uploaded the db, and none of my images are getting inserted in the posts as they should.

SO I re-created all the posts (stupid, I know, but it worked).

Next I upgraded my drupal install from 6.12 to 6.13, and upgraded a bunch of modules, including the image module.

Again, no images. This time, I don't want to go through all that hassle!

Finally I went to a test site, uploaded an image, and looked at the DB to see how the url was stored, and its like:
sites/mysite.tld/files/images/imagefile.jpg

So I took that format and re-formatted all of the entries in my non-working DB, and it STILL doesn't work.

The only thing that works is deleting all image nodes and remaking them. I tried that, and didn't see any differences in the url, but it still didn't work.

Finally, all the images are in their proper places, with proper permissions. This is really strange.

So what gives? What is the proper structure for the filepath field in the files table?

Thanks, and I hope this is fixed soon. Seems pretty bad that this bug has made it through from 4.7 to 6.13...

Possible Solution:
I found the problem:
during the upgrade, the table 'image' got emptied somehow. So even though all the stuff was there, it was corrupted. This doesn't explain why the images stopped showing up after the migrate, but at least I have a solution!

Anyway, Good luck!

Still got this problem at Drupal6 + image-6.x-1.0-beta5.
Is there some patch to fix this?

Problem was fixed at the following way

Go to
http://yoursite/admin/settings/file-system
and change public mode (HTTP) to private (Drupal handling)

Closing old bug reports.