Every time I run an update, everything populates in the node, but the only issue I'm encountering is this error I'm seeing in watchdog:

"File sites/default/files/pirets_img/912917/912917-1.jpeg couldn't be saved (probably a permissions problem)."

It happens for almost every property that imports (and the right number of log entries show up based on the image count)—but occasionally 1 or 2 properties will generate the files fine but they'll appear in the root of the "files" directories. All of the permissions and ownership are set correctly in the files directory.

Anyone have any ideas?

Jake

Comments

Garrett Albright’s picture

Garrett Albright commented

All of the permissions and ownership are set correctly in the files directory.

Define "correctly."

Remember that the directory needs to be writable by the user that the web server daemon is running as (in most cases).

If you go crazy and do a chmod -R 777 on the files directory, do things still not work?

jakecamara’s picture

jakecamara commented

:) Yeah, so by "correctly" what I meant was that I went hog wild and did a chmod -R 777 on the files directory to see if that would clear it up, but the same thing happened. It does create all of the directories of MLS IDs in the pirets_img directory, but they all end up empty...

jakecamara’s picture

jakecamara commented

Just bumping this. Any thoughts?

Garrett Albright’s picture

Garrett Albright commented

Sorry, no. I'm out of ideas…

madrock9’s picture

madrock9 commented

I've got this to using dRealty. [but it's based on PIRETS, I couldn't connect PIRETS to my MLS]. I am going nuts. I'm using dreamhost, and for a while i used drush from command line to import the listing, under fcgi; then, when I got tons of errors, dreamhost suggested to move for cgi. This went well [no more internal server eroors/404, no more premature end of headers]; but my pictures would not get copied on the server. Well, some did, some didn't. I imagined that that's because the file_copy takes longer to copy the files, or the connection is slow.
Than I decided not to import the pictures at all, and just use the placeholder, and than leave imagecache to fetch the pictures on demand. This worked a bit [first import], then for some, it would create the node correctly with images placeholders. For some [very large percent of them would not]; the error is "File sites/default/files/drealty_img/21025552-1.jpeg couldn't be saved (probably a permissions problem)".

This error is very vague, it has permission [unless dreamhost has some other deamon running that changes permissions? for security purposes?] to create the files [and come on, it creates some]. I've read somewhere that I have to change a line in image.api, than to disable the image attach for the content type, but nothing help. Basically, what happen is the images are not added to the node.

So if any of you guys do not have this problem, can you tell me which versions of modules [imagecache, imageapi, filefield, etc ] are you guys using? Or maybe what would be your hosting recommendations?
Again, this is for drealty, which is based on Pirets [this part is identical].

Thank you so much!

Garrett Albright’s picture

Garrett Albright commented

Dreamhost is a shared hosting provider. For both your and their sakes, I really don't recommend running PIRETS on a shared hosting provider.

That aside… Sorry, I'm not sure what else may be going wrong.

madrock9’s picture

madrock9 commented

Hi - I think I got this, at least part of it.
Diable image attach on the Content Type. For my Property Content Type, I configured the Path Settings of the Image directory to the correct path where my property images are saved. And rets import is working, I still don't get all the images, I wonder if the file sizes are larger than 7M [whatever I did, I didn't manage to change this setting].

These changes, they make sense: running drush rets-import => my contents are created by Anonymous User. I set up the file uploading directory images/%uid for user, admin can do whatever he wants, but no setting for where images uploaded by Anonymous to be saved. So I think it was a permission conflict after all.

I managed to set up a cron job that runs only drush for import, and so far so good. I probably messed up the batch file :-( .
Is there any way to not save the node if the image copy/retrival fails? In my case, for the few listings that don't have images, the image data is there, but the image itself not.
Anyway, I hope this will save somebody some time, as it might not be the problem, it is something I overlooked!

Thanks again!

HallSL’s picture

HallSL commented

Some additional data for this intermittent image permissions problem. When we manually update the class, all is good. When we run through cron or cron/drush, even when having drush run as a particular user, sometimes we get images and sometimes we get the permissions complaint.

The additional piece of info is that IF we keep a browser open and logged in as either user 1 or the user set up for cron, we get images from the cron run. Logout completely, and the drush/cron run makes the nodes (with the appropriate user shown in the log), but we get the permissions problems and no images. We don't interact at all with the update, it's being run via drush/cron through cpanel.

This makes no sense to me, but I'm hoping it helps Garrett figure it out...

Garrett Albright’s picture

Garrett Albright commented

We don't interact at all with the update, it's being run via drush/cron through cpanel.

Hmm. Can you ascertain what user cron jobs are being run as when they're configured through cPanel? May be a question for your hosting company.

HallSL’s picture

HallSL commented

Sorry, Garrett, before I bug support, let me make sure I understand the question. So far as we could tell, cron jobs are anonymous and thus cannot do things that require a user. That's why we set up the drush/cron (at your suggestion) to work as a logged in user, uid 3. And that user is shown in the log as doing the things, successful or not, that get done on the cron job. Perhaps worthy of note -- the nodes still show anonymous authorship.

Does this answer the question or can you explain the proper question for me to ask?

I'd appreciate any brainstorming you could do about why the process works flawlessly if we push manual update, fine if we run cron manually, but very spottily if cron is triggered automatically. We always get the data, but sometimes we get the placeholders, sometimes not. Generally (and I'm still examining this) if several items in the same class are added, they either all will or all won't get their placeholders. However, another class in the same run might or might not succeed. It seems that the larger classes (and we're only really trying two of the smaller ones right now) fail more often. None of this seems to make sense to me since it would seem that the process is always the same and not dependent on the connection since it's just copying that placeholder file. Any thoughts about how it could fail frequently and randomly?

Garrett Albright’s picture

Garrett Albright commented

Okay, there's two concepts of "user" here. One is the Drupal user; the other is the user according to the operating system, which is the one I'm mostly focusing on here. Without delving too much into the guts of Unix, every process that runs does so under a user account of some sort; for example, cron processes. And usually that ownership stays the same for processes started by that process. So if you're using cron to execute commands with Drush the PHP process which runs Drush and therefore creates files will probably run under the same user account that cron is running under, so the files will be identified as being created by that user. Now it may be that that user does not have permission to copy the placeholder image (they do not have permission to read the placeholder image and/or permission to write to the directory where it should be placed; though the latter is unlikely because the same user likely created that directory).

As for why this is happening seemingly randomly, I have no idea, but one thing to check would be to see if the directory that PRIETS wants to move the placeholder image into is already created at the time of failure. Completely flushing your listings and emptying out the RETS image directory may help you make sure you're starting at the same point for every listing.

ptoly’s picture

ptoly commented
Category: support » bug

I've run out of time for this today but here is what I found when running the script through my debugger:

In my case it's actually not a permission issue but the file simply does not exist!

When I'm running through this section of code during a batch process (line 331 pirets.batch.inc):

if (file_copy($fclone, $prop_dir, FILE_EXISTS_REPLACE)) {
                  $_file->filepath = $dest;
                  drupal_write_record('files', $_file);
                  $_placeholder['fid'] = $_file->fid;
                  $_placeholder['filename'] = $fname;
                  $_placeholder['filepath'] = $dest;
                  $node->pirets_images[] = $_placeholder;
                }
                else {
                  watchdog('pirets', 'File @file couldn\'t be saved (probably a permissions problem).', array('@file' => $dest), WATCHDOG_ERROR);
                }

What is actually causing the watchdog entry is not a permissions problem but the core function file_copy from line 262 of file.inc (see line 287).

From what I understand the PIRETs module is just supposed to copy a 1x1 pixel file there and rename it, right? It's not for some reason.

Hope to get more time on this later this week. In the meantime I hope this helps someone.

Garrett Albright’s picture

Garrett Albright commented

ptoly: Yes, inside the PIRETS module directory, there should be an "img" directory, and inside there, a "pirets_blank.jpeg" file. Do you see it?

If not, I suggest downloading a new copy of PIRETS.

ptoly’s picture

ptoly commented

Yes I see that file. Funny that it's not calling for it. Instead it's calling for a 1x1 pixel file at the root of the files directory.

Cheers

Garrett Albright’s picture

Garrett Albright commented

Whaaaaaat.

Well, I guess as a dumb workaround, you could copy the file to where it's looking for it…

ptoly’s picture

ptoly commented

Well, renaming and moving several thousand 1x1 pixel images to the coded name would be a tad impractical... :)

It seems that the process involves gathering an image, creating a coded name for it, then moving and renaming a placeholder 1x1 pixel image into it's place, until it's called by the imagecache preset.

What's not happening is that script is not correctly recreated the generic 1x1 pixel file into the placeholder...

Anonymous’s picture

Anonymous (not verified) commented

Has anyone figured out why this is happening?

I changed line 342 of pirets.batch.inc to read:
watchdog('pirets', 'File @file couldn\'t be saved (probably a permissions problem). fclone: @fclone, prop_dir: @prop_dir', array('@file' => $dest, '@prop_dir'=>$prop_dir, '@fclone'=>$fclone->filepath), WATCHDOG_ERROR);

fclone is coming up blank. Reading the docs on file_copy it says it should be an object with 'filepath' as a property.

Anonymous’s picture

Anonymous (not verified) commented

I serialized $fclone and $prop_dir before file_copy was run and below is what i received.

fclone: O:8:"stdClass":8:{s:8:"filemime";s:10:"image/jpeg";s:8:"filesize";i:0;s:3:"uid";i:0;s:6:"status";i:1;s:9:"timestamp";i:1301085130;s:8: "filepath";s:60:"sites/default/files/pirets_img/B091/B0913394/B0913394-4.jpeg";s:8: "filename";s:16:"B1003889-12.jpeg";s:3:"fid";s:6:"765212";},

prop_dir: sites/default/files/pirets_img/B100/B1003889

If the source should be pirets_blank.jpeg, thats not what it's getting. I'll try and trace back where it is getting set.

Anonymous’s picture

Anonymous (not verified) commented

After spending some time on this today (3 hours!) I think i understand a little more, but still as lost as ever. I hope someone can help shed some light here.

Here is what i know. When you flush all the listing and re-import, you get all of the images. As the days go by you start to see less and less images. Apparently it is only (for me) and update problem.

Here is the code in question (Line 321 of pirets.batch.inc)

for ($index = 1; $index <= $value; $index++) {
	// Save the placeholder for each image attached to this listing.
	$fname = "{$prop[$_mls_field['system_name']]}-{$index}.jpeg";
	$dest = "{$prop_dir}/{$fname}";
	// Since file_copy stupidly doesn't let us specify the destination
	// path explicitly, stopping us from copying a file so it has a
	// different filename than the source, we do this workaround.
	$_file->filename = $fname;
	$fclone = clone $_file;
	if (file_copy($fclone, $prop_dir, FILE_EXISTS_REPLACE)) {
		$_file->filepath = $dest;
		drupal_write_record('files', $_file);
		$_placeholder['fid'] = $_file->fid;
		$_placeholder['filename'] = $fname;
		$_placeholder['filepath'] = $dest;
		$node->pirets_images[] = $_placeholder;
	}
	else {
		watchdog('pirets', 'File @file couldn\'t be saved (probably a permissions problem).', array('@file' => $dest), WATCHDOG_ERROR);
	}
}

Now, the problem i think is that after the first picture of the first listing update (which i believe is successful) the constant variable that has the filepath to pirets_blank.jpeg is set to the $dest so the rest of the updates fail. Below are the changes to the code i made, but still don't work. Here is where i need light shed.

for ($index = 1; $index <= $value; $index++) {
	// Save the placeholder for each image attached to this listing.
	$fname = "{$prop[$_mls_field['system_name']]}-{$index}.jpeg";
	$dest = "{$prop_dir}/{$fname}";
	// Since file_copy stupidly doesn't let us specify the destination
	// path explicitly, stopping us from copying a file so it has a
	// different filename than the source, we do this workaround.
	$_file->filename = $fname;
	$fclone = clone $_file;
	if (file_copy($fclone, $prop_dir, FILE_EXISTS_REPLACE) != 0) {
		$fclone = drupal_write_record('files', $fclone);
		$_placeholder['fid'] = $fclone->fid;
		$_placeholder['filename'] = $fclone->filename;
		$_placeholder['filepath'] = $fclone->filepath;
		$node->pirets_images[] = $_placeholder;
	}
	else {
		watchdog('pirets', 'File @file couldn\'t be saved (probably a permissions problem).', array('@file' => $dest), WATCHDOG_ERROR);
	}
}

To explain my changes...

  • I think as long as we don't change $_file->filepath the script will work, so we set the filename to that file_copy will make the copy with the correct name instead of a bunch of pirets_image.jpegs in the directory.
  • before file_copy we clone the $_file object to $fclone (this for some reason always fails for me) but i made other changes which i would like someone (Garrett, time willing) to review
  • when passed an object for the source, file_copy should return us back a populated object, correct? so... i wasn't sure if it would pass as true so to be sure about this i changed it to != 0 (this is probably a noob move, but that was in the 2 hour of messing with this)
  • also since drupal_write_record should pass back an obj with unique ids so i used that to populate the $_placeholder array

Anyway, I hope i'm not wasting my time here. There is a lot of this i just don't understand.

Anonymous’s picture

Anonymous (not verified) commented

I finally got this working again. Thankfully without having to flush and re-import all the listings. I gave up on drupal file_copy. For some reason it wasn't working. I will go back later and figure out why that is. For now, this is what is getting the images in the site for me.

for ($index = 1; $index <= $value; $index++) {
	$fname = "{$prop[$_mls_field['system_name']]}-{$index}.jpeg";
	$dest = "{$prop_dir}/{$fname}";
	$fclone = clone $_file;

	if (copy($fullpath . $fclone->filepath , $fullpath . $dest)) {
		$fclone->filepath = $dest;
		$fclone->filename = $fname;
		drupal_write_record('files', $fclone);
		$_placeholder['fid'] = $fclone->fid;
		$_placeholder['filename'] = $fclone->filename;
		$_placeholder['filepath'] = $fclone->filepath;
		$node->pirets_images[] = $_placeholder;
	}
	else {
		watchdog('pirets', 'File @file couldn\'t be saved (probably a permissions problem).', array('@file' => $dest), WATCHDOG_ERROR);
	}
}

NOTE: When I finally got the placeholder images copied to the right spot, i went and looked at the listing page and the 1px image was showing instead of the imagecache version from the RETS server. The problem was that I had created my own 1px placeholder image at some point and it was a different size and this is how it detects if the image has been copied to the server yet. I just updated the pirets.imagecache.inc to reflect the size of my image.

Anonymous’s picture

Anonymous (not verified) commented

I made some additional updates to this section of code, that seem to be working a lot better. Again, this starts on line 321 of pirets.batch.inc. I'm hoping this is my last post to this issue.

for ($index = 1; $index <= $value; $index++) {
	// Save the placeholder for each image attached to this listing.
	$fname = "{$prop[$_mls_field['system_name']]}-{$index}.jpeg";
	$dest = "{$prop_dir}/{$fname}";
	// Since file_copy stupidly doesn't let us specify the destination
	// path explicitly, stopping us from copying a file so it has a
	// different filename than the source, we do this workaround.
	$fclone = clone $_file;
	$fclone->filename = $fname;

	if(file_copy(&$fclone, $dest, FILE_EXISTS_REPLACE)){
		$fclone->filepath = $dest;
		if($fclone->filename!=""){ 
			drupal_write_record('files', $fclone);
			$_placeholder['fid'] = $fclone->fid;
			$_placeholder['filename'] = $fclone->filename;
			$_placeholder['filepath'] = $fclone->filepath;
			$node->pirets_images[] = $_placeholder;
		}
	}
}
kevinquillen’s picture

kevinquillen commented

I use dRealty, but had this issue. I could not get file_copy to work at all. As others stated, the first import works fine, but after that, images get lost since file_copy fails.

I posted an alternative method for getting images on the fly here:

http://drupal.org/node/1314536

It does not deal with copying placeholder images or anything. I have had 100% success getting images with this (note that with our Innovia vendor, they are no bigger than 290x210 source image, YMMV).

Anonymous’s picture

Anonymous (not verified) commented
Status: Active » Closed (won't fix)