I know that the Drupal 6 branch is closed, but I thought I would offer some code I wrote as an alternative to the way drealty imports photos.

I had major issues with file_copy not copying the placeholder over. Sometimes it would, sometimes it wouldn't. It seems like everything works great on the first import, but subsequent imports cannot recopy the placeholder even though it is told to with FILE_EXISTS_REPLACE. Since dr_images is cleared out during listing updates, assuming photos would be imported correctly, the node is saved. When file_copy fails, dr_images is not populated, thus listings show no images.

I literally spent 8 hours yesterday trying half a dozen ways to make the image placeholder copy method work. I tried using regular copy(), as well as debugging file_copy() with watchdog. It -seems- that the whole thing fails because $fclone does not clone correctly and/or realpath($source) (in file.inc) cannot find the file, so returns blank, which errors out. I could not figure out why and have never had an issue with the file system or FILE_EXISTS_REPLACE until now. All permissions etc are correct, all file owners are correct.

I figured that there really isn't a need to copy placeholders and deal with all that trouble. I came up with another method that seems to work great so far.

I implemented a nodeapi call when the node is viewed to fetch images, which is basically the same as the imagecache action. I have had a lot of success with this so far in the last hour of testing. I am offering up my code to help others who may be stuck in the same spot.

In drealty.module, add this nodeapi call:


function drealty_nodeapi(&$node, $op, $a3 = NULL, $a4 = NULL) {
  switch ($op) {
    case 'view':
      if ($node->type == 'drealty_property' && $node->dr_photocount[0]['value'] > 0 && !is_array($node->dr_images[0])) {
        $node->dr_images = array();
        
        include_once "phRets/phRets.php";
        $phRets = new phRets();
        
        if ($phRets && $node->drealty_property_conid[0]['value']) {
          if (drealty_rets_connect($phRets, $node->drealty_property_conid[0]['value'])) {
            watchdog('drealty_imagecache', 'Attempting to fetch photos for listing @id.', array('@id' => $node->dr_listingid[0]['value']), WATCHDOG_NOTICE);
            $result = $phRets->GetObject("Property", "Photo", $node->dr_listingid[0]['value'], '*', 1);
          }
          
          if (!$result[0]['Success']) {
            watchdog('drealty', 'An attempt to fetch photos failed for listing @id.', array('@id' => $node->dr_listingid[0]['value']), WATCHDOG_ERROR);
            return;
          }
          
          $upload_directory = file_directory_path() . '/drealty_img';
          
          if (!is_dir($upload_directory) && !mkdir($upload_directory, 0755)) {
            watchdog('drealty', 'Directory @dir doesn\'t exist and couldn\'t be created (probably a permissions problem).', array('@dir' => $upload_directory), WATCHDOG_ERROR);
            return;
          }
          
          foreach ($result as $photo) {
            $listing = $photo['Content-ID'];
            $number = $photo['Object-ID'];
            
            $filename = $listing . '-' . $number . '-' . $node->drealty_property_conid[0]['value'] . '.jpeg';
            $destination = $upload_directory . '/' . $filename;
            
            $file['uid'] = 1;
            $file['timestamp'] = time();
            $file['filename'] = $filename;
            $file['filemime'] = $photo['Content-Type'];
            $file['status'] = FILE_STATUS_PERMANENT;
            
            $result = file_save_data($photo['Data'], $destination, FILE_EXISTS_REPLACE);
            
            if ($result) {
              $file['filesize'] = filesize($destination);
              $file['filepath'] = $destination;
              
              //write record in files table
              drupal_write_record('files', $file);
              $new_file = field_file_load(db_last_insert_id('files', 'fid'));
              
              if ($new_file) {
                $node->dr_images[] = $new_file;
              }
            } else {
              watchdog('drealty', 'Could not save photo @photo for listing @id', array('@photo' => $filename, '@id' => $listing), WATCHDOG_ERROR);
            }
          }
  
          node_save($node);
          cache_clear_all();
          drealty_disconnect($phRets);
          
          // reload the page so the images are viewable immediately instead of on the second view
          watchdog('drealty', 'Reloading node for listing @id to display images.', array('@id' => $listing), WATCHDOG_NOTICE);
          drupal_goto(request_uri(), array('stop_redirect' => true));
        }
      }
      break;
  }
}

Change the drealty.batch.inc file to the following. My logic here is that when a listing is being updated, wipe out its dr_images record, and delete and images associated with the listing, since we will get them the next time the node is viewed:

around line 394:


elseif ($_fields[$field]['correlation'] === 'pic_count' && $_file/*  && $_placeholder */) {
          unset($node->dr_images);
          
          // clear out existing images, we pick them up the next time the node is viewed
          for ($index = 1; $index <= $value; $index++) {
            $filename = "{$prop[$id]}-{$index}-{$conid}.jpeg";
            $path_to_file = file_directory_path() . '/drealty_img/' . $filename;
            
            if (file_exists($path_to_file)) {
              watchdog('drealty', 'Deleting file @file', array('@file' => $path_to_file), WATCHDOG_NOTICE);
              $success = file_delete($path_to_file);
             	if ($success) {
                db_query('DELETE FROM {files} WHERE filepath = "%s"', $path_to_file);
              }
            } else {
              watchdog('drealty', 'No file found for: @file, nothing to delete.', array('@file' => $path_to_file), WATCHDOG_NOTICE);
            } 
          }
        }

I am sure this code is by no means perfect but its getting me out of my jam and hope it helps others. If anyone has suggestions to improve this please share. I have not noticed any issues so far (load times, etc).

Comments

kevinquillen’s picture

Also, the code above only affects properties. You'd have to add to it for offices or agents, for example, which I am not using at the moment. After implementing this, I also had no need for the imagecache action on my presets, since we've already fetched the images.

camidoo’s picture

Thanks Kevin! Like you said we can leave this here for others to view if they are having problems as well.

In the 7.x branch this is a non-issue, as there are no placeholders used, and on updates i just remove "all" the images associated with a listing and mark that listing as needing it's images refreshed.

For now, all the images are downloaded / pre-fetched from the RETS feed. This decision was made because listings with "many" listing photos, 25+, were taking an inordinate amount of time to load due to the fact that the images were being fetched and processed with image actions or what not. Additionally, in the event of heavy load on the RETS server or the RETS server being unavailable there would be a lengthy wait, or no images at all.

Having said that, which do you prefer? The on-demand loading of the images or the pre-fetching of all the images? The pre-fetching does take a considerable amount of time on the initial import of the listing data. However, once the initial import has taken place, it really becomes a non-issue.

Would love to hear comments, opinions, and thoughts on whether or not there should be an option to load images on demand or pre-fetch them. This could very well tie into #1296680: Image retrieval Hot Linking vs. Downloading discussion in the 7.x branch.

Additionally, would you have any interest in maintaining the 6.x branch? I'm not sure how many people are even using it, and if it's even worth maintaining. And do you have any interest in helping with the 7.x branch?

kevinquillen’s picture

It probably depends on hosting resources? I am running a site on a dedicated box so I don't see any issue with what I have done above. The only issue is when the listing (with the code above) is seen, drupal_goto probably triggers two hits to analytics trackers. I have not thought about how to solve that yet.

I would think that doing file operations for placeholder files is unnecessary, which is why I gutted it. I just could not figure out why file_copy kept failing.

I think the data import should focus on getting only static data in, with the photo count. When a node is loaded or viewed, thats when it should reach out for photos, and it would know to do that based on the photocount value on the node.

Although, I don't think hotlinking is bad either, which would be the fastest case if the RETS vendor allowed it. Then again, if the server is unavailable, thats an issue. But what are the odds of that?

kevinquillen’s picture

As for the branch, we use d6 DR for one site and will likely use D7 for future projects. This was really the only issue we had other than the connection issue which was patched in another thread.

kevinquillen’s picture

Actually, you know what? You can make me a maintainer for the D6 version.

kevinquillen’s picture

We also use Innovia, and all our images are 290x210 and no bigger which makes them very small and quick to download. I've read other vendors allow much larger images, so this method can be slow on first load.

Is it normal for images to be large, or are they typically small for most vendors?

camidoo’s picture

In my 'limited' experience i'm finding that most boards allow medium sized images, mostly 640x480 ish.

And i'll go ahead and give you maintainer status / access, and thanks for taking that on! If you want to chat / talk hit me up on my cell, (it's not too hard to find the number).

kevinquillen’s picture

Sure, add myself and user 'inclind' (coworker). Thanks.

camidoo’s picture

Status: Active » Closed (fixed)