I am importing 20,000 of 100,000 products now in a batch. I used the same image URL for groups of 2000 or so products that have different attributes that do not affect the picture used.

I would assume the file ID for the image should be duplicated from new item to new item down the list to reduce files on hand and clutter to name a few reasons,

But the upload is taking 4 hours because each item being imported is taking that same URL, downloading the image, giving the image a file ID in the image table, and then referencing the file ID to the product ID.

Why can't we just specify the same file ID among multiple products? Or even a hosted file on Amazon or Photobucket, why a single image for each product? 100,000 is a lost of the same image!

Not sure if this is drupal commerce or feeds, but I am noticing it now as I have been using your module all day.

Charlie

Comments

rfay’s picture

rfay
he,him
Primary language English
Location Palisade, CO, USA
commented
Title: new image for each product » How to avoid processing/downloading a new image for each input record
Project: Commerce Feeds » Feeds
Category: feature » support

This is really a feeds question, so moving to Feeds.

Is there a way to specify a feed so that new images aren't replicated?

rfay’s picture

rfay
he,him
Primary language English
Location Palisade, CO, USA
commented

If there's no good answer it can be turned back into a feature request.

cesanford30’s picture

cesanford30 commented

I just confirmed that by importing the products without images 20,000 takes 5 minutes as opposed to 8 hours. Then you must go in the database and massage 3 tables, and basically tell the entity ID to look at the existing image file ID. Works for me, so should be able to automate. But using a URL instead of a URI is another story.

rfay’s picture

rfay
he,him
Primary language English
Location Palisade, CO, USA
commented

Thanks, @cesanford30, but please give the complete recipe of what you did for future students of this technique. Hoping we can do better, but your recipe may be "the thing".

carn1x’s picture

carn1x commented

subscribe

5n00py’s picture

5n00py commented

Hi all, I have same issue.
I have ~4k items for import.
For first import feeds download images for all nodes, this take few hours.
But i need update products db daily without re-downloading images.
And i need some feature to re-download updated images.

I can create my own mapper for file/image fields and hadle this with custom code.
But I think we need more complex solution for this issue.

Someone know how to check for image(or any other file) changes via http/ftp ??

5n00py’s picture

5n00py commented

I made some research and code review.
1. feeds use cache table for store response info about each file download. Before re-download file feeds check http headers for file update.
But feeds need configured CURL for this, so first check this.
And be care while do drush cc all ))) I think feeds need personal cache table for this.
2. Feeds is realy good. Most time is spent on entity hooks.
So now im loking how to boost entity save/update actions...

5n00py’s picture

5n00py commented
Status: Active » Closed (works as designed)

Ok, perfomance with configured CURL looks good.
2Author: Feeds has great api, you can write your own plugin to handle items with the same images URL.

socialform’s picture

socialform commented
Issue summary: View changes

This may help in the situation where you have the image URL on Amazon S3 or even locally and don't want it duplicating.

After many days hunting, I found a way to reference files remotely with Feeds... so that it doesn't have to download the files. It has been a while but I used a combination of Feeds, Remote Stream Wrapper, Remote File Source, and the kicker... Media Feeds. Without Media Feeds it is not possible so even if you don't need to use Media your stuck enabling it. Media Feeds allows you to reference remote files using the Remote stream wrapper module. It gives you a new option in Feeds. Hope it saves someone time as this was one of the many threads I looked through for answers.