I'm using this module to import images into galleries at:
http://sphop.perlucida.com/

The files are pre-sorted by name to be in the correct order, but they appear to be imported into the gallery apparently at random.

For example I imported a series of files with names:
sphp_2000_01.jpg
sphp_2000_02.jpg
sphp_2000_03.jpg
through to:
sphp_2000_39.jpg

(35 was missing somehow, so there were only 38 images)

They were imported in this order:
sphp_2000_26.jpg
sphp_2000_25.jpg
sphp_2000_23.jpg
sphp_2000_24.jpg
sphp_2000_12.jpg
sphp_2000_10.jpg
sphp_2000_11.jpg
sphp_2000_09.jpg
sphp_2000_08.jpg
sphp_2000_07.jpg
sphp_2000_06.jpg
sphp_2000_05.jpg
sphp_2000_02.jpg
sphp_2000_03.jpg
sphp_2000_04.jpg
sphp_2000_01.jpg
and so on, it's not creation date, not file size, not alphabetical...what is it sorting on?

Here is the resulting gallery:
http://sphop.perlucida.com/image/tid/60

There may be some math problems too:
On this import it claimed to have imported and deleted 18 of 18 images and no captions, when it had actually done 38!

CommentFileSizeAuthor
#11 image_import-sleep_throttle.patch534 bytesadrinux

Comments

syscrusher’s picture

syscrusher commented
Component: User interface » Code
Assigned: Unassigned » syscrusher

I believe I take the images in whatever order is presented by the filesystem. I only run Linux and other *NIX systems, which always sort at the o.s. level unless you go out of your way to tell them not to. You aren't, by chance, running a Windows server, are you?

I can add a sort into the import function, though. No big deal. :-)

Scott

adrinux’s picture

adrinux commented

This is a linux server, albeit a shared host.

I just did a second import, similar file names. It's a little more sane, but still:
http://sphop.perlucida.com/image/tid/61

A taxonomy list is similarly messy:
http://sphop.perlucida.com/taxonomy/term/61

I'm thinking that this may be a problem with the way Drupal lists nodes. All of these image nodes have the same creation time (at least as shown in the node view which only shows minutes not seconds), since these are usually listed by creation time is it possible that Drupal is simply unable to order them in the right order?

Is it theme related? Or theme engine related? By which I mean does the node time stamp need to be pulled from the db with seconds as well as hours:minutes?

Basically I'm thinking that this isn't a problem with image_import.module per se...

syscrusher’s picture

syscrusher commented

I'm thinking that this may be a problem with the way Drupal lists nodes. All of these image nodes have the same creation time (at least as shown in the node view which only shows minutes not seconds), since these are usually listed by creation time is it possible that Drupal is simply unable to order them in the right order?

Very good thought, and I'll bet that's exactly what's happening. I think Drupal sorts only on the timestamp, and doesn't sub-sort on title. I can easily change my code to put the exact timestamp on each image, but on any reasonably-fast server, multiple images are imported per second, so this won't help much. I'll do it anyway, just because it's "the right way to do things", but I'm not optimistic it will fix your error.

Is it theme related? Or theme engine related? By which I mean does the node time stamp need to be pulled from the db with seconds as well as hours:minutes?

Nope -- the database stores an "integer number of seconds since the epoch" timestamp internally, and the displayed format has no effect on ORDER BY clauses in the node fetch SQL.

Scott

syscrusher’s picture

syscrusher commented

This is now fixed in the latest CVS version.

Thanks for the suggestion!

Scott

syscrusher’s picture

syscrusher commented

Ummm....Let me clarify that word "fixed". It's as fixed as Drupal will let it be. I sort the names of the files before import, and I apply a to-the-second-precise timestamp. If the system imports multiple files per second, though, I have no control over the sort order. This may be something to suggest to the core team, that nodes in listings sort by title if their timestamps are equal.

Scott

lewiz’s picture

lewiz commented

Would it not be possible to sort by filename and *then* by filestamp? That way images all with the same timestamp with filenames like 001, 002, 003 will all nicely be in order?

syscrusher’s picture

syscrusher commented

Sorting by name is exactly what I do already. It's Drupal core that sorts by timestamp.

My current code sorts by name, then imports in that order. The problem is that on many systems, more than one image can be imported per second, so they get the same timestamp. Drupal sorts by timestamp first; I don't know if it has a secondary sort key or not.

I don't think it's within my control to fix, unfortunately.

Scott

adrinux’s picture

adrinux commented

This does seem to have improved. But then again I'm noticing that it works better with larger images. Presumably it's small images that get imported at the rate of more than one a second!

syscrusher’s picture

syscrusher commented

That is a correct assumption.

I had another thought on this: The gallery display is under the control of image.module. Perhaps we can persuade walkah, maintainer of image.module, to change the sort order in galleries. I'll write to him (I have his direct email address), but you may want to add (politely, please) your request also. :-)

Scott

adrinux’s picture

adrinux commented

But sort on what? In my case I'm not intending that the titles should remain as they are so sorting on title is not really an option, and having to add a number to the start of each title to make them sort properly would also not be a good solution.

Drupal's standard node weighting system would be really unwieldy in a large gallery, so that's out (it also goes against the idea of image.module remaining stripped down and simple with better gallery functionality being added elsewhere).

I think the order of import - and thus timestamp - is a good solution for sorting, I'm considering whether it would be possible to throttle the import of images to one per second.
Actually someone on #drupal mentioned that importing images sucked up a lot of resources too.

I'm changing this to a feature request :)
(optionally) throttle import of images

adrinux’s picture

adrinux commented
StatusFileSize
new image_import-sleep_throttle.patch534 bytes

Here's a single line patch that crudely implements a throttle by sleeping for 1 second during each image import.
I don't doubt that a better programmer could come up with a better solution :)

This works well to ensure all images have different timestamps and thus sort nicely according to the creation timestamp in galleries and taxonomy listings.

syscrusher’s picture

syscrusher commented

Hi, adrinux, and thanks for the patch.

As you can see from issue 23839, I've done a major rewrite of the main parts of the module. Your patch as it stands won't work any more, because those lines of code are totally gone.

I'm inclined to accept your basic concept, though, but to make it an admin-settable option. I'll try to get that feature into the module quickly, but tonight I was entirely focused on the rewrite to improve reliability.

Kind regards, and thanks again for the testing help,

Scott

syscrusher’s picture

syscrusher commented

A variant of the throttle mechanism proposed by adrinux is in the 2005-06-06 version of image_import.module. This ensures that each image gets a timestamp that differs by at least one second from the previous image.

Scott

Anonymous’s picture

(not verified) commented