FTP harvest files from a token folder instead of with a token filename append.

Made some good progress on this today, should have a patch tomorrow for review.

You might want to look at this closely, as the patch really is against BETA-1, so i think it take a backward step on a few things, sorry about that :(

I added 3 options to the config,

1) Harvest from token folders
2) Delete token folders after harvest
3) Delete any non token folders

The code to make this work is in inserted into a few places

1) media_mover_ftp_connect() : It now creates $connection['ftp_folders'] (if the correct config is in place) with an array for each folder in the base dir containing a ftp_rawlist of the folder.
2) media_mover_ftp_harvest() : it now loops though these extra folders in the same way as it did for the base folder
3) media_mover_ftp_token_extract() : it now checks for a folder token if it doesn't find a file token
4) mm_ftp_parse_file() : a minor modification to allow returns of directory information

I also added a helper function ftp_rmdirr() which was kindly donated by Aidan Lister (http://aidanlister.com/repos/v/function.ftp_rmdirr.php) to allow ftp directories to be deleted recursively (ftp_rmdir() requires the folder be empty). He is going to re-write it shortly to allow it to work on servers that dont support the LIST -R command.

What do you think?

there are some extra dsm()'s in there that i forgot to take out. will roll another patch if you want.

http://drupal.org/node/442078 is a good point, perhaps a pre-download check per file might be in order, or a wrapper for ftp_rawlist which does the double request check and only returns valid results.

Token gen yes could do with some work, i think the token is there for 2 reasons, to uniquely identify the user and to be able to change, so perhaps a simple 2 part key, part 1 based on the uid and part 2 is a random much shorter string. Currently there is no collision detection - right?

Re-rolled minus the dsm() statements with a new function called _media_mover_ftp_completedlist() which solves the http://drupal.org/node/442078 issue, with an extra flag to allow for "strict" compliance. This means when using folders, it wont return ANY results if anything in that folder is still uploading - this avoids problems later when the system deletes folders its harvested from.

I also took a simple shot at improving the token generation, its now adds the UID to a rand() (instead of time) for the seed, the generation is done inside a do-while loop to insure the key is unique then the old token is removed (so in theory it can be re-used). The issues we have still would be on VERY large sites that do-while loop could go on for some time, hitting a lot of duplicates, but thats more a function of the char limit. I'm sure a cryptographer or someone else might be able to take a better shot at this, but for me, it works.

1) Will do!
2) In media_mover_ftp_connect the first rawlist is for the base dir say "/" in then also get rawlists for any of the folders inside "/" e.g. "/123456", this is so later, when we harvest were not getting more listings adhoc, seemed "media mover"-y to get it all in one batch like that.
3) Looks ok, will test later as we are putting live the work i've done on it so far right now, but I will update to the approved version once its ready.
4) Looks good, im in IRC if you want a more interactive conversation.
5) Will do :)

in that last commit - you have a "print 'here';" ;)

http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/media_mover...

i will check it out when i get a free moment or 2. My alpha code has so far processed 4k uploaded files!! But I know your refactored code will probably be nicer.

2 things we've seen.

1) Recursive delete of folders is a pain, still for servers that dont allow recursive listing
2) Hidden files, starting with . which are sometimes generated by systems are not caught by my script so can also cause issues when trying to delete harvested folders.

Other than that, its been working really well.