Please Update Programmatic Documentation

Following this: http://drupal.org/node/330421 as is does not quite cut it when doing advanced things and trying to make sense of all the comments can lead to mistakes.

It took me a while to figure out how to do this correctly and get this working, so where is what I did and how I did it, and why I did it.
Feel free to use this and make any modifications/corrections.

My Scenario
I needed a way to give an end-user a way to upload multiple files (lets say from as few as 1 to as many as 1000).
With the current implementation, clicking add another 1000 times should be an obvious problem.
The solution is HTML 5, but that is not supported here yet and if I am correct HTML 5 is not yet officially released. (specs could change..)
This means that an HTML 5 drag+drop multiple file (such as all files in a dragged and dropped directory) will not work for the time being.

My solution is to have the users compress/zip a file and make a single (large) upload.
The problem here is that the data must be presented in such a way that there must exist 1 node for each file in that zip.
A second problem (that is out of scope of this post) is the php size limit with arbitrarily large zip files.

This is where programmatic creation of filefield entries come into play.
I found some good example code from the importer module: http://drupal.org/project/importer that handles processing of a zip file and I used that code as a starting point.
In this situation I have three content types:

1. simulated folder
2. single document
3. multiple documents

The "simulated folder" is there for organization and will be referenced via the node reference module.
The "single document" is the content type that will contain a single uploaded document as well as a reference to a single "simulated folder".
The "multiple documents" is the content type that will accept only zip files that are to be expanded into multiple "single document" nodes each referencing a single "sumulated folder" as specified in the "multiple documents" node.

I am using the rules module with that will react to the "multiple documents" content type on create and on delete triggers.
When a "multiple documents" node is created, the rule will be called and the custom php snippet presented below will be used to process the zip file and create each individual "single document" node.
The delete event will not be discussed here because that may also be out of scope of this thread.
However, for the record, the one could use node reference such that all nodes created from this zip file are referenced by the given "multiple documents" node and that can be used to delete all such files when the "multiple documents" node gets deleted.

My working code

<?php
  function process_zipfile(){
    /* Unzipping code copied from: http://drupal.org/project/importer */
    $zip_filename = '[node:field_custom_00003_00002-filefield-filepath]';
    $real_destination = 'sites/default/files/some_subdirectory/';

    if ($z = zip_open($zip_filename)) {
      $temp_destination = file_destination(file_directory_temp() . '/' . basename($zip_filename, '.zip'), FILE_EXISTS_RENAME);

      mkdir($temp_destination, 0750);
      while ($entry = zip_read($z)) {

        if (zip_entry_open($z, $entry, 'r')) {
          $zip_entry_filesize = zip_entry_filesize($entry);
          $entry_name = zip_entry_name($entry);
          $data = zip_entry_read($entry, $zip_entry_filesize);

          $directory_name = $real_destination . dirname(drupal_strtolower($entry_name));
          if (!file_exists($directory_name)){
            mkdir($directory_name, 0755, TRUE);
          }

          // enforce directory permissions because the recursive mkdir does not do such when creating a directory
          chmod($directory_name, 0755);

          if (is_dir($entry_name) || preg_match('/\/$/', $entry_name)){
            zip_entry_close($entry);
            continue;
          }

          if ($file_path = file_save_data($data, $real_destination . drupal_strtolower($entry_name))) {
            $custom_file = new stdClass();
            $custom_file->uid = $user->uid;
            $custom_file->filename = basename($file_path);
            $custom_file->filepath = $file_path;
            $custom_file->filesize = filesize($file_path);
            $custom_file->filemime = file_get_mimetype($file_path);
            $custom_file->timestamp = time();
            $custom_file->status = FILE_STATUS_PERMANENT;
            $custom_file->uid = $user->uid;

            drupal_write_record('files', $custom_file);

            $loaded_file = field_file_load($file_path);

            $custom_node = new stdClass();
            node_object_prepare($custom_node);
            $custom_node->type = 'custom_00001';
            $custom_node->field_custom_00001_00001 = array($loaded_file);
            $custom_node->field_custom_00001_00003 = array(array('nid' => '[node:field_custom_00001_00003-nid]'));
            $custom_node->title = $desired_entry_name;
            $custom_node->body = '';
            $custom_node->uid = $user->uid;
            $custom_node->status = TRUE;
            $custom_node->promote = FALSE;
            $custom_node->active = TRUE;

            $custom_state = array('values' => (array) $custom_node);
            $custom_state['values']['op'] = t('Save');
            $custom_state['values']['name'] = $user->name;
            drupal_execute('custom_00001_node_form', $custom_state, $custom_node);
          }

          zip_entry_close($entry);
        }
      }

      zip_close($z);
    }
  }

  process_zipfile();
?>

Breakdown of what I did

<?php
    $zip_filename = '[node:field_custom_00003_00002-filefield-filepath]';
?>

Whats happening here is that I am utilizing the token module as provided via the rules custom php action.
The use of token code here allows me to avoid having to go through the discovery process for finding out where the data is stored and how to get to it.
This will end up with the full pathname of the zip file so that I can open and process the zip file.

<?php
    $real_destination = 'sites/default/files/some_subdirectory/';
?>

I really do not know how I should be handling this part, but doing it this way is working.
I am open to suggestions.
What is happening here is that there are some cases where I seem to have to specificy the entire path to where my files will be stored as well as any subdirectories.

<?php
          $directory_name = $real_destination . dirname(drupal_strtolower($entry_name));
          if (!file_exists($directory_name)){
            mkdir($directory_name, 0755, TRUE);
          }

          if (is_dir($entry_name) || preg_match('/\/$/', $entry_name)){
            chmod($directory_name, 0755);
            zip_entry_close($entry);
            continue;
          }
?>

This code here was added to deal with zip files that had directories of any depth within them.
There are three problems that cropped up here:

1. sub directories needed to be manually created
2. using mkdir recursively does not set the permissions appropriately
3. detecting whether a compressed object is a file or a directory (folder) is not as obvious as it could be

The first problem is handled through a recursive mkdir command.
The second problem is created by the first solution, so I then added a manual chmod on directories.
The third problem was a little weird.
For some reason the php is_dir() function was failing to detect that a given name was a directory or not, so I added a preg_match to handle that case.
I could simply be doing something wrong here, but the current code works for me.
This preg_match command might only work for the unix file system directories and the \/ part may need to be changed to \\ for other systems.
Also, if I do not use continue here, the code below this will end up creating (or try to) a file instead of a directory.

<?php
          if ($file_path = file_save_data($data, $real_destination . drupal_strtolower($entry_name))) {
            $custom_file = new stdClass();
            $custom_file->uid = $user->uid;
            $custom_file->filename = basename($file_path);
            $custom_file->filepath = $file_path;
            $custom_file->filesize = filesize($file_path);
            $custom_file->filemime = file_get_mimetype($file_path);
            $custom_file->timestamp = time();
            $custom_file->status = FILE_STATUS_PERMANENT;
            $custom_file->uid = $user->uid;

            drupal_write_record('files', $custom_file);

            $loaded_file = field_file_load($file_path);
?>

This is where the file gets created.

The part: $custom_file->status = FILE_STATUS_PERMANENT; will not work with the current filefield code because on line #452 of filefield_widget.inc the following causes a problem:
form_error($element, t('Referencing to the file used in the %field field is not allowed.', array('%field' => $element['#title'])));.
I would like to know the correct way to do this, but I do not know.
I need files to be permanent but this check requires that the file to not be permanent.
The solution here is to either set the status to something that is accepted, comment out the mentioned code (not recommended, but its what I did..), or find a way to call field_file_save().
I would prefer to use the field_file_save() function, but I am not sure how to do that part here.

That problem aside, the above code will manually insert the file into the drupal database and then load that file so that it may be attached to the soon to be created node.

<?php
            $custom_node = new stdClass();
            node_object_prepare($custom_node);
            $custom_node->type = 'custom_00001';
            $custom_node->field_custom_00001_00001 = array($loaded_file);
            $custom_node->field_custom_00001_00003 = array(array('nid' => '[node:field_custom_00001_00003-nid]'));
            $custom_node->title = $desired_entry_name;
            $custom_node->body = '';
            $custom_node->uid = $user->uid;
            $custom_node->status = TRUE;
            $custom_node->promote = FALSE;
            $custom_node->active = TRUE;
?>

This chunk of code will setup the node as I see fit.
The names field_custom_00001_00001 and field_custom_00001_00003 are simply the name of the fields I am using.
The name custom_00001 is the name of the content type I am using.
field_custom_00001_00001 represents the filefield field/widget.
field_custom_00001_00003 represents the node reference field/widget.

The function node_object_prepare($custom_node); is used to have drupal structure the node array.
Hopefully, this means that if any contrib modules are called during this process, their modifications will make it into the node object.
I have not confirmed if any contrib modules are called during this function.

Some of the core fields, such as title, uid, status, promote, active and maybe body must be included.
Without these fields set, drupal will say "I created content X", but when you look for content X drupal will then say "what is content X? it does not exist!".
I later discovered that somehow drupal_execute() can create nodes with invalid database structures.
The nodes will be in the database, but through the drupal interface the nodes will not be accessible!

<?php
            $custom_state = array('values' => (array) $custom_node);
            $custom_state['values']['op'] = t('Save');
            $custom_state['values']['name'] = $user->name;
            drupal_execute('custom_00001_node_form', $custom_state, $custom_node);
?>

This is the dreaded drupal_execute() function that gave me nightmares.
With an improperly configured state array and node object, this will not work correctly and node_save() will look like the better alternative.
The reason to use this over node_save() is that contrib modules will be able to do their checks and changes.
Using node_save() instead of drupal_execute() here introduces the possibility of security exploits because contrib module validation checks are not called.
(Or at least that is my experience).

Adding the 'op' field to the state is required.
I am adding the 'name' field to the state just to be on the safe side.
Let me know if it is not needed.

<?php
  process_zipfile();
?>

I prefer to wrap all of my code in functions to prevent any variables from being at a near global scope.

All of the other code chunks that were skipped were deemed not important enough to discuss.

This post is not an authortative solution to the problem.
This is simply how I managed to get things to work because the documentation mentioned here http://drupal.org/node/330421 is no longer completely accurate.
I am completely open to any suggestions, solutions, or better ways to do what I am doing.
I am hoping at the very least my case discussed here can be used to help update and improve the documentation listed here: http://drupal.org/node/330421

Also, I apologize for any typos or mistakes.
This post is way to long for me to not have made at least one unnoticed mistake.