Port node-import to Drupal 6

This is a first stab at modifying the file supported/event.inc for working with Event module for Drupal 6.

Any comments and suggestions for improvements welcome.

Frank Ralf

Worked beatifully for me to import a large CSV file into Drupal, with a content type that has CCK fields.
There was an error when I clicked "delete uploaded file from server", though.

Also, it gave me some SQL warning about a duplicate insert into this module's table.

But, it did the job. I can't thank tanoshimi enough for posting the patch, and it should probably be released as Alpha.

There were 2 errors for me. The first had to do with supported/cck/content.inc trying to access an array element that didn't exist. I fixed that with the attached patch.

The second has to do with the fact that I don't have something installed right. The node_import module is looking for a function called date_iso2array() and can't find it. The full error message reads:

Fatal error: Call to undefined function date_iso2array() in /opt/sws/avnet/app/httpd/htdocs/www.avnet.com/modules/node_import/node_import/supported/cck/content.inc on line 103

what am I missing?

Never mind... the patch from http://drupal.org/node/227681 fixed this issue. Looks like the date functions were still using older libs from date instead of date 2.

All ahead full.

I needed to fix the issue described in this node to get this to work.

http://drupal.org/node/192211

jwaxman -

The patch from the article you posted looks almost identical to the patch I posted here for content.inc. If you compare the 2 patches, only the variable names are different. Nice to know I'm not alone.

Jon

The updated Drupal 6.x module works great so far in my installation.

In the meanwhile the official Drupal 5.x module supports uploads for nodes. Using the upload.inc file in the Drupal 6.x version fails on submission and results in a blank screen. The upload.inc file really isn't big. What needs to be done to upload.inc to work in drupal 6.x? Can anybody help?

@ak

Just a very quick guess:

upload.inc does some direct modification of the database in its implementation of hook_node_import_postprocess().

You might first have a look, whether the database tables and fields of Upload are the same in Drupal 5 and 6.

However, Drupal 6 now uses a special Schema API for database access (http://drupal.org/node/114774#schema-api) so that might be the place to look.

Regards,
Frank

Ok, here's my progress in detail. I didn't really dig and understand the Drupal Schema API due to my lac of skills. But indeed the database tables aren't the same.

Drupal 5 table: fid, nid, filename, filepath, filemime, filesize
Drupal 6 table: fid, uid, filename, filepath, filemime, filesize, status, timestamp

It's significant that the nid became a uid and the file_revisions table of Drupal 5 doesn't exist in Drupal 6 anymore. So heres the current Drupal 5 upload.inc hook_node_import_postprocess() implementation:

function upload_node_import_postprocess(&$node, $preview, $error) {
  if (isset($node->node_import_files)) {
    if ($preview) {
      return t('File attachment %filename.', array('%filename' => $node->node_import_files));
    }
    else if (!$error) {
      $fid = db_next_id('{files}_fid');
      $filename = $node->node_import_files;
      $filepath = file_create_path($filename);
      $filemime = mime_content_type($filepath);
      $filesize = filesize($filepath);

      db_query("INSERT INTO {files} (fid, nid, filename, filepath, filemime, filesize) VALUES (%d, %d, '%s', '%s', '%s', %d)", $fid, $node->nid, $filename, $filepath, $filemime, $filesize);
      db_query("INSERT INTO {file_revisions} (fid, vid, list, description) VALUES (%d, %d, %d, '%s')", $fid, $node->vid, 1, '');
      unset($node->node_import_files);
    }
  }
}

To get a further idea of what has to be done I took a look in the devel module that is able to auto generate nodes with attachments. Heres the related devel function:

function devel_generate_add_upload(&$node) {
  $source = 'misc/blog.png';
  $size = filesize($source);
  
  // $after this call, $source contains the new path.
  file_copy($source);
  $file = new stdClass();
  $file->filename = 'blog.png';
  $file->filepath = $source;
  $file->filemime = 'image/png';
  $file->list = variable_get('upload_list_default', TRUE);
  $file->description = 'b log.png was here';
  $file->filesize = $size;
  $file->weight = rand(0,10);

  // If we made it this far it's safe to record this file in the database.
  db_query("INSERT INTO {files} (uid, filename, filepath, filemime, filesize, status, timestamp) VALUES (%d, '%s', '%s', '%s', %d, %d, %d)", $node->uid, $file->filename, $file->filepath, $file->filemime, $file->filesize, FILE_STATUS_TEMPORARY, time());
  $file->fid = db_last_insert_id('files', 'fid');
  
  $_SESSION['upload_files'][$file->fid] = $file;
  $node->files[$file->fid] = $file;
}

Some things are different here. A $file class is created and stored in $node->files and a $_SESSION variable I'm not aware of. This wasn't necessary in the Drupal 5 version. Most fields are hardcoded here. I can't see the list, description and weight fields get stored in the files table nor how the node nid the file belongs to is saved for further reference anymore. No idea where this information lives in the Drupal database. Maybe somebody can bring me some light into the darkness.

Never the less, here's my attempt to merge these two functions into one that potentially works for Drupal 6:

function upload_node_import_postprocess(&$node, $preview, $error) {
  if (isset($node->node_import_files)) {
    if ($preview) {
      return t('File attachment %filename.', array('%filename' => $node->node_import_files));
    }
    else if (!$error) {
	
      $file = new stdClass();
      $file->filename = $node->node_import_files;
      $file->filepath = file_create_path($file->$filename);
      $file->filemime = mime_content_type($file->$filepath);
      $file->list = variable_get('upload_list_default', TRUE);
      $file->description = $file->filename.' (file attached by node import)';
      $file->filesize = filesize($file->filepath);
      $file->weight = 0;

      db_query("INSERT INTO {files} (uid, filename, filepath, filemime, filesize, status, timestamp) VALUES (%d, '%s', '%s', '%s', %d, %d, %d)", $node->uid, $file->filename, $file->filepath, $file->filemime, $file->filesize, FILE_STATUS_TEMPORARY, time());
      $file->fid = db_last_insert_id('files', 'fid');

      $_SESSION['upload_files'][$file->fid] = $file;
      $node->files[$file->fid] = $file;

      unset($node->node_import_files);
    }
  }
}

Regarding the issues above that I don't understand, it looks as if this code could work. If someone would look over my mockup I'd be very thank full.

MfG

In my experience, a blank screen can be the result of:

1. Not enough memory. Increase your php memory limit to 100MB and see if that helps. If not, then...

2. display_errors and display_startup_errors are turned of, so PHP isn't telling you anything.

To change any of these settings, use the ini_set() php function, documented at php.net. The three variables in question are:

memory_limit
display_errors
display_startup_errors

A good place to put the code is inside $drupaldir/sites/default/settings.php, right at the top.

If you're working on a production site, and can't display errors on-screen, you can use:

ini_set('memory_limit','100M');
ini_set('error_log',dirname(__FILE__).'/../../php_errors.log');
ini_set('log_errors','On');

This should set your memory limit up to 100MB, and turn on file-based error reporting. Assuming you are putting this code in sites/default/settings.php as mentioned above, then a file called php_errors.log will appear in the site root.

Error log file permissions: If you're still getting a blank page and still don't see the file, make sure your drupal folder is writable by apache. Apache can't always write to all folders everywhere.

Avoid setting chmod o+w however, beacuse you don't want public users to write to your webroot folder. Best way to do this is to create php_errors.log as yourself with $:/> touch php_errors.log from the command line, then chown it to user.apache where 'user' is your user name and 'apache' is the apache group. You may have to fiddle with user group membership, but Linux group membership is out of the scope of this response. There are lots of great places you can go to learn about that online.

Anyway, you get the idea.

Oh... if you're running in safe_mode, the ini_set() function may not work... But I don't think drupal can run in safe mode anyway. Not sure.

Does this version support updating existing nodes based on a unique key such as NID or title?

Hello guys,

Can someone give me a link to the latest Node Import for 6.x snapshot please?

I'd like to import profiles set up with content_profile. Profile is an option but it doesn't see the added fields. The CSV import files are not parsed. I am probably going to have to resort to writing the script but if there is hope for node_import, I'd like to give it a shot.

The zip file worked fine. All the profiles are attributed to the root user. A few of the "profiles" have signed up as users and it was easy to transfer the profile.

Are the patches listed here being added to the file linked at the top, or not yet? (File doesn't list a time stamp)

I will be doing some testing of this -- extremely glad to find node-import for 6.x!

Changed title back after reading http://drupal.org/node/244931#comment-1029298 ("Please make a 6.x version of this module").

I think we really need an "official" HEAD version for Drupal 6 to bundle all development efforts.

Frank Ralf

I'm trying to use this version of Node Import to move data from another CMS into a D6 install and am having problems with a CCK 'datetime' field I've added to a standard node content type. The Title and Description comes through fine, but both values (start and end) for the date field are filled in with 'now' instead of the dates in the CSV file.

Anyone seen this behaviour?

More information. Node Import for 6.x doesn't work with CCK fields set to 'datetime' and with multiple values in the field (from and to in this case). I've not dug far enough into D6 to figure out how to fix this yet.

There's the suggestion over in http://drupal.org/node/244931 ("Drupal 6.x port") to port the Import/Export API module instead (http://drupal.org/project/importexportapi). Might be a good idea.

Frank

following... :)

What's the latest on this? And does it import your URL re-directs?

Subscribing.

Hi, and many thanks for the module upgrade.

any chance to have the book page: parent link function working (soon)?

Sorry for this direct question, I just think that this functionality, together with CCK and a little work in views-view-xml.tpl.php (Views Datasource module), can turn a basic D6 installation into a powerful database-publishing workstation for pre-press purposes.
I'm currently using node_import (in D5) and export_dxml to render complex publications into a large and unique Indesign-compatible XML file, but D6 and views2 offer a much more powerful and elegant approach to this kind of issue.
Of course if anybody is interested I will post about this.

Thanks,

Miro

sub

Subscribing

sub

subscribing

subscribing *bump*

subscribing too

I have tried the D6 port and it gives a lot of errors.

    * user warning: Duplicate entry 'service' for key 1 query: INSERT INTO node_import_mappings (type, csv_headers, mapping) VALUES ('service', 'a:7:{i:0;s:5:\"title\";i:1;s:11:\"Plan Format\";i:2;s:15:\"Number of units\";i:3;s:10:\"Price Code\";i:4;s:5:\"Price\";i:5;s:28:\"Short Inspection description\";i:6;s:22:\"Inspection description\";}', 'a:7:{i:0;s:1:\"1\";i:1;s:22:\"node_import_taxonomy_4\";i:2;s:22:\"node_import_taxonomy_7\";i:3;s:3:\"sku\";i:4;s:5:\"price\";i:5;s:22:\"node_import_taxonomy_3\";i:6;s:1:\"1\";}') in /var/www/html/dev.aspedia.net/dpl_qbm/sites/all/modules/node_import_D6/node_import.module on line 580.
    * user warning: Duplicate entry '/feed-' for key 2 query: INSERT INTO url_alias (src, dst, language) VALUES ('node/350/feed', '/feed', '') in /home/svn/codebase/drupal/aspediaweb/trunk/core/modules/path/path.module on line 112.
    * user warning: Duplicate entry '/feed-' for key 2 query: INSERT INTO url_alias (src, dst, language) VALUES ('node/351/feed', '/feed', '') in /home/svn/codebase/drupal/aspediaweb/trunk/core/modules/path/path.module on line 112.
    * user warning: Duplicate entry '/feed-' for key 2 query: INSERT INTO url_alias (src, dst, language) VALUES ('node/352/feed', '/feed', '') in /home/svn/codebase/drupal/aspediaweb/trunk/core/modules/path/path.module on line 112.
    * user warning: Duplicate entry '/feed-' for key 2 query: INSERT INTO url_alias (src, dst, language) VALUES ('node/353/feed', '/feed', '') in /home/svn/codebase/drupal/aspediaweb/trunk/core/modules/path/path.module on line 112.

EDIT: Ignore this - it is a duplicate titles problem causing duplicate paths, i can work around it.

This code needs the patch at #227681: Date 2 API breaks node imports with dates.

With this port, I have problems if I'm importing content that doesn't have location information into a content type that has location information enabled (but not mandatory). I get the following errors:

warning: Invalid argument supplied for foreach() in /sites/all/modules/location/location.module on line 1677.

You must fill in both latitude and longitude or leave them both blank.

Just a note as well that it seems the module maintainer has started to port the module to D6 (see the dev snapshot), though it doesn't look to be based off this port as the interface is considerably different.

I don't seem to get any fields except lat/lon for location when doing the mapping. My solution is to create the location fields as CCK, do the import to the CCK fields, and then use a custom script to move the information to the correct field using node_load and node_save.

The port in 6.x-1.x-dev is a rewrite to finally make use of the new features of Drupal 5.x and 6.x especially the extended form functionality (drupal_execute()). This will for the biggest part remove the need for many duplicated form validation code and weird data manipulation of the $node object to get it in the right format for saving which will make the module as a whole more stable.

Normally each day some new part will be added with the basic functionality ready by next weekend (with complete support for most core modules like upload and taxonomy).
Next week CCK should follow.

So I'm not going to commit the patch of this issue to CVS. Just to let you all know.

Thanks for the update Robrecht, great to hear it's being refactored. Looking forward to trying it out.

Good to see.
I sneaked a look at the DRUPAL-6--1 in CVS. Looks like the code will be good and clean, but not doing much yet ;-)
You probably know, but the 'text delimiter' isn't working yet. Isn't there a PHP func for that? http://nz.php.net/fgetcsv or are your needs even more clever than that?

comment #37 : yes, still very clean :-)

'text delimiter' doesn't work yet. I know about the fgetscsv function but previously there were many problems with it (not reading UTF8 characters, text length is limited). Probably due to people using PHP < 4.3.5 (at which point it became binary safe).

Wonderful work! Will you be porting the location import feature?

Subscribing. Oh yes.

Also see this: http://drupal.org/node/333031

Subscribing

I know Robrecht Jacques is working on a more official release of this module for 6.x. However, for those of you who don't want to go through the trouble of applying the patches mentioned at the top of this thread, I have created a tar.gz file of the 6.x node_import module with those patches applied.

Please note that this is not the official release, but will probably work. Just use with caution and understand that Robrecht's release is coming soon and will be completely different.

EDIT: I actually couldn't get this working on my development machine for some reason. I think there might be something wierd with my install. Some feedback on whether this roll-up wroks would be appreciated. This actually caused my drupal install to hang accessing any admin page until I deleted the node_import folder. Not sure why this is happening. Sorry.

I was just trying the latest dev release from csv but a text cck field did not show up unfortunately.

sub

This version (jcwatson11 #43) installs fine for me and the content-mapping all looks correct, but when I click the "Next (preview)" button I get the error:
Fatal error: Call to undefined function date_unix2iso() in W:\www\sites\all\modules\node_import\supported\cck\content.inc on line 99

If I comment out that function, then I get the error:
Fatal error: Call to undefined function date_iso2array() in W:\www\sites\all\modules\node_import\supported\cck\content.inc on line 104

I tried removing the Date information from the CSV file but got the same error, so I believe it's related to the CCK Date field used for this content type.

Hope this works soon... feel free to let me know if I can help with testing.

Feedback (possibly unwanted, but still...) on Robrecht's dev snapshot of 6.x-1.x-dev from 15 November:
- still no CCK fields in mapping
- interface now gets to the point where I can ask it to start the import
- but the importing task never completes :(

Please let me know if I can help with any testing - am waiting for this module so I can import a large batch of data.

subscribing. thanks.

adam_b - do you have the date module installed?

@jcwatson11 (#43)

Got it working with my development installation.

Only "Delete file from server" still creates duplicates (instead of deleting the file).

Thanks!
Frank

re #49 - yes, I have the date module v 6.x-2.0-rc4 installed

the CCK fields for the content type into which I'm importing the data are:
- field_author Content Taxonomy Fields
- field_editor Text
- field_date Date
- field_publisher Content Taxonomy Fields
- field_subjects Content Taxonomy Fields

re #50 - That's good news. The delete file from server was an issue that existed prior to my combining of patches. There must simply be something wrong with the drupal instance I tried it on. Thanks also to adam_b for testing.

re #51 - Sorry. I got nothin else for you to try on that front. You'll have to figure that one out on your own. I know the module does require the date module for date support. And there may be issues with your version of PHP with regard to the date module. Note that the date module can be difficult to get working properly on PHP <=5.2.

re #51 - okay, I changed the date field to an integer (it's only holding the year, so doesn't really need to be a date at all). I can now import content, but there are some issues:
- the "author" and "publisher" field are mapped, but don't appear in the preview and the data isn't imported
- when I tried importing 10 records there's no error, but when importing 100 records I got an error message:
user warning: Duplicate entry '0-300' for key 1 query: INSERT INTO term_node (nid, vid, tid) VALUES (286, 300, 0) in W:\www\modules\taxonomy\taxonomy.module on line 702.

Downloaded version on Nov 19.
All OK up to reading tab delimited files (my file was created by Text Wrangler on Mac). The column headings in first line not recognized (all put together in first column as if it doesn't see the tabs between fields. All records were loaded into first column just like the column headings - no field delimiter recognized (tab separated)

While not critical the file extensions don't appear to do anything my file was read and uploaded without the correct extension. I changed it to tsv just to be sure that wasn't causing any problems.
I have Attached 4 record file called test no extension. The first record is field headers.

thnx

Edit
After much messing around I was able to get all 8 screens to work. Apparently I needed to create the file using , delimited fields with "(quotes) delimiting text and then save file (using Text Wrangler) using Unix file structure and UTF-8 Unicode. Column names recognized and imported file looked like it was recognized.
Problems:
1. Can't use none in options for text and escape delimiters throws off errors
2. Doesn't recognize CCK field in the mapping screen
3. When import is started it just stays busy - no import.
4. Related to import and busy - no way to stop process just hangs. Then must go and uninstall module to get it to stop.

I can help test if you need it. Using Mac, Text Wrangler, xCel and Drupal 6.6. I then have working test site on a hosting provider to test - not using MAMP to test this.
Thnx for your efforts.

For more people willing to help testing, the port to Drupal 6 node_import 6.x-1.x-dev (currently unstable) is available through the View all releases link of the module.

An update on the status of 6.x-1.x-dev:

First of all, it is a development version which means not everything is implemented yet and may or may not work on a day to day basis. If you want to have something reliable, wait until a proper 6.x-1.0-rc1 release.

The only module that is currently supported is node.module. This means the following (and only the following) fields: Title, Body, Input format (of Body), Authored by, Authored on and the Publishing options.

What are the next steps before a proper 6.x-1.0-rc1 release:

1. Support for core modules:
   • Node (check and debug, eg Dates don't yet import and the Authored by seems to be Anonymous),
   • Taxonomy (multiple select, tags, ...)
   • Path (set a URL alias)
   • Book (select Parent)
   • Comment (only the Comment Disable/Enable/Read-Write option)
   • Menu (select Parent)
   • Upload (one - maybe more - Attachments)
2. Release node_import-6.x-1.0-rc1
3. Support for CCK modules:
   • The CCK types included in the CCK module itself (Text, Number, ...)
   • Date
   • Link
   • Email
   • maybe a couple of others depending on how difficult adding those are
4. Release node_import-6.x-1.0-rc2
5. Support some common contrib modules
   • Auto node titles
   • Location
   • Event
   • Token/Pathauto (if this needs something)
   • ImageField, FileField, ...
   • Image module
6. Then, depending on the number of bugs etc and the cry to add support for more stuff, release one or more -rc versions.
7. Release node_import-6.x-1.0

Hope this clears up things.

Thanks Robrecht. It's helpful to have the list of what's supported – now that I know CCK definitely isn't supported then it's clear why my import isn't working.

I'll await the release clients with impatience – sorry to be harassing you but it's just such a useful module :)

@Robrecht, do we have any idea of timescales for these RCs? I need CCK functionality quite urgently, and may be able to help move the development along.

I will refer the guy who is contracted to implement my node imports to this thread and see if he can be of any assistance, if this is going to be helpful?

@NikLP: difficult to say.

Aim is to have -rc1 (core) ready/usable by Tuesday - I'm 95% sure this will be possible.
-rc2 (+CCK) could be ready by next week Friday - depending on how much I'm able to work on it during the week (and this weekend - which is unfortunately quite full) - 50% chance, otherwise it will be a week later.

Do you have a list of CCK modules you use for your content type(s)?

A sample CSV file to import is always useful. Including how the content type is configured.
Other than that, the code is still somewhat in a flux so it's difficult for others to help out on short term.

RE: Robrecht Jacques

Hey I need this working with CCK, Location and possibly Ubercart for 3 different projects (2 of which are paid) so I offer my services to help you out.

I made wrote a similar module for Ubercart, which already had an XML Importer interface so I wrote a CSV -> XML which was then pasred and imported into Drupal/Ubercart. I made a similar API, supplying hooks to to integrate any/all other modules, I see you've come to the same design decision. I have tried to figure out what your API is but would feel better if you could give out a little info on which hooks are available and what their input/output is.

Perhaps then, myself (and others?) could start an effort to get other modules (and possibly their developers) to start integrating with it and speed up this development process. The alternative is that I will just take what you've done so far and start hacking into it to make it work for what I need it to and the code will be thrown away at the end of the job. Would seem like a waste to me.

Send me an message if your interested in my help.

@cpill

Yeah, I need to write docs too. You can look at supported/node_import.inc which has some documentation, but maybe a more global document describing stuff would be better.

Ubercart support is a long term goal, so if you could help me out with that, I'd appreciate it.

The XML format uc_importer uses is a lot more flexible than CSV. Maybe I should follow a similar path although the goal of node_import is to import tabular data right now. This means we have to do some strange stuff to allow for multiple vocabulary terms to be assigned to a node (eg "term1||term2||term3" in 5.x). A XML format may be supported later on.

Currently working on the -rc1 release... This will make clearer how to map columns to contain multiple values (such as terms). If you can wait another day, we can work together to get a cck, location and uc release out by the end of the week. The commit I'm working on right now extends quite a bit the API (to allow for taxonomy mainly).

Basically, for the wizard, following hooks are called:

// To define content types you can import. The node content types
// are builtin (see supported/node.inc).
hook_node_import_types();

// For node types, the $type passed is "node:XXX" where XXX is
// "story", "page", ...
// Here you would return all fields that are available. This is the
// important stuff.
hook_node_import_fields($type);

// Only for the wizard, specify the default values and the options.
// These two hooks return a form.
// $type is "node:XXX"
// $defaults and $options are the filled in values on the form,
// $fields is what hook_node_import_fields() returned
// $map is a mapping from file columns to fieds. The format:
//   $fieldname => $column_number
// or
//   $fieldname => array of $column_numbers
hook_node_import_defaults($type, $defaults, $fields, $map);
hook_node_import_options($type, $options, $fields, $map);

The import itself uses following hooks to get a list of
values:

// To know what function to call to create the object
// $type['create']:
hook_node_import_types();

// To know the preprocessors of a field. Preprocessors are
// simple functions that convert a string value (from the
// CSV file) to a value that can be submitted to the form.
// For example: the user will probably want to specify
// "Dogs" as a term for "German sheppard", while the
// form submit function needs a tid. So there is a
// preprocessor function to lookup the "Dogs" term and
// replace it with the corresponding tid.
hook_node_import_fields($type);

// This hook is to specify static form values, such as
// $node->created:
hook_node_import_static($type);

// Lastly, if you can't use the simple preprocess functions
// to convert the input value from the file to something
// the form can use, you can use:
hook_node_import_values_alter(&$values, ...);
// to alter all values.

// After all this, the $values are submitted to the form.

I'll create some documentation for developers after -rc1.

@Robrecht

The stuff I'm using is listed here. I would suggest that implementing support for the more commonly used cck field types would be widely welcomed. These are listed in the order that I think is most appropriate.

Date
Email
Link
Node Reference
Number
Text
User Reference
FileField (tricky!)
ImageField (tricky!)

Also if support for content_profile and autoassignrole could be implemented alongside any support for importing users (even if only in the case of users as nodes) that would probably be a noble cause as well! :)

You should look at this patch that may allow you to do some stuff with maintaining nids on import

Enhanced data import - node_save() and user_save()

Re #62: the only field type I'd like to add is "Content Taxonomy Fields". Even if it only handled a single value to start with, this would be useful for my selfish purposes...

Content_taxonomy isn't even clear in its instructions as to how it actually works. See my post here on that module: http://drupal.org/node/332454

Further to that, if c_t actually worked how it *said* it does, it would be possible to import the taxonomy as normal, which would be less complicated than importing the data as cck values (c_t's "alternative" (but seemingly enforced) storage method).

Is CCK support meant to work yet? Because it doesn't seem to with me...

Can you shed any light on this, please / give an ETA?

Regards,

heebie.

subscribe... awaiting CCK support

Subscribing...

subscribe... awaiting Ubercart support

@Robrecht Jacques (#61)

Thats a good start with the documentation mate! Once you know what the hooks are and have an idea of what they are supposed to do then it makes sniffing them out of the example code easier.

I guess if you get some API doc out after RC1 then the "NON-Programmers" here can redirect their pressure towards the actual developers of the modules they want to see integrated with your fine work.

The hard part is now to figure out how CCK works so i can make an attempt at an integration. I just helped out on the imagefield_import module and got that working thanks to the code here: http://drupal.org/node/292904 which doesn't really give many insights as to what is happening under the hood of filefield but its a starting point.

Will check back in when RC1 comes out. I guess I'll be on holiday before any of this happen.

merry Xmas ppl :)

subscribed, also awaiting CCK support

CCK and Location support is key to me; I'm anxiously awaiting information about this (and having one column of data being able to be applied to vocabularies would be a bonus; too!). Looking forward to having this working with the D6/Location3 setup! If you need some test data; I have ~1200 records with CCK, Event, 3 Taxonomy vocab lists, and Location data I want to import to a site I'm working on - I can set you up as a privileged user or just keep good notes of the process when the next release comes out.

subscribed

Subscribe

Release Candidate 2 has been released yesterday (no CCK yet; maybe for RC3?). Thank you for the work.

Subscribe

Subscribe

Eagerly awaiting CCK support for 6.x. I have 20,000 CCK nodes to import. I wish I could help bring it along, or help in anyway for that matter.

Thanks for all the Node-Import team efforts

I'm a Node Import fan too! Eagerly and appreciatively waiting CCK support. Thank you for all you do.

In the interim I really had to find something that would bring in my thousands of CCK nodes. If anyone else is desperate I just tried Feed api + CSV Parser + Feed Element Mapper and it worked! I had to do multiple sheets to avoid memory errors, but I had to do that with node import too.

http://drupal.org/project/feedapi
http://drupal.org/project/parser_csv
http://drupal.org/project/feedapi_mapper

Thanks Robrecht for all the great work on this module and thanks aqua for posting an alternative while we wait for CCK support. I'm going to try it now.

subscribing

Subscribing, and awaiting full CCK support

Aquamaureen,
could you eleborate a little on how tou did set-up those 3 modules to import cck-nodes?
Than you very much in advance.

Subscribing, waiting full CCK support plus Location support.

subscribe, can't wait for cck, my fields of interest are text, number and image.

thank you aquamaureen for the workaround, I'll try it out.

Subscribing and waiting CCK Import too.. Thanks for all efforts!

subscribing - eagerly awaiting CCK text, date.

Sorry for becoming off-topic and poisening this thread. But in case you, like me, can't wait here's an alternative to import your nodes programmatically.
In my case the CCk fields where pretty straight forward. Writing an import script was easier for me then configuring feed.api (but i thinks that's because I dont understand feeds.)

In case you are interested:

<?php
function import_mycontent() {
  global $user;
  $results = db_query("SELECT * FROM {csv_table}");
  module_load_include('inc', 'node', 'node.pages'); 
  while ($mycontent = db_fetch_object($results)) {
    $node = array('type' => 'mycontent'); 
    $form_state = array();
    $form_state['values']['title'] = $mycontent ->titel;
    $form_state['values']['field_textfield1'][0]['value'] = $mycontent-> textfield1;
    $form_state['values']['field_textfield2'][0]['value'] = $mycontent-> textfield2;
    // fill in the other values the same way as above
    $form_state['values']['op'] = t('Save');
    $form_state['values']['name'] = $user->name; 
    drupal_execute('mycontent_type_node_form', $form_state, (object)$node);
  }
  return FALSE;
}
?>

In case there are questions about the code I suggest to open a new thread.

Does the roadmap need to be updated to mention CCK coming in another rc?

Subscribing. Eagerly waiting for support to CCK custom fields & location.

Actually, I was planning a site along these lines working for 4.7 (KML feeds and Gmaps with multiple markers based on Taxonomy) and had everything else settled when I finally came to the CSV import part and surprisingly found the last step met an obstacle.

Drupal seems to be an amazingly promising CMS but the lack of support for CCK custom fields & Location is definitely an Achilles' heel hampering its potential right now.

Also awaiting CCK support - none of my custom fields are available in the "map file columns" step.

Hi Clive,

Could you post some more info about doing it this way. Where do you put the csv file? If you have a content type for 'food' with fields for title, description, size, price etc. And is the final script above just added to the server as a php file that you go to the url to invoke the script?

Thanks

Stu

awaiting CCK support, thanx!

@StuartDH
I like to work with modules, so I wrote one.
It is very straight forward: Just a menu to call the function and the function itsself.

You can import your CSV-table into the drupal database. There are several way to do this. You can do it with a MYSQL function or by PHP or use PHPMyAdmin.
I use Navicat and that's really very easy to use.
You could also read the file from yor harddrive and import the nodes, like node_import does, but this way is easier.

The example puts a menu called "import food" into admin/settings. By selecting this menu the import will start.
You can call it also from the url: www.mysite.com/admin/settings/importfood

The difficult part is to find out how the data of your cck node is structured. You can use print_r to find out.
I like to remind you that this is really a dirty way to do the job. It works fine but remember to back-up your data.
Good Luck

// $Id$

/* this is the code for a module "site_helper" to import 
 * your cck node
 * So this code goes into site_helper.module
 * Be sure to make site_helper.info as well
 */


// this function provides a menu item
function site_helper_menu() {
  $items = array();
  $items['admin/settings/importfood'] = array(
    'title' => 'Import food',
    'description' => 'Import food',
    'page callback' => 'import_food',
    'access callback' => user_access('administer site configuration'),
    'type' => MENU_NORMAL_ITEM,
  );
	return $items;
}

function import_food() {
  global $user;
	// food_old_nodes is the CSV-table containing your old food nodes
  $results = db_query("SELECT * FROM {food_old_nodes}");
	module_load_include('inc', 'node', 'node.pages'); 
	$i = 0;
  while ($oldfood = db_fetch_object($results)) {
    $node = array('type' => 'food'); // food is your cck-type 
    $form_state = array();
    $form_state['values']['title'] = $oldfood ->title;
    $form_state['values']['size'][0]['value'] = $oldfood-> size;
    $form_state['values']['price'][0]['value'] = $oldfood-> price;
    // and so on..
    $form_state['values']['op'] = t('Save');
    $form_state['values']['name'] = $user->name; 
    drupal_execute('food_node_form', $form_state, (object)$node);
    $i++;
  }
  return FALSE;
}

While waiting for Node Import's CCK support, there is also the workaround mentioned by aquamaureen on this issue, with CSV Parser and other modules. See also the handbook page Migrating to Drupal.

I was successful at importing data from CSV file into drupal CCK nodes using what aquamaureen suggested above #78 so this is now a workaround for me.

First I installed FeedApi and FeedApi_mapper and I was successful in importing some feeds and their components into CCK nodes and fields. I chose SimplPie parser as it offers more feed components to map.

Then I installed CSV parser and the only difference is you enter the link to the CSV file into the Feed field instead of the feed link.

It was a bit confusing reading the instructions at first so I'll try to give some useful info:
You need to have one content type to be a container of all the nodes to be created, and that container content type has to be "Feed enabled" (that's not the exact expression but you'll recognize it in the handbook when you get to it). I guess you can say that this content type resembles (or is equivalent to or something like that) the CSV file you're importing. I recommend using the already created Feed content type for testing it out.

The other content type is the actual result you want to achieve, CCK you probably already have and you're eager to get CSV data into it, this content type resembles (or is equivalent to or something like that) the rows of CSV data. This content type should not be "Feed enabled".

The rest shouldn't be too hard to get while reading the instructions, if needed I'll write a more detailed instructions of how I did it.

Sorry for hijacking the thread but this is just a workaround, as soon as CCK in Node import is available I'm switching :)

Another workaround -using the Node Import version for Drupal 5, instead of 6- has been mentioned by WorldFallz on the forum thread Importing data into cck:

"What I did was setup a bare d5 site with just the modules I needed for the import, then imported the data, and upgraded the site to d6."

See the CCK project page for details on updating from version 5.x to 6.x.

Looking forward to CCK support. Many thanks. Subscribing...

www.reallifedesign.co.uk

About importing into CCK, an additional workaround apart from the already mentioned modules (Node Import 5.x and upgrading D5->D6, or CSV Parser for D6, etc.) is the one suggested on this issue by others of developing custom PHP scripts. For those interested, there are for example migration tips with some techniques. I've just added a brief paragraph about this on the handbook page Migrating to Drupal.

Also for CCK, there is a recent related thread (alternatives to node_import?) on the support mailing list.

Hi Robrecht,

I see rc3 is out. Does that mean CCK support is there?

also interested in cck Address field is that's possible.

Are there any work instructions on how to use this module?

ie. is there an example csv file or some doco.

cheers
Ed

juicytoo wrote:
>I see rc3 is out. Does that mean CCK support is there?

See the release notes: no support for CCK yet. I think that's currently under development by Robrecht. There are workarounds for the time being, mentioned on this issue.

Waiting for cck support , Subscribing . thanks
Guzzus and Meikan

A related issue is When to expect CCK imports for 6.x? This seems to be coming soon, at least partially for RC4.

Hmm. So now there's two threads for the maintainer to update.

<rant>did someone think that by starting a duplicate thread on this issue, that it would get resolved any quicker??</rant>

Marked the other thread as duplicate. Everyone calm down! :)

subscribe

Really GREAT module!!!

i'm very much looking forward to CCK support as well.

check out the repository, cck is available there. be carefull. no official release.

could not find rc4 which I heard had cck. Any hints to where it is?

could not find rc4 which I heard had cck. Any hints to where it is?

subscribing

subscribing

I really need the ability to import products for ubercart. All the product fields including attributes fields and images are my main desires.

www.lenabrown.com

Lena

subscribin'

subscribing

subscribe

Subscribed and awaiting the CCK port

Should the dev release (http://drupal.org/node/328041) be tested? If so maybe should be put a bit more visible on the project page, with the remarks that it is unstable so it is easier to find? thx

-dev releases should only be tested if you know what you are doing. If you know what you are doing, you'll find it. Because I want to avoid many issues on an unstable -dev release I prefer that the majority tests the -rcX or official releases. It is easier for me to have an issue on -rcX then "I have downloaded -dev on X feb and ...".

Let's split this issue up and set it as "duplicate":

• http://drupal.org/node/374352 : Support CCK ImageField module
• http://drupal.org/node/374351 : Support CCK FileField module
• http://drupal.org/node/374348 : Support CCK Taxonomy fields module
• http://drupal.org/node/374346 : CCK Date module
• http://drupal.org/node/374344 : Content profile module
• http://drupal.org/node/374343 : Core upload module
• http://drupal.org/node/349408 : node import and ubercart 2
• http://drupal.org/node/366271 : Node reference fields not visible in mapping screen
• http://drupal.org/node/354077 : Import Locations in Drupal 6

Working on getting -rc4 out. It will include most of the above except probably the file ones (Upload, ImageField, FileField, Ubercart).

If you have any ideas on how to support file related modules, post them in http://drupal.org/node/374343 : Core upload module. My problem is mostly: how are the files uploaded (what directory)? Should there be an options to specify the directory? etc.

I see Node Import 6.x-1.0-rc4 has just been released, with support for Profile, Location and CCK modules. Thank you very much!

Yes, thank you!

I've done one test run with some data, and it looks like cck, taxonomy, location are working great. it's even doing automatic geocoding from the location data i'm feeding it, which i guess i wasn't necessarily expecting.

the only bug i've encountered so far was going "back" a page (from the field mapping step) and encountering a script that hung. let me know if you would like more details, i'll replicate it.

Kevin

Normally I don't read duplicate issues.

it's even doing automatic geocoding from the location data i'm feeding it, which i guess i wasn't necessarily expecting.

Node_import has to do fewer things then in 5.x because it programmatically submits the node creation form. Location acts on the submitted values and node_import gets this geocoding automatically. This is the advantage of the new design.

Of course, the problem is now to submit the correct values... which gives other problems all together (eg Drupal caching things it shouldn't) :-)

the only bug i've encountered so far was going "back" a page (from the field mapping step) and encountering a script that hung. let me know if you would like more details, i'll replicate it.

If you can replicate this all the time, I'd like to investigate this further. Although I have no clue right now why going from "Map file columns" back to "Set file options" would have the script hang. Create a new issue for it if you feel like looking into it.

Great work Rob! I'm having an odd problem with a CCK checkbox field, I can't seem to get anything past it's illegal values field (even not importing data for it, and it's not a required field!) - I've tried a few options (using a value pulled from the DB, using a number, using NULL in the field, and it rejects everything.

Also, the default values page doesn't seem to list CCK fields; it'd be a lot easier to troubleshoot this if I didn't have to re-upload a file every time :)

Going back to the CCK field itself, I didn't have key values defined, adding in key|label values for these seems to have repaired the validation problem

much as i hate adding nothing with a "subscribe" comment, i really want to track this :) Promise to test at some point soon and feed back if i find anything.

This is something I need so I am subscribing too for CCK imports. I think if I really do need it then I will try it as it is but if not wil wait.

Cannot wait for this. CCK and Location import is soooo very awesome.

subscribing

I think I'm in the right place. Trying to import Location data, but none of the location fields are showing in the node import wizard.

I'm new to Drupal and still in the process of finding my way around. I created the location content type using CCK. From what I understand node import supports Location but not CCK Location. How can I create a Location fields(long, lat and address) that can be used to display pinpoints on a google map, without using CCK?

Apologies if I should be raising this query elsewhere.

subscribing

This event.inc in post #1 seems to have trouble with dates. Is there an updated version that works with the latest D6 Node Import module?