The documentation surrounding this module and its parent a very sparse. I understand that they are both in alpha, but it would help me test this stuff out if I could figure it out.

Anyway, I figure even if I understand how to use these modules, I may need to code some additional stuff and that's OK.

Could someone point me in the direction I would need to go to import a CVS file as nodes where each line becomes a node that has its columns mapped to CCK/node fields?

Does a pipeline always start with a 'pipeline parameter' block and end with a 'pipeline outputs' block? What other blocks would I need along the way and what blocks might I need to code myself? I see a few likely candidates like 'Read file contents (line by line)', 'CSV text lines to list of records', 'Extract single item from list', and 'Evaluate PHP code', but I'm not sure what they do and how to set them up.

Any help would be great. Of course, if I need to write something, I'll contribute back.

Comments

jpetso’s picture

jpetso commented

Hey, sorry, I haven't been able to take the time to write up the tutorial that I had planned, and as my thesis deadline is dangerously near already, I'll only tackle this when I'm done with thesis work (>= May 12).

The short version goes like this, three pipelines:

1. A pipeline for extracting the first line out of the CSV file, for mapping purposes in pipeline 2. Includes "Read file contents (line by line)", "CSV text lines to records", and "Extract single item from list". Wiring is (hopefully) straightforward, remember to also connect the output of "Extract single item from list" to a pipeline output. When you execute that pipeline, you can see the first line showing up as result.

2. A pipeline that takes a single CSV line, and creates a node from it. Includes "Extract items from structure (simple mapping)", "Create node object", "Set fields in node object", and "Save node". The trick is with the mapping operation: edit the "key map" input (click on "key map", then on the "edit" icon above) and assign the result of pipeline 1 with "Import value from pipeline". (Alternatively, you can just provide an array of the form array(0 => "First column name", 1 => "Second column name", etc.) in the plain input value edit widget.) If that worked, the mapping operation then shows the columns as outputs that can be connected to the inputs of the "Set fields in node object" operation. Now, just connect the mapped "item structure" to a pipeline parameter, the "created node object" to the "existing node object" input of "Set fields", the modified node object to the "Save node" operation, and optionally the saved node to a pipeline output.

3. Lastly, a pipeline that decomposes the CSV file into a list of records, and runs pipeline 2 on each record. Includes "Read file contents (line by line)", "CSV text lines to records" (a.k.a. CSV parser), and a "Foreach" operation that executes pipeline 2 on the resulting list. If the first CSV line represents the header, you want to enable the "Skip first line" option in the CSV parser operation. Execute, and it might just work.

Well, the quick write-up is not that bad actually, just a bit short on exact explanations. I hope that this roughly conveys the idea, and that I didn't forget anything. Leaving the issue open, this is definitely something that should be in the handbook (in a bit more refined form).

Island Usurper’s picture

Island Usurper commented

OK, I got it to work from following this, though I just dumped the data instead of saving the nodes, and I loaded the ones I already had instead of making new ones.

One thing to mention in future versions of the documentation is that you need to make sure that the inner operation of the foreach actually has outputs before trying to wire the "list of outputs" to something. Since I was just dumping data to status messages, my third pipeline was trying to draw on resources that didn't exist, which caused a nice hard backtrace. Disconnecting that output worked fine, as I'm sure actually outputting something would as well.

To get back to the original question, handling CCK data is probably more in the scope of transformations_drupal, or possibly worth its own module by itself.