Update existing nodes on import

this is very interesting... could this be adapted to ignore empty fields, so that we could basically import one new column of data into our nodes (say we wanted to add a new field to existing nodes, and had it in a spreadsheet).

thanks!

Yes, that's exactly what I'm saying. If I don't select a column during the "mapping" stage, it should keep the existing value for that field.

There could be a node ID column that, if a column is mapped to it, would automatically perform the query and update the existing nodes instead of creating new ones. Even better would be to allow one field (other than nid) to be set by the user as the "primary key" for checking if a node already exists and updating in that case.

This is a great hook, thanks a lot.

Hi John, shouldn't there be a patch for review and future commitment?

Subscribing

Hey,

This looks like a very valuable feature. Could someone please tell me how i would go ablout implementing this i.e. where would i place this code and is there anything else i need to do to get it functioning, as i haven't been able to find a way so far.

Any help would be appreciated

Thanks

Alex

Thanks for the help John

I haven't got around to modifying the module for my needs yet but it looks like a great feature and one that i hope will be included in the r5 release.

Alex

The ability to update nodes is a very welcome feature and should be added.
Thanks for your hard work!

Thank you so much!

This is a different approach than #9, but should allow users to skip the custom code. I'm not sure there is anything wrong with the hook_form_alter approach, so long as users know how to write the db query. It could also break if the db location of the cck field changes.

This patch (against 6.x-1.x-dev) should:

1. Add a "key" selection field to step 5 of the node_import process (only for node imports)
2. If a column is selected as a key, get the existing nid and insert that into $values before the node is updated/created.

Still lacking:

1. Allow non-cck columns as keys (title and nid could be useful)
2. Limit the types of columns that can be selected as a keys. You can pick anything that's mapped right now, but not everything will actually work
3. Fix the output on the preview display so it isn't an ugly hack to indicate the node is being updated instead of created
4. Expand it to work with other object types
5. Testing

subscribing

Hi Tauno. I tried your patch with ubercart products.

It worked but I noticed that it only updated one of the columns e.g. I changed price & stock quantity, but the stock quantity update which is the last column is the only one that took effect.

Please confirm if this is a bug or how current version works. Thanks.

subscribing

+++++ Allow non-cck columns as keys (title and nid could be useful)

That would be great!

@zeezhao - it should update all the fields, but I haven't tested it much. I definitely haven't testing it against ubercart product nodes. I do have an ubercart site that I can try to test it against, but it will be a few weeks before I can get to it. Does a normal ubercart import work fine with all the fields?

I've tested it with Ubercart nodes and it works like a charm.

@tauno - thanks for your reply. Yes normal import works fine for all fields. I am also using a patch for stock levels - see:
http://www.ubercart.org/contrib/11013

So maybe this patch is causing conflict or, may be an issue related to the version of node_import 6.x-1.x-dev I am using. Using an old one from Mar 2009 I had, as the latest (from 2009-Apr-22) did not seem to work.

Also, looking forward to the non-cck fields selection once it is available. Thanks

Hi there,

I would like to make the nid as the key value if using this patch,
Been trying to set the default value for a cck field to be the same as the nid to quickly achieve the desired results,
But having no luck,
Can anyone help with this?

Thanks in advance..

hi!
applied #13 by tauno for ver. 6.x-1.x-dev.
works nice. waiting for other fields to use as keys and inclusion into stable version of module.
thank you!

At first, thanks Robrecht Jacques for this great tool.

like psychoman, i'd applied the patch in #13, wrote by fauno. Thanks fauno.

WORKS WELL !!! in a test site, with basic cck fields. A little tweak in theme (garland). Not important, at all. I'll make some tests with the reall stuff and report any issue.

I would like to help Robrecht with code, but my skills in php is "near none" :). sorry.
@robrecht - how could I help with documentation? (translation for portuguese? or spanish?)

cheers for all

subscribe, looks very usefull :)

An update to the module that I definitely am interested in. Subscribing.

+1 for this feature. Subscribing...

Suscribing - this would be a great core feature

I needed to be able to import from CSV and update existing nodes identified by node id, not based on a key CCK field, so I adapted the patch from #13 to do that. This patch doesn't allow for the possibility of using a key field, but perhaps the final version should incorporate both features. As it happens the content type I'm working with didn't have any applicable CCK fields so I removed that functionality because I couldn't test it.

To use this, upload a CSV file with a Node ID field. If the field is blank, a new node will be created.

I see there's a lot of interest in this feature; if everyone who has subscribed can help test this patch I'm sure it will ready for commit soon.

Hi mvc,

I used both patches, 13 and your addition. Almost there...
My csv has also an image name for the CCK module. In this case, I got the error message "The path is already in use.". This is because the image is already found and imported in a previous run. This is correct and if the name did not change, no problems with that part.

Can you point me into the right direction or update it in your patch?

furtheron, I need to export as well the list to update the NID. which module is best to use? The products from my website are frequently updated by a csv file which comes from several suppliers.

Thanks in advance!!

@hessie: just to be clear, which CCK image module are you using? I personally only tested this with emimage (I submitted a patch for that elsewhere).

To export your nodes, try the CSV format feed display provided by the views bonus module.

Also, I never tested this with both my patch and #13 applied, so if you do that you're on your own, I'm afraid.

@mvc, thanks for your quick reply.

Changed the original node_import with only your patch gave me the same result. CCK is talking about that the path already exists.
How can i handle that exeption to ignore it because it is ok?

Did you mean by emimage, Embedded Media Fields?

The details you asked:
drupal core: 6.13
CCK version: 6.x-3.1
node_import: 6.x-1.x-dev (latest)
ubercart: 6.x-2.0-rc6

My 'problem' is?
I have a csv file with products. This one is attached in this mail. In there I have the SKU and an imagename. All other fields (exept for attribute color) I have nog problems.
The image is uploaded to a temp folder for drupal. With the import I link the image to the product. Only when an image name is already linked, it should not give an error.
The list is frequently update by the supplier, so this list contains new and current products. the current ones should update the current records in drupal, the new ones should be added.

I hope you have a solution. I do not yet master the drupal essentials, but i am getting there.

Did/Does anyone else has a similar 'problem'?

hope you can help me. Thanks in advance.

@hessie: Yes, when I said emimage, I meant Embedded Image Field, part of the Embedded Media Field project.

I can tell from your export that you're using a different module to handle images, but you didn't answer my question: which image module are you using?

I suspect that whatever module you're using is not supported by Node Import. This makes sense, because the image file can't be included inside the CSV file. So, the only time someone would want to support the image handling module you're using would be when updating existing nodes, which isn't possible without this patch anyways.

I suggest you either 1) switch to either storing images outside the nodes you wish to import, using Node Reference or Embedded Image Field, or 2) you write (or pay someone to write) a Node Import extension which handles whatever image handling module you're using. Good luck.

(PS: Changing issue title for clarity.)

@mvc: I am learning the vocabulairy. Thanks.

I use Image toolkit GD2 to handle the images. All within drupal. I hope this will answer your question.
The import goes great! Update however not. That one stalls on the node_import update. without everything goes as the software (including patch) is written.

However, I installed the embedded field image but will that one also take care of the images i put in the temp/files folder to process?

Thanks in advance for this answer. Then I know what to do.

@hessie: First of all, this issue is for updating existing nodes on import. So, I'm changing the title back. Please wait for that feature to land, and then you can file a new feature request to extend this to handle images better :)

No, you didn't answer my question about how you are handling images, you merely told me which toolkit you are using with the Image API module. What I asked is which module you use to attach images to a node.

The embedded image field module is used to reference images stored elsewhere on the internet, such as on Flickr or Picasa, not those in a folder on your webserver. I gather you are working with a CSV file from a supplier who will perhaps not want to start storing all their photos on Flickr, so perhaps that's not the solution for you. If you do decided to use the embedded image field module, you will also need the node import patch I wrote to handle that module: #565424: Support for embedded media fields

I gather you are not an experienced Drupal developer. So, my advice to you is to use the simplest solution. Just upload all the images to your webserver somewhere, and store the filenames in a plain CCK textfield. Then, in your theme templates, use the filename to generate the IMG tag to display the image.

applying tauno's patch in #13 against 6.x-1.x-dev I get the following SQL error in step 7:

user warning: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '= "PC-60A" LIMIT 1' at line 1 query: SELECT nid, vid FROM content_ WHERE = "PC-60A" LIMIT 1 in /var/www/html/sites/default/modules/node_import/supported/node.inc on line 293.

I can go on to step 8 and run the import, however new nodes are created.

By customizing the dbquery code, replace:
$row = db_fetch_array(db_query('SELECT nid, vid FROM {%s} WHERE %s = "%s" LIMIT 1', $key_dbinfo['table'], $key_dbinfo['columns']['value']['column'], $key_value));
with

$row = db_fetch_array(db_query('SELECT nid, vid FROM {uc_products} WHERE "$key" = "%s" LIMIT 1', $key_dbinfo['table'], $key_dbinfo['columns']['value']['column'], $key_value));

I 'm able to eliminate the error message, however new nodes are still created not updated.

Any ideas?

mysql 5.1.37

I don't wanna start more threads, so can I ask the status of node update on import? Seems there are two flavors cooking: using a NID and not. I need the "not" version, as my CSV has no knowledge of NID's but I don't mind requiring a unique key-like field whether it be Title or otherwise.

Is this working now in a patch-induced state or is its release pending? Thanks...

Is it likely one of these two flavours (of node import update patches) will be supported in a future release?

cheers
scotjam

I vote for "not NID". But a unique CCK field can be set as primary key, for example an article number.

For the record, I believe this module would ideally allow both possibilities. There's certainly no technical reason to choose just one or the other. Judging from this issue queue, both are of significant interest to the community.

So, I would say the next step is to merge the patches in #13 & #27, after which we can call for review. Marked "needs work" until this is done, and "feature request" because this is a new feature for node import.

Hi mvc,

Applying patch in #13 gives me errors...

* warning: mysqli_real_escape_string() expects parameter 2 to be string, array given in /blahblah/includes/database.mysqli.inc on line 323.

* user warning: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '= "" LIMIT 1' at line 1 query: SELECT nid, vid FROM content_type_store WHERE = "" LIMIT 1 in /blahblah/sites/all/modules/node_import/supported/node.inc on line 293.

Got no idea what this means, but I thought it might be useful to share.

best wishes
scotjam

Quick Update. I'm getting the same error as 'scholar' see #34 above.

+1 for this feature. Subscribing...

For anyone interested, I hard coded the product SKU field into node.inc
Replace:

$row = db_fetch_array(db_query('SELECT nid, vid FROM {%s} WHERE %s = "%s" LIMIT 1', $key_dbinfo['table'], $key_dbinfo['columns']['value']['column'], $key_value));

With:

            $row = db_fetch_array(db_query("SELECT nid, vid FROM {drupal.uc_products} WHERE $key = \"$key_value\" LIMIT 1", $key_dbinfo['table'], $key_dbinfo['columns']['value']['column'], $key_value));

The error I was getting in #34 is caused by an empty array call. The above code will only suit people updating products in Ubercart 2 as drupal.uc_products is a hard coded table.

scholar,

Thanks for the solution.
I have the same error on a CCK field, do you have any idea how I can solve it ?

This would be a fantastic addition to an already excellent module - Subscribing

I am going to set my sights on getting this working (to an extent) over Christmas, as I definitely need it for my website I am working on.
I will post findings here if I get it working!

Subscribing

Right, well it seems I'm all done getting it working. I built a custom module which I hard coded to accept the SKU as the field to use to check for duplicates.

Let me summarise what I have done

Downloaded latest dev release of node_import
Added support for stock and attributes, using this contrib http://www.ubercart.org/contrib/11013
Corrected the errors in the contrib file (uc_product_node_is_product changed to uc_product_is_product)
Added my node_update code (taken from John's post above) to my custom module and enabled it.

Imported. Job done.

My custom module code is below

//module below ensures existing nodes are updated, to avoid duplication
function update_nodes_form_product_node_form_alter($data)
{	
	$sku = $data['#post']['model'];
	if (is_numeric($sku)){
		
		$sku = intval($sku) ;
		$row = db_fetch_array(db_query("SELECT nid, vid FROM {uc_products} WHERE model = %d LIMIT 1", $sku));
		if (!empty($row)){
			$data['nid']['#value']= intval($row['nid']);
			$data['vid']['#value']= intval($row['vid']);
			$data['changed']['#value']= time ();
		}
	}
}

I think pretty much the only think I changed was the MYSQL query to check for the matching SKU, and getting the SKU in the first place ($sku = $data['#post']['model'];)
If you want to run a node update using a certain field, you can use change these two lines to look for the match, and BANG. Hopefully your node_import life will be as simple as mine is!*

If you need help, I'll be watching.

* I've only test-updated 5 records so far, but I've got a monster upload of 40,000 to test this week, so i'll get back to you on the ease of it.

One problem with the way I have been doing this update, is that if an image is added manually (which will be the case with my website) is that the image is lost on update. I know exactly why this happens - the node_import applies NULL to the image field of the product on update.
I am working to solve this, should be able to get some code on it for next week - My proposed solution is to copy the current image data from the record if not NULL and then apply the data to the product before update.
If anyone knows an easier way, such as getting node_import to ignore the image field and not try to overwrite it, please let me know. Thanks
Al

OK, So I spent a little time with this, and I'm not hugely impressed with the cleanliness of my solution, but it works for me, so I thought it would be worth posting.
I should make it clear this is going along the lines of Johns route, not changing the node_import core, as Robrecht made it clear in the past he wasn't going to do much on this. (http://drupal.org/node/349408#comment-1293786) Given the lack of updates in this module, you'll probably find this might not be implemented into core for some time.

So my code focuses on a custom module to solve my problem. Since my client will not be importing images using CSV, but will be adding them manually when required, I didn't want my import to overwrite the images already uploaded. To do this, I modified the MySQL to get the current image info and overwrite the imported data to keep the image information.

My code looks like so (modified from above post)

function my_module_form_product_node_form_alter($data)
{	
	//get my 'unique identifier' so I can find the product in the database
	$sku = $data['#post']['model'];
	//my SKU is always numeric, so this is fine to keep
	if (is_numeric($sku)){
		
		$sku = intval($sku) ;
		
		//select the product nid and uid from the database. Also grab all available image from the related table
		$row = db_fetch_array(db_query("SELECT uc_products.nid, uc_products.vid, content_field_image_cache.field_image_cache_fid, content_field_image_cache.field_image_cache_list, content_field_image_cache.field_image_cache_data FROM uc_products, content_field_image_cache WHERE uc_products.model = %d AND uc_products.nid = content_field_image_cache.nid LIMIT 1", $sku));
		if (!empty($row)){
			$data['nid']['#value']= intval($row['nid']);
			$data['vid']['#value']= intval($row['vid']);
			//set the data to equal the current information. An IF statement could be used here to check if there is an image being imported, and therefore just leave it to overwrite the old image, but I don't need to do this.
			$data['#post']['field_image_cache'] = Array(Array('data' => $row['field_image_cache_data'], 'list' => $row['field_image_cache_list'], 'fid' => $row['field_image_cache_fid']));
			$data['#post']['cck:field_image_cache:data'] = $row['field_image_cache_data'];
			$data['#post']['cck:field_image_cache:fid'] = $row['field_image_cache_fid'];
			$data['#post']['cck:field_image_cache:list'] = $row['field_image_cache_list'];

			//Set the node updated time so that there are no complaints later :-)
			$data['changed']['#value']= time ();

		}
	}
}

Note: This does not check if an image is being imported, but I guess it could be easily be implemented if you want it. Hope that helps some people out.

subscribing

ok, one step forward and two steps back -

I have had issue after issue with node_import. I really have to have this for my solution to work. I decided to start completely over. So I per #47 I did the following:

**Downloaded latest dev release of node_import
**Added support for stock and attributes, using this contrib http://www.ubercart.org/contrib/11013

x**Corrected the errors in the contrib file (uc_product_node_is_product changed to uc_product_is_product) - I could not find where this needed to be corrected. I assume the latest dev version already contains this fix?
x**Added my node_update code (taken from John's post above) to my custom module and enabled it. - not ready to attempt this step yet. Want basic upload working first.

Now I get a new error I have never seen before and can't seem to find a solution in the forums....

warning: Parameter 2 to drupal_retrieve_form() expected to be a reference, value given in C:\xampp\htdocs\drupal\modules\node_import\node_import.inc on line 1929.

ANyone have any thoughts..... i'm a beginner and I am running in circles with this thing. Please helps. Thanks!

this is a longshot and I haven't really read into it fully, but it looks like the retrieve_form() method wants a reference instead of a value as the second parameter. On that line in the function call, try adding a '&' (ampersand) before the second $variable you are passing into drupal_receive_form(). i.e. you call drupal_retrieve_form($form_id, &$form_state) instead of just $form_state.

Thanks goodeit! That seemed to get me past the errors. I continue testing but incase anyone else hits this issue I changed the following

Original -
function node_import_drupal_execute($form_id, &$form_state) {
$args = func_get_args();
---- $form = call_user_func_array('drupal_retrieve_form', $args);
$form['#post'] = $form_state['values'];
$form_state['must_validate'] = TRUE;
drupal_prepare_form($form_id, $form, $form_state);
node_import_drupal_process_form($form_id, $form, $form_state);
}

Updated -
function node_import_drupal_execute($form_id, &$form_state) {
$args = func_get_args();
++++ $form = call_user_func_array('drupal_retrieve_form', &$args);
$form['#post'] = $form_state['values'];
$form_state['must_validate'] = TRUE;
drupal_prepare_form($form_id, $form, $form_state);
node_import_drupal_process_form($form_id, $form, $form_state);
}

I'm not a developer.... but i'm working on it!

quick update - everything seems to be importing correctly. I have tested multiple product/attribute combinations. I also have tested with multiple vocabularies and everything is working fine. I have found one slight issue with the attribute.inc file obtained from here: http://www.ubercart.org/contrib/11013. When inserting multiple attributes it sets the initial attribute to inactive and stock level is set to 0. I will submit a support request in that project.

Hello

I've implemented http://www.ubercart.org/contrib/11013 but can't get the attribute and stock fields to show up in my form, all the rest seems to work as it should. Any ideas?

Rgds Claes

Hello Alzum

How do I implement your custom module. I'm a beginner but know som PHP and am trying to set up a ubercart store.

Rgds Claes

Hi Claes,

I am not by my main computer at the moment, but when I get the chance, I will see if I can upload a file attachment with the module, save you the fuss.

In the meantime, if you want to try, all you need to do is create a .info file, and a .module file. All that needs to be in the .module file is the code I pasted above. Have a look at other modules to get an idea of what the .info file looks like, you should be able to do it from there.
I'll be able to post my code up tomorrow I hope.

Regards
Al

Here you go Claes. Just unzip and install.
Hope it works ok for you!
Let me know if there are any problems with the file. I literally just did a quick edit and zip.

subscribing and wishlist:

I've used import node to import CSV files containing the following
- Partnumber
- Machine
- Model

A partnumber can fit in multiple machines, so instead of creating a new node for every combination it would be usefull to be able to update the Partnumber-node with the new Machine-details.

For example I have partnumber 40K1044 which fits in machine 8863. Later this year it will also fit in the new machine 8864, so updating would be saving me nodes. Currently I have 68,623 nodes which could less more.

Any help is appreciated and looking forward for the updates on the module.

Just downloaded the latest version and it looks like node update is working!

scotjam,
which version did you download? was it my module?
It works for me, but if it is also working for a few other people, I could set up a separate module to focus on node_update (or uc_node_update). It would be more sensible than having this almighty thread running about the topic.
I am in the process of getting a CVS account for another module I am working on. If I am successful, I will put in an application to get a module set up for ubercart node updating.
Thanks
Alun

Alun,

My mistake. I thought the latest version of node import had built in support for updating nodes, but I was wrong. I made a mistake thinking that my existing nodes where being updated when they were in fact new ones. It was only once I deleted every single node that this became apparent.

I'm not an ubercart user, so I'm trying to find a plain solution that lets me update existing nodes. As far as I can tell there are two solutions in this thread for normal node updates, one is #13 by tauno, another is #27 by mvc, after this the rest of the thread focuses on ubercart.

I've tried the previous two and I couldn't get them working before, and then I assumed the latest version of node import had this issue already sorted, but it doesn't

Would / could your ubercart update solution work for normal nodes updates?

cheers
scotjam

Alun,
I am not sure splitting this module for updating (or ubercart) is a good idea. That makes twice as much code to maintain and will further hurt the development of this module. A better suggestion would be to become co-maintainer of this module, and work on proper updating support for all nodes, and then hook into the module and add ubercart support (as has been done). It would be really great to see some forward development and actual releases on this project; it would be a shame to see that come at the expense of maintaining support for updating all types of nodes.

Thanks to everyone who has been working on this!

I agree with goodeit. This module should be able to update all nodes, not only ubercart ones. I have created product nodes with CCK fields, such as an article number and 3 prices (numeric fields). It would be great if it is possible to update these price fields using the article field as reference.

My apologies - from reading the bulk of replies on the thread, I forgot that the discussion was originally focused on a general node import, and not just on Ubercart.
goodeit, I agree with the point you make about the module needing a co-maintainer. The issue queue alone needs a good sort out! My development work on Drupal is currently focused on a few other modules. If everything goes well, I would like to think I will have learnt enough to push forward with some development on node_import in the summer, but this is merely speculation at this point in time, I don't know what I will be doing in the summer!!

scotjam, in theory, the update code could work for other modules. You just need to make some changes to the db_query and a few other lines - I've commented it well to explain what my code does, so you could give it a shot. Its not an ideal solution, but it works!

periodic node_import from an external source is very much wanted here.
being able to exclude already known nodes or update these nodes would be very nice to have.

subscribe! :)

Alun, we definitely need a Ubercart product update module... Hopefully you will get your CSV account and once you do you can create the module yourself—once you have an account you can freely create modules. If you don't get approval immediately, perhaps I could set up the module for you, and then give you CVS access to the module once you are approved.

Hi Alun,

My friend was able to do the specific modifications to your code in #58 for my website and got it working :)

I understand that it wasn't too difficult to do.

Thanks for finding a solution to updating existing nodes using node import. It's massively increased the utility of this module.

cheers
scotjam

scotjam,

good to know it is working. I think I will be focusing some time in the future helping out getting node update working. My next website I am being hired to do includes the use of product attributes, which my previous work didn't. I hope, once I get a good understanding of how the attributes work, I can get a module like node_update working well.

I have a question and I am unsure where to ask it. I am using the node import (with all the additions listed in this thread - thanks to everyone for your work here... excellent utility!). I have encountered a strange issue. When I import data and then look at the related nodes the size drop down list is out of order.

Here is what I posted under the ubercart modual tracker but got to thinking this may be an import issue instead:

Can anyone help me understand why my attributes are out of order? I have imported my products with node import and my attributes "size" show up out of order. they should be 7,8,10,12 but they show up as 10,12,7,8. When I look under options in the product node they are listed out of order as well. However, under the stock tab they are in the correct order.... Thoughts?

Any help is appreciated!

subscribing

dbabbage,
I got my CVS access this weekend :-)
I will be adding node_update as a module as I think it is essential. However this won't be happening until the summer when I get some time to spend on it. I am currently focused on another Drupal project which I want in decent shape by the summer.
While I agree with goodeit and tomsm regarding keeping all of the development for this module in-house, I neither have the time or the knowledge required to maintain the whole of node_import, and becoming a maintainer just to add my update code would be selfish. If this changes (mainly the time aspect, learning is easy when you can spend time on it!) then I would happily become a maintainer and help out where I could. Until that time, I believe to respect Robrecht's wishes of keeping this module as an import module.
Should I get update working well, perhaps it would be wise to merge them in the future.
I will keep this thread posted on my work.

I don't understand. You say that you agree that all module development should be done "in-house" (aka here in this module, node_import, unless I am misreading) but yet you are creating a new module to do something that has largely been added to this one already. Why can't you become a maintainer just to add the patches and work on making the update capability work? I don't think you would be expected to completely take over the module, or even actively develop the rest of it. I have seen plenty of modules with "maintainers" that say up front they they do not have time to develop but would gladly apply patches from the community, once tested. I don't see that as a problem at all. These two functions are so closely related that it just seems there would be too much overlap to justify a separate module.

</$.02>

goodeit,
Perhaps you are right. I feel the same way with the node_import via cron module (a module that largely uses node_import but has a little added function), so I agree with your thoughts.
Either way, I'll be looking at working on it towards the summer. My main issue with putting update capability into this module would be stepping on peoples toes. Robrecht specifically stated here (http://drupal.org/node/349408#comment-1293786) that he did not want to add update capability just yet. Should this module continue to go unmaintained for the next few months, and therefore it would be safe to assume that Robrecht is not interested in the development of the module, I will apply for maintainers access and we can work from there.

I apologize for this newbie question..

I need to updae nodes also so I downloaded the patch to my local pc..but How do I apply a patch in drupal?

http://drupal.org/search/apachesolr_search/how%20to%20apply%20patch

Thanks, I saw that but I dident think I was looking at the right thing.

How can applyg a patch be so complex?

its talking about installing Cygwin... notepad ++

What the heck? How can it be harder then installing the actual module.

Am I looking at the right doc?

If you're running Drupal on a Windows-based server, and you don't want to install Cygwin, just open the patch in Notepad, study it up, and make edits manually (especially if it's not a large or complex patch).

Otherwise, if you're running Drupal from *nix the instructions at http://drupal.org/patch/apply are pretty straightforward.

Alun,
I imagine, and would hope, Robrecht would be open to someone assisting and maybe even adding update functionality :) either way, best of luck when you do start working with it!

JackThompson,
If you are on windows, you can apply a patch using patch.exe, a native port of patch that does not require cygwin, from the awesome GnuWin32 package. http://gnuwin32.sourceforge.net/packages/patch.htm

Play with it a bit and hopefully it won't seem so intimidating. good luck!

I am on linux server

Hi i have been using dev version of node_import to insert products into my ubercart store. All products from my csv have been uploaded successfully and taxonomy works well.

One of the main problems is that i cannot update the current products with another csv with the SKU, they are inserted as new products. I have read through most of these posts and cannot find a solution.

Post #47 seems like a possibility but i cannot work out where this should go and be used, any ideas?

Also is there a functionality to update stock levels of a product? I have checked some links but they are no longer available.

Any help would be much appreciated.

In post #58 you will find the custom module I created from #47.
It might not completely solve your update woes, but it should take you a few steps closer :-)

For the stock levels, have you added the stock level contrib to node import? The website has been updated, but the link should still work in a little bit.
http://www.ubercart.org/contrib/11013

Hope that helps!

Aha! that is fantastic. I shall give those a go, thanks so much for your time.

webspring -

I use the node import for adding products to my store but for managing stock and price I use a light modules I found here:

http://www.ubercart.org/contrib/12428

It is quick and easy for only updating the stock levels and price. A lot less clicks to kick off the file run too!

Give it a try.

I also wanted to ask if it is possible to use this module to only update images? I am no developer for sure so sorry if this is a stupid question. I use the node impost to upload products ll the time. Works great. What I have found I need now is the ability to map new images to existing nodes(products). Is this possible with node import? My wishlist would be to drop the image files into a folder on the webserver and have cron run some type of mapping scripts daily..... but I will be happy with manually using node import if it has the capability.

Thanks!

Hi razorback

I think we both are looking for the same. Why don't you try feeds?

http://drupal.org/project/feeds

it can import AND update nodes with a custom mapping.

well... we are coping with the maping thing (with no great success at present, I must say...) but its promising.

subscribe

Hi, aluns solution works great.
However I hava both numeric and alphanumeric sku's.
I tried several modifications to his module with no particular success?

Can anyone point me out?

Also it worked a couple of times but now i am getting

Fatal error: Cannot unset string offsets in blahblah/sites/all/modules/filefield/filefield_widget.inc on line 442

when i click next at step 6 of 8

@Agapiou
Good to know it is working. For it to work with alphanumeric values, you will need to take out the if statements
if(is_numeric($sku)){

and later take out
$sku = intval($sku);

Let me know what problems you have. It might be worth posting your code here.
As for the unsetting the string offsets, I'm not sure what that might be. Its not an error reported by Node Import though, as its being reported by filefield... have a search for the issue in filefield to see if someone else has encountered it.

Also in the querry i will have to change this part as well right?

... WHERE uc_products.model = %d AND ....

to

... WHERE uc_products.model = %s AND .... or ... WHERE uc_products.model = %'s' AND ....

?

I thinkg i tested it like this and was still creating duplicates. I will try to give it a go again tho.
If i can manage to surpass this Fatal Error. I was importing with no problems, did no particular changes anywhere else than the custom module trying to make it accept alphanumeric sku's and BANG !!! Out of the blue, now everytime i try to import at step 6 - > 7 get the error. Will try to read some forums see why it happens.

Either you just had a mistype or that is your problem -
You should do this WHERE uc_products.model = '%s' AND
But you just typed this WHERE uc_products.model = %'s' AND
Notice the difference in location between %'s' and '%s'
If that is in your code, it will be causing the problem, I assume it is just a mistype though?

What type of file are you using? I always find that TSV files provide a better result that CSV files, as you don't have to worry about string delimiter values and such.

Hmmm,
When i disable the module import works fine. When i enable the custom module and try to import a product works fine. When i try to update the content of the imported product leaving the sku untouched and editing the description for example the filefield module throughs the above mentioned error.

My question though is that i saw it work. i did several tests at first and products where being updated normally. All of a sudden it stoped working and gives that error. What changed? I didnt change change anything anywhere else besides importing several different csv's to test that everything works as it should.

Oh thanks Alun. Probably was that mistype error.

I am using csv.

Subscribing

Hi Ive been following this and other thread on the topic of updating ubercart products. Ive got as far as updating products based on the sku and not overwriting the images with null however My taxonomies are getting overwriten by null when updating these nodes. Im trying to put together code like that used to save the image information. Anyone got any insight?

Hi Alun, I'm a beginner in Drupal with very little PHP programming experience...
I've taken your module and tried to modify it to update one of my CCK content types
(not UberCart content type)

My CCK content type is called "record" and I am trying to perform update based on one of its unique fields called "filename" (alphanumeric).

My updates just end up as duplicates.. as if nothing happened.
Please could you (or anyone) take a look and tell me what I'm doing wrong?
I'm really a beginner :)

<?php
// $Id$

//module below ensures existing nodes are updated, to avoid duplication
function my_module_form_record_node_form_alter($data)
{	

$sku = $data['#post']['field_field_filename_value'];
         	//if (is_numeric($sku)){
		//$sku = intval($sku) ;
		$row = db_fetch_array(db_query("SELECT content_type_record.nid, content_type_record.vid, content_type_record.field_filename_value FROM content_type_record WHERE content_type_record.field_filename_value = %s", $sku));
		if (!empty($row)){
			$data['nid']['#value']= intval($row['nid']);
			$data['vid']['#value']= intval($row['vid']);
			$data['changed']['#value']= time ();

		}
	//}
}

Just a quick check.. shouldn't

$sku = $data['#post']['field_field_filename_value'];

be

$sku = $data['#post']['field_filename_value'];

?

That might be an issue. What I did when I wasn't sure was lots of printing to the screen. If you want to check if you are getting the values to compare correctly, try printing it out and calling die(). If you are printing arrays, call print_r($array) and it will show you the whole array!
it blanks your screen and shows the value you printed out. Not the nicest way to go through code, but it shows what you are looking for.
Hope that helps.
Al

Hi Alun:

Thanks a lot! After some debugging with your print_r() method, update is working now..
and as you pointed out

$sku = $data['#post']['field_filename_value'];

for some reason had to be changed to the following, (I still don't know why)

$sku = $data['#post']['cck:field_filename:value'];

Here's the code if anyone is interested..
thanks again, Alun!

<?php
// $Id$


//module below ensures existing nodes are updated, to avoid duplication
function my_module_form_record_node_form_alter($data)
{	

$sku = $data['#post']['cck:field_filename:value'];

if (!empty($sku))
	$row = db_fetch_array(db_query("SELECT content_type_record.nid, content_type_record.vid, 	content_type_record.field_filename_value FROM content_type_record WHERE 	content_type_record.field_filename_value ='%s'",$sku));

		if (!empty($row)){
		
			$data['nid']['#value']= intval($row['nid']);
			$data['vid']['#value']= intval($row['vid']);
			$data['changed']['#value']= time ();			

		}
	//}
}

After a few hours of going back and forth over this feature I found my way.
I'd like to share in case there are any other newbies like me who may need a leg up.
Firstly I downloaded 6.x-1.x-dev and uninstalled my working module.
I then applied the patch manually because I'm on windows and found it easier.
Basically just remove the few lines of code and replace them with the patch code.
I then ran into the problem of #34 and after reading #42 by scholar I made the changes to the file again,
the only difference is that I replaced the table uc_products with nodes.
It is working now although I am still stuck with multi-groups but that is another issue I will now move onto.

subscribe

Is there anything available like #13 or #27 that works against the latest rc4 or dev release?

See full version I posted here based on: http://drupal.org/node/349408#comment-2260858

You can easily comment out the hardwired line for ubercart. It is based on an earlier dev version, but more recent than rc4.

Zeezhao:

Thanks for the reply. I tried your version however i'm getting tons of unknown column errors.

Unknown column 'offset' in 'field list' query: INSERT INTO node_import_tasks (name, uid, created, changed, fid, has_headers, file_options, headers, type, map, defaults, options, offset, row_done, row_error, status) VALUES ('test.csv', 1, .............................

That's only the first 2 lines or so.

What am I doing wrong?

I can't really say what's causing your errors. Works for me on drupal 6.x. I have not upgraded node_import versions since then.

Depending on which version of node_import you ran before, it may have changed tables when you ran update.php. So maybe your database no longer compatible with this one.

Mustang, that error means your code is reffering to one version of node_import while your database is running another. I believe offset got renamed to file_offset between -rc4 and the current dev version, hance your error "unknown column". I could be wrong, and its the other way around, which version of node_import are you running, and did you run update.php when you updated your node_import version?
Thanks
Alun

Yea. I had the new dev version of node_import installed earlier. I ran through the module uninstall process however the tables are probably still in my db.

I just published a project using the code from #58 above: Node Import Update.

Please focus discussion and development efforts there, as we now have an issue queue devoted to adding update functionality to the excellent Node Import module. Should the maintainer(s) of Node Import ever want to roll this project into their own, it will likely be easier this way, rather than wading through a bunch of stale patches scattered across multiple issues.

BTW, I am actively seeking co-maintainers, as I won't be able to devote a tonne of time to this project myself.

I've installed the node Import Update module but there is no differend: no update form, no choice to choose to make an update instead.

Anything lese to do for using the update module?

All -

I have been struggling with what seems as if it ought to be a core feature of Drupal or for any Content Management System for that matter. I do realize and appreciate how Drupal works and that everyone who contributes is normally donating to the community. Amazing group, amazing software - I just can't believe this issue would be so sketchy.

I have a reasonably large Drupal site that has many thousands of nodes. There are many relationships with about 15 or so Content Types. The relationships are defined using mostly nodereference fields, backlinks in views and to some extent nodereferrer. I am using Ubercart. I am on Drupal 6.16. Any other information that anyone needs will be gladly provided.

I imported these nodes for the most part into Drupal using Node Import. It took me a long bumpy time but I finally figured out how to get the multi-value fields to work (the text field widget must be a Select List).

Here is what I (and I have to think many other people) need to do. Editing one node at a time is not only foolish it is probably impossible. Once the nodes are entered some will get changed - in fact one way to make this process sweet is to fully edit one node as a template, export it and then add rows using the exact format of the csv file. The only thing that makes sense to me - unless there is some easy XML way - which I looked at but found daunting) is to export the existing nodes out as a CSV file - including the multi-value fields. Then work on this file using Excel, with sorting, search and find, cut and paste, etc. I would call that "bulk edit" but that term gets me nowhere in Drupal. Then, once edited I need to reimport this csv file back to Drupal adding new nodes as appropriate and updating old nodes based on some key field - exactly the sort of thing that is being discussed in this long thread.

This is of course critical for updating prices

Many hours later....

Here is what I have found out:

1) I tried Feeds. Feeds does a good job with importing and updating nodes. However, it does not map multi-value fields and that doe not seem to be on the developer's agenda.

2) I have looked at Table Wizard and Data but Feeds seems to be the most useful of the bunch.

3) Node Import - does import multi-value fields - reliably (use || between values and use the Select Widget for the text field).

4) If I import nodes using the Node Import Module then I can't get Feeds to update the nodes that are imported using Node Import because it is looking for GUID as the key field - which is created by Feeds - and not there unless Feeds was used to import the nodes in the first place.

5) The node that is the product of this thread, called Node Import Update seems like it might work. However, the documentation is this thread and, not being a coding developer I find this confusing and really a non-starter for me. You have to hack the code to tell it which field to use as the key field and what node type you are working with. Even that is not obvious and I can't get it not to overwrite previous nodes.

6) I am able to reliably export to a cvs file using the Views Bonus Pack with a Feed page. The multi-value fields don't have the delimiters but I can add them in using a search and replace for the most part. Not perfect because of this problem but still the only solution I have found that is close.

7) I have tried VBO and Editview. VBO is useful for bulk editing a few fields perhaps but is way too dependent on computer memory and is unstable when you are looking at lots of nodes. Nowhere near what Excel would do for developers. Editview is even worse. You have to save each node as you edit it and that can take a long time.

8) By the way I had to disable Feeds because it interferes with Node Import.

I do have a wonderful developer I am working with to help with this problem but I thought I would also put a cry of help out there to see if anyone has worked this out. It just seems so obvious that I wouldn't have even imagined it would be the issue that prevents my web site from being useful.

I don't know if any of this will help you. I do need to get this updating working pretty soon.

I have been happy as well with NI, and the update issue does bug me as well. Thought that I would give you a head's up on #6 above - try this #366120: Provide for cck fields that render custom node structure. and it has worked flawlessly for me.

And thanks for the feeds/NI interaction - I had not yet figured that one out!

You might want to take a look at UUID (http://www.drupal.org/project/uuid) and my post #827466: UUIDs for export/import regarding content import via UUID, but there isn't anything useful now.

Basically I imagine a patch for node export/node import that updates existing nodes based on UUIDs and not NIDs or some other key.

Subscribing ...

Hi,
I used the patch #27, when I select "none" and it deleted all my existing data on the TEXT AREA,.. . how do I let the module treat "NONE" means no change in the TEXT AREA, skip and continue.
thanks

Subscribing

subscribing

I followed some of the steps liste by @Alun in #47, and create my module based on #58 and #99 (I'm not using Ubercart). It works very well. I can import nodes and then re-import to update additional fields not present in first load. However, I can't figure out how to manage fields that can/cannot be changed.

I thougth this part of the code would be responsible for this.

            //set the data to equal the current information. An IF statement could be used here to check if there is an image being imported, and therefore just leave it to overwrite the old image, but I don't need to do this.
            $data['#post']['field_image_cache'] = Array(Array('data' => $row['field_image_cache_data'], 'list' => $row['field_image_cache_list'], 'fid' => $row['field_image_cache_fid']));

So I used something like:

            //set the data to equal the current information.
            $data['#post']['cck:field_code:value'] 		= $row['field_code_value'];
            $data['#post']['cck:field_description:value'] 	= $row['field_description_value'];

both fields are in the SELECT statement. The import works fine, no duplicates generated. However, this fields are being updated and it shouldn't.

Am I missing something. Please note that I'm kind of new to Drupal development.

Thanks.

Hello! Thank you for your patches! Pretty cool!
Can I import nodes with widget type: select list and define column "keys" to my values in csv?

Thank you in advance.

Hi! I made a great progress since #117. Work is almost done. The only missing piece is how to make the taxonomy change according to some values passed in the csv.

Trying to explain it better. What I need to do is to take field "Item" from the csv and do a lookup in a database table to determine what taxonomy this record should use, then I have to setup the node taxonomy according to the lookup result.

As you can see my question isn't really about node import functionality, but how to programatically setup a node's taxonomy.

Any help pointing me to the right direction is very much appreciated.

Thanks.

k3n3dy, glad my code helped in #117, sorry I couldn't have replied to help then.
All the taxonomy information is stored in the database in the tables starting with term_
So you need to get the node id, and the taxonomy id of the term you want to use and then set it up in the table.
Thats one way anyway, otherwise you could alter the node object directly, but you need to know how the taxonomy is stored in the node object to do this. print_r($node); always helps when dissecting the object :-)
Good luck.
Alun

Thanks again for all the help Alun. I have most part of it sorted out, thanks to your directions. However I'm stuck in the last part, which is to make the node keep current taxonomy instead of what is being imported.

I can get node's current taxonomy with $terms = taxonomy_node_get_terms(node_load($data['nid']['#value']));, which give as result:

Array ( 
	[3149] => stdClass Object ( 
		[tid] => 3149 
		[vid] => 14 
		[name] => R8 
		[description] => 
		[weight] => 0 ) 
	[3015] => stdClass Object ( 
		[tid] => 3015 
		[vid] => 16 
		[name] => Audi 
		[description] => 
		[weight] => 0 ) 
	[188] => stdClass Object ( 
		[tid] => 188 
		[vid] => 5 
		[name] => Sports 
		[description] => 
		[weight] => 0 ) 
	[187] => stdClass Object ( 
		[tid] => 187 
		[vid] => 5 
		[name] => Autos 
		[description] => 
		[weight] => 127 ) 
	[211] => stdClass Object ( 
		[tid] => 211 
		[vid] => 4 
		[name] => Joe's Best Autos 
		[description] => 
		[weight] => 0 ) 
	)

print_r($data); on the other hand provides as result for the taxonomy data:

[taxonomy] => Array ( [17] => [13] => [4] => 211 [5] => Array ( ) [tags] => Array ( [14] => ) [16] => )

How do I make $data taxonomy equals $terms?

Thanks.

Subscribing

subs

I really just need to update a date field on my nodes. Is this possible to do?

Subscribe. This is very interesting ... and desirable functionality.

phelix: you might want to try the latest dev release of VBO instead. The "Modify node fields" action now allows that.

There was some progress with Node Import Update, now there is no need to touch the code to set it up. Have a look at it, hope it helps.

http://drupal.org/project/node_import_update

Hi

I modified the existing node import codes to allow node update and create a sandbox project for it. Feel free to have a look.

http://drupal.org/sandbox/hyu/1819292