Hi,

I'm not sure if this was asked before; since the Amazon associate tools module is abandoned for over a year and the maintainer already declared that he will not port the module to Drupal 6.x, the question arises, if there will be an upgrade path (or something like this) for data from Amazon associate tools module.

Thanks & regards, -asb

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

eaton’s picture

Version: » 6.x-1.0-beta1
Status: Active » Needs work

Yes. I've got a very rough skeleton 'Amazon Associate Tools legacy' module that takes control of AAT's two existing node types and allows any existing data to continue in use. Moving forward, though, I'm hoping to keep the CCK field as the primary integration point with the node system.

reikiman’s picture

Rather than preserve existing amazon nodes I'd like a route where those nodes can be converted to an equivalent CCK-based node type. On one of my sites I have hundreds of amazon nodes.

asb’s picture

Same here, with several thousands of "Amazon" and "Amazon item" nodes on a bunch of sites. However, migrating them to CCK-based node types programmatically is not trivial, if you have a mixture of DVDs, CDs, books and other stuff amazon is offering.

It might be possible to code a solution for sites facilitating exactly one product type from Amazon (like books), or for sites with very solid tagging.

I think, most of us will be stuck with 'Amazon Associate Tools legacy' module for a long, long time. I'm pretty glad that we have at least this migration path!

Greetings, -asb

Ashford’s picture

Sorry to be redundant and ask for clarification, but the info I've been reading in regards to the Amazon Assoc Tools is conflicted.

This is my understanding based on what I just read in this post. The basic affiliate links will work in the new Amazon module, but you are just trying to further refine it to do something else.

In other words, if I used the Drupal 5 + the regular amazon assoc tools, I could upgrade to Drupal 6 + the new amazon.module and my affiliate links would continue to work?

Thank you for sharing your knowledge with those of us Drupal users who do not have your skills. I bow to you each day before turning on the computer.

Ashford

bomarmonk’s picture

I just upgraded to Amazon for Drupal 6, and I cannot get this module to work at all, starting with new CCK fields and data. The old associates module worked for me though, so I'm not sure if there aren't some major bugs affecting people with this. See http://drupal.org/node/269335
I'm glad Amazon wasn't "mission critical" for me. If it is critical for you, I would stick with Drupal 5 or test with a development environment first.

asb’s picture

Version: 6.x-1.0-beta1 » 6.x-1.0-beta5

@Ashford:

> This is my understanding based on what I just read in this post: The basic affiliate links will work in the new Amazon module,
> but you are just trying to further refine it to do something else.
> In other words, if I used the Drupal 5 + the regular amazon assoc tools, I could upgrade to Drupal 6 + the new amazon.module
> and my affiliate links would continue to work?

I'm still not sure what Amazon.module for D6 is supposed to do and what to do not, but as far as I can see, it only offers a partial upgrade path, at best. My testings so far show that you will lose all "amazon" and "amazon-node" content; those Amazontools-specific content types are IMHO not handled (i.e.: not migrated) through Amazon.module at all; the nodes still exist in some kind of limbo, but don't show affiliate links anymore, and can't be edited. Of course, your Amazontools setting aren't handled by the upgrade, also.

In my testing scenarios, neither CCK fields of the "ASIN/ISBN" type were migrated, nor "Related product(s)" attached to other legacy content types like "story" or "blog" through an "Related Amazon.com product id" field: They don't show up after the migration, either. As far as I have learned so far, practically any Amazontools content gets lost during the migration. However, I'll have to investigate this further.

Please keep in mind that the Amazon.module is as of this writing still Beta and it never promised a full upgrade path for Amazontools module. I think we'll have to stick with D5 for some more time; maybe someone is able to contribute a migration script, similar to the migration from Image.module to CCK Imagefield.

Greetings, -asb

asb’s picture

Btw...:

> Yes. I've got a very rough skeleton 'Amazon Associate Tools legacy' module that takes control of AAT's two existing node types
> and allows any existing data to continue in use.

@Eaton: I seem to be unable to enable this 'Amazon Associate Tools legacy' module; is this already part of the new "Amazon" module, or more some kind of concept in your huge drawer of Drupal treasures??

Greetings, -asb

asb’s picture

Priority: Normal » Critical

Hi,

I'm sorry to sound impatient - but: Is there any progress regarding an upgrade path for the Amazon associate tools module since March 2008?

On our D5 sites more and more contributed modules become "unsupported", thus "Update status" issues warnings on a daily basis; because the Amazon associate tools won't be upgraded to D6, my upgrade path is blocked; I'm not even sure how long Drupal 5 core will be supported.

So please give us an update, or a short notice, if there's anything we can do to help.

In the meantime, I'm changing the status to "critical"; please change it back if you disagree.

Thank you very much & greetings, -asb

EvanDonovan’s picture

I noticed that the aat_legacy directory has been emptied on cvs.drupal.org. Is that because the legacy support didn't work very well? Are there any plans to bring it back? This is a critical feature for our site...

EvanDonovan’s picture

Status: Needs work » Needs review

The code posted below is a rough version of a script to import Amazon Associate Tools nodes into the new Amazon module. Note that it still contains code specific to my site's implementation of the module (i.e., the part of the code that imports taxonomy terms), but this should at least give you a base to build off when creating your own import solution.

Also note that the code has *not yet been tested* on more than one node at a time. If that scares you, wait & I should have a tested version soon. I thought considering how important this feature is that I should get this out as soon as possible.

To run this code, just put it in a page & set the input filter to PHP code. A more refined version should be coming.

/* 5.x Amazon.module -> 6.x Amazon.module import script by EvanDonovan, 4/17/09
Based on http://www.lullabot.com/articles/quick_and_dirty_cck_imports
Cf. the comments, starting with the one titled "D6" */

module_load_include('inc', 'node', 'node.pages'); // Needed for drupal_execute as per API docs
$amazon_nodes = db_query("SELECT * FROM {amazonnode} LIMIT 1"); // Get nodes from Amazon.module -- remove the LIMIT 1 if you want to try this with multiple nodes at a time
while ($amazon_node = db_fetch_object($amazon_nodes)) { // Iterate through the nodes 

  // Make an Amazon book node & create an alias for it
  $node = array('type' => 'amazon_book', 'pathauto_perform_alias' => 1); 
  
  $amazon_details = db_fetch_object(db_query("SELECT * FROM {amazonitem} WHERE asin = %d", $amazon_node->asin)); // Get the extended info for the old Amazon nodes

  $form_state = array(); // form_state is needed for drupal_execute
  
  // Set Node Basics
  $form_state['values']['title'] = $amazon_details->title; // Node title
  $form_state['values']['teaser'] = $amazon_details->editorialreview; // Node teaser
  $form_state['values']['body'] = $amazon_details->editorialreview; // Node body (the same)
  
  // Set CCK Field
  $form_state['values']['field_amazon_item'][0]['asin'] = $amazon_node->asin; // ASIN (needed to pull data from Amazon API)
  
  // Get Taxonomy Terms
  /* $old_node = new stdClass();
  $old_node->vid = db_result(db_query("SELECT MAX(vid) FROM {node_revisions} WHERE nid = %d", $amazon_node->nid));
  $terms = taxonomy_node_get_terms($old_node); */
  /* This method is too awkward b/c of the way that taxonomy_node_get_terms returns its results.
  Instead I will go 1 vocabulary at a time & use the helper functions from um_common.module */
  $voc_12 = taxonomy_node_list_terms_by_vocabulary($amazon_node->nid, 12);
  $voc_3 = taxonomy_node_list_terms_by_vocabulary($amazon_node->nid, 3);
  $voc_7 = taxonomy_node_tid_array_by_vocabulary($amazon_node->nid, 7);
  $voc_10 = taxonomy_node_get_tid_by_vocabulary($amazon_node->nid, 10);
  $voc_8 = taxonomy_node_tid_array_by_vocabulary($amazon_node->nid, 8);
  $voc_43 = taxonomy_node_tid_array_by_vocabulary($amazon_node->nid, 43);
  $voc_11 = taxonomy_node_tid_array_by_vocabulary($amazon_node->nid, 11);
  $voc_24 = taxonomy_node_tid_array_by_vocabulary($amazon_node->nid, 24);
  $voc_4 = taxonomy_node_get_tid_by_vocabulary($amazon_node->nid, 4);
  $voc_5 = taxonomy_node_tid_array_by_vocabulary($amazon_node->nid, 5);
  
  // Set Taxonomy Terms
  if($voc_12 != '') { $form_state['values']['taxonomy']['tags']['12'] = $voc_12; } // Speaker/Author (free-tagging, terms as comma-delimited string)
  if($voc_3 != '') {$form_state['values']['taxonomy']['tags']['3'] = $voc_3; } // Other Tags (free-tagging, terms as comma-delimited string)
  if(count($voc_7) > 0) { $form_state['values']['taxonomy']['7'] = $voc_7; } // Audience (array of tids)
  if($voc_10 != '') { $form_state['values']['taxonomy']['10'] = $voc_10; } // Category (tid as string)
  if(count($voc_8) > 0) { $form_state['values']['taxonomy']['8'] = $voc_8; } // Channel/Issue Area (aot)
  if(count($voc_43) > 0) { $form_state['values']['taxonomy']['43'] = $voc_43; } // Content Display (aot)
  if(count($voc_11) > 0) { $form_state['values']['taxonomy']['11'] = $voc_11; } // Content Type (aot)
  if(count($voc_24) > 0) { $form_state['values']['taxonomy']['24'] = $voc_24; } // TM Program Affiliation for Content (aot)
  if($voc_4 != '') {$form_state['values']['taxonomy']['4'] = $voc_4; } // UrbanResource.net (tid as string)
  if(count($voc_5) > 0) { $form_state['values']['taxonomy']['5'] = $voc_5; } // Organization Affiliation (aot)
  
  // Settings needed to create the node
  $form_state['values']['is_new'] = TRUE; // New node
  $form_state['values']['status'] = 1; // Published
  $form_state['values']['op'] = t('Save'); // Save it!

  //kprint_r($form_state);
  drupal_execute('amazon_book_node_form', $form_state, (object)$node); //Programmatically submit this to the Amazon Book form
}
EvanDonovan’s picture

I forgot to include the taxonomy functions in my last post. You'll need to include those if you want the script to run (unless you comment out those lines). They are as follows:

  function taxonomy_node_list_terms_by_vocabulary($nid, $voc_id) {
  	$node = new stdClass();
	$node->vid = db_result(db_query("SELECT MAX(vid) FROM {node_revisions} WHERE nid = %d", $nid));
  	$terms = taxonomy_node_get_terms_by_vocabulary($node, $voc_id);
  	$term_list = "";
   	foreach($terms as $term) {$term_list .= $term->name . ", "; }
    $term_list = rtrim($term_list, ", ");
    return $term_list;
  } 
  
  function taxonomy_node_tid_array_by_vocabulary($nid, $voc_id) {
  	$node = new stdClass();
	$node->vid = db_result(db_query("SELECT MAX(vid) FROM {node_revisions} WHERE nid = %d", $nid));
  	$terms = taxonomy_node_get_terms_by_vocabulary($node, $voc_id);
  	$tid_array = array();
   	foreach($terms as $term) {$tid_array[$term->tid] = $term->tid; }
    return $tid_array;
  } 
  
   function taxonomy_node_get_tid_by_vocabulary($nid, $voc_id) {
  	$node = new stdClass();
	$node->vid = db_result(db_query("SELECT MAX(vid) FROM {node_revisions} WHERE nid = %d", $nid));
  	$terms = taxonomy_node_get_terms_by_vocabulary($node, $voc_id);
   	foreach($terms as $term) {$tid = $term->tid; }
    return $tid;
  } 
Rhino’s picture

Subscribing (and playing with that highly interesting upgrade path toy, thank you EvanDonovan !)

EvanDonovan’s picture

FileSize
5.76 KB

Somehow the text of this comment got lost. Anyway, attached is the updated version of the script, tested to work with 100 nodes at once. When added to a page in the PHP input format, it shows a link that you click to start the import.

EvanDonovan’s picture

FileSize
5.82 KB

Here's an updated version of the script. I added stripslashes() in, so that the editorialreview field won't have the annoying backslashes in it when it's turned into the node body.

Junro’s picture

subscribe :)

MGParisi’s picture

subscribe

asb’s picture

Hi,

the project page still reads: "Also in the works is an automated upgrade path for users of Drupal 5's Amazon Associate Tools module."

Is there any progress with this task? Since more and more modules become unsupported for D5, I'm getting quite nervous about not even having seen the outline of a migration route yet...

Is there anything we non-developers can do to help, e.g. testing development versions?

Thanks & greetings, asb

asb’s picture

Hi Evan,

thank you for your efforts to at least start working on an upgrade path!

I took some time on the weekend to try a sandbox upgrade since an "official" upgrade path seems... let's say "unsure" at the moment. I was hoping to get an idea what your script would to, not expecting a click-and-run-solution. That said, here's my approach:

After upgrading core, then cck, then date, then installing "amazon" module, the first error (related to the upgrade of "amazontools" data) I encountered when running update.php:

user warning: Table 'migration_drupal.amazon_item' doesn't exist query: ALTER TABLE amazon_item ADD mpn varchar(128) in /var/www/migration/update.php on line 69.
...
The following queries were executed
amazon module
Update #6001
Failed: ALTER TABLE {amazon_item} ADD mpn varchar(128)

(No idea what this means and if this has any effect on your script)

Then I created, as instructed, a node with the PHP input filter; saving this resulted in a node with a link "start Amazon node import: rows 0 to 99"; clicking on this, resulted in "continue Amazon node import: rows 101 to 200", then it went back to "start Amazon node import: rows 0 to 99". Nothing else happened accorting to ./tracker or ./recent_changes. According to ./admin/reports/systeminfo I now have

* amazon 2042 nodes
* amazon_node 6023 nodes

Beforde the D5->D6 upgrade (and before running the import script):

* Review (= amazon) 2042 nodes
* Product (= amazon node) 6023 nodes

None of these node type are selectable/filterable from ./admin/content/node. Old nodes from "Amazontools" have still no nodetype anymore, are not connected to Amazon's ECA REST interface or whatever (meaning: they don't show a cover image or pricing information), and they're not editable. Neither data from Amazon associate tools 5.x-1.6, nor from the ISBN/ASIN 5.x-1.4 CCK field (re-) appears. As far as I have seen, absolutely no content has been migrated or even touched.

If your time permits maybe you could write a few words about your script, targeted to those without coding skills: What does it do, what is the expected behaviour? What has to be modified to make it work in a different environment? What skills are required to use the script? Or asked the other way aroung: Will people without substancial coding skills have any chance to use this script?

Thanks & greetings, -asb

EvanDonovan’s picture

asb:

To be honest, I haven't run the update script in several months. It is a very hackish solution & relies on several assumptions that I could make only because I knew the configuration of my site. I posted it up here since I thought something is better than nothing. I am not sure whether it will help people without coding skills. I don't have a great deal of PHP knowledge myself, however, and I based the script off of several things I saw on Drupal.org & most specifically off http://www.lullabot.com/articles/quick_and_dirty_cck_imports. If you have any interest in importing or exporting data from a Drupal site, I highly recommend you read that article, as the details given in it and the comments are the foundation for my script.

If you are not interested in coding, I highly recommend you try the latest version of the Amazon module (the dev version) because it contains an Amazon Legacy module which is more sophisticated in some respects than my solution. It won't create nice new CCK nodes for you, so you may not be able to use things like the Amazon Store module, or even, I think, get updated data, but you won't have to run a script. Basically, it looks like the Amazon Legacy module just makes your old Amazon nodes appear to have an ASIN field in the new format, so they will show up again.

That said, on to the documentation of my script: I know it may seem unusual that it is a click & run script, but that's because something more sophisticated (like Batch API) is currently beyond my coding abilities. Note that it will not actually turn any of your existing nodes into nodes of the new Amazon module. Instead, it will create new nodes programmatically which have the same information represented in the new format.

Before testing the script, ensure that you are using the most up-to-date version of the script (the one I actually used) is in #14. I don't have time to be too detailed in documenting the script tonight, but here's a few key assumptions that it makes:

1) You have a CCK content type with the machine name of "amazon_book".
2) You have the Pathauto module enabled.
3) Your new CCK content type has an Amazon Item field called "field_amazon_item".
4) You have taxonomy terms on your books in particular vocabularies (vid's).

Assumptions 1 & 2 you can alter accordingly for your site by changing the line that reads
$node = array('type' => 'amazon_book', 'pathauto_perform_alias' => 1); . If you do change the name of the node type, make sure to change the name of the form (in the line starting with drupal_execute) to match.

The first part should be the machine name of your CCK content type (that is, the content type which you create to have an Amazon Item field in it). The second part should be removed if you aren't using Pathauto.

Assumption 3 you can alter accordingly for your site by changing the line that reads $form_state['values']['field_amazon_item'][0]['asin'] = $amazon_node->asin;

Assumption 4 you should work around by either removing all the Get & Set Taxonomy Terms code from the middle of the script, or else (if you feel confident about it), altering the second parameter of my taxonomy_node_tid_array_by_vocabulary custom function to the proper vocabulary id.

Beyond these four assumptions there may be other things that the script assumes which I currently have forgotten about. I know that the script did not run exactly the same in my production site as it had in testing. I believe we ended up having to run it as admin (user 1). You definitely have to run it with a user who has the privilege to create nodes of the new Amazon Book type & to both view & edit the Amazon Item field in them.

You can tell if the script has run correctly because the script will get filled with green messages saying "Book Name has been created". You will also be able to see the new nodes on admin/content/node.

Once the number of Amazon Book nodes equals the number of Amazon nodes you had under the old module, you can go ahead & delete the old Amazon nodes. But this may be difficult, especially if you have a lot of them, and it doesn't hurt anything to leave them in there.

asb’s picture

Hi Evan,

thank you very much for the explanations, I'll play around the these parameters even if I don't fully understand what the Taxonomy section does. At least when matching the CCK content type (e.g. "film" instead of "amazon_book") and the ASIN field (e.g. "field_asin" instead of "field_amazon_item") it still does not create new nodes for me.

However, I ran into another issue (I had experienced before with other migration attempts, when abandoned modules were involved): CCK does neither disply nor allow to edit these abandoned CCK fields:

This content type has inactive fields. Inactive fields are not included in lists of available fields until their modules are enabled.
ASIN (field_asin) is an inactive Amazon item field that uses a asin widget.

CCK requires a widget to get access to those data back; in other cases I tried to manipulate the database directly to force CCK into deisplaying those fields (based on suggestions from 500360; my attempt), which led - due to the unmanageable complexity of CKK - nowhere except to a bunch of follow-up errors. As far as I have understood your script, those ASIN fields (coming from the abandoned ASIN Field modules) couldn't be handled by your script anyway?

Thanks & greetings, -asb

EvanDonovan’s picture

I'm not sure what the field_asin is. You need to create a field that is of the type Amazon Item, which comes from the Amazon module. Otherwise, the script will not work.

I would highly recommend you try upgrading to the latest dev of Amazon module. I haven't tested it myself, but there is an Amazon Legacy module which should be much easier to use. In fact, I would suggest that everyone who is following this thread should test the Amazon Legacy module. If it works as advertised, then I think we can finally mark this thread fixed.

asb’s picture

> I would highly recommend you try upgrading to the latest dev of Amazon module. I haven't tested it myself,
> but there is an Amazon Legacy module which should be much easier to use.

That's what I'm hoping for many, many months.

Enabling Amazon legacy support from 6.x-1.x-dev (2009-Jun-25) results in:

Fatal error: Cannot use object of type stdClass as array in /var/www/migration/sites/all/modules/amazon/aat_legacy/aat_legacy.install on line 13

The module simply can not be anabled.

Greetings, -asb

eaton’s picture

aat_legacy module is definitely incomplete. (As can be seen from the crashing...) The first portion of upgrade code for the 'ASIN' CCK field type from Drupal 5 has been committed to the -dev branch. Once the CCK field has a clean upgrade path, I'll try to put together some utility functions to migrate an old-style amazon node to a new CCK-field type.

eaton’s picture

I'm not sure what the field_asin is. You need to create a field that is of the type Amazon Item, which comes from the Amazon module. Otherwise, the script will not work.

The script that was posted above appears to be very specific to one site's specific set of taxonomy terms and CCK field types.

EvanDonovan’s picture

EDIT: this comment is a bit verbose. Scroll to the next comment for the condensed version.

asb:

Ok, I guess that rules out the aat_legacy module, at least for now. I hadn't tested it myself, since I had already imported my site's Amazon nodes using the script I posted above. Sorry if I misled you; I thought from a quick scan of that module's code that it might work.

I quickly reviewed the code of the ASIN Field module & it looks like it is storing its information in the {amazonnode} table, just like the default Amazon Associate Tools node types (which my script was built for). So the script should be able to import data, as long you have made the proper changes to it, as I described in #19.

What I meant to say in #21 was that the content type that will be "receiving" the old Amazon nodes' content has to have a CCK field from the 6.x Amazon module in it. The name of this field should not matter if the script is modified as I stated above:
[change] the line that reads $form_state['values']['field_amazon_item'][0]['asin'] = $amazon_node->asin;

The modifications for the taxonomy terms are definitely trickier. You need to know what the vid's are for the vocabularies on your legacy Amazon nodes and make sure that the same vocabularies are available on the "receiving" 6.x Amazon content type. If you can't figure that part out, you may simply remove it from the script.

So here are the steps for running the script, best as I can tell:

1) Log in to your site as admin (uid 1) -- the script may not run otherwise.
2) Install & enable the 6.x Amazon module, ensuring that all its tables have been created correctly. (You mentioned an error relating to {amazon_item} in #18. Check to make sure that table exists.)
3) Configure the Amazon module.
a) Set up your Amazon Web Services key & test to make sure it is properly pulling data from Amazon.
b) Create a new content type to "receive" the data from your legacy Amazon nodes. To match my script, give this content type the machine name of "amazon_book". Otherwise, you will need to alter my script.
c) Create a new field of type Amazon Item. To match my script, give this field the machine name of "field_amazon_item". Otherwise, you will need to alter my script.
d) (OPTIONAL) Add the same vocabularies to your new content type as were on your legacy Amazon nodes.
4) Perform the Amazon node import.
a) Copy my script from #14 into a node & select the PHP input format. (Enable the PHP module & that input format if it is not available.)
b) (IF NEEDED) Modify the script as per #19 (i.e., changing the name of the "receiving" content type & "receiving" field, removing pathauto variable if necessary, altering or removing taxonomy import settings)
c) Click to run the script, repeating until all your Amazon items have been imported into the "receiving" content type. You will know when the script runs correctly because it will say Book name created, where "Book name" is the name of the Amazon item.

EvanDonovan’s picture

Status: Needs review » Needs work

Let me try to be more succinct :)

I'll play around the these parameters even if I don't fully understand what the Taxonomy section does.

Then remove the Taxonomy section, at least for a test.

At least when matching the CCK content type (e.g. "film" instead of "amazon_book") and the ASIN field (e.g. "field_asin" instead of "field_amazon_item") it still does not create new nodes for me.

I think you are matching based on the old CCK content type & CCK field. You need to create a new CCK content type & field in order to store the Amazon data in the new format. You can call the content type whatever you want, but the script won't have to be modified if you call it "amazon_book".

As part of this new CCK content type, you need to create a new CCK field of the type "Amazon item". Again, you can call it what you like, but the script won't have to be modified if you call it "field_amazon_item". (Make sure content permissions are set so that users in the role which will run the import script have the ability to view & edit this field.)

Sorry if my explanations have been confusing. Hopefully, this is a bit better.

eaton’s picture

I quickly reviewed the code of the ASIN Field module & it looks like it is storing its information in the {amazonnode} table, just like the default Amazon Associate Tools node types (which my script was built for). So the script should be able to import data, as long you have made the proper changes to it, as I described in #19.

A quick note -- that only applies to the Drupal 5 version of the module. The D6 version of ASIN FIeld stores things in the 'proper' CCK way. Thus the in-progress import path for it.

EvanDonovan’s picture

eaton: Thanks for the clarification. I think what I said in #26 applies in this case, though, since I believe asb is using the 5.x version of the module.

eaton’s picture

I've just checked in an update to ASIN module that allows people using the ASIN field under Drupal 5 to migrate their CCK field data forward to D6 properly. The next step is a migration path for Amazon Associates Tools style 'amazonnode' content, but this is definitely a step in the right direction.

EvanDonovan’s picture

Well, that sounds like it would fix asb's issue anyway. Great!

asb’s picture

Hi Eaton,

thank you for working on an upgrade path!

Trying to enable "Amazon legacy support" after upgrading to Amazon 6.x-1.0-beta7: Still no luck, damn! ;)

Fatal error: Cannot use object of type stdClass as array in /var/www/migration/sites/all/modules/amazon/aat_legacy/aat_legacy.install on line 13

Any prerequisits I might have missed? Dancing three circles around the kitchen sink?

Greetings, -asb

eaton’s picture

Alas, AAT_Legacy is still incomplete -- the upgrade path referred to in the above post is specifically for people who were using Amazon Associates Tools with the 'ASIN' module on Drupal 5 to add CCK fields that held Amazon products. the AAT Legacy module is/will be for those who were using the native 'amazonnode' content types...

asb’s picture

Version: 6.x-1.0-beta5 » 6.x-1.0-beta7

Hi,

thanks; I'm using both (native 'amazonnode' content types and 'ASIN' module) together, as probably most legacy users will do (the 'ASIN' field requires 'Amazontools', and, if I remember correctly, 'Amazontools' automatically creates the two native node types, so basically everybody using 'ASIN' field will also have those node types). So currently the migration path handles very special case where someone might have used 'Amazontools' exclusively to enable the 'ASIN' field and build own CCK content types on top of that, but has never touched 'amazon_node' and 'amazon' nodes. Such an setup I've never used, so there's currently nothing where I could help testing?

Thanks & greetings, -asb

asb’s picture

Related issuees:

* aat_legacy
* aat_legacy fixes

Greetings, -asb

EvanDonovan’s picture

Relinking issues using the [#issue number] syntax, so that they will show up w/changing status color as they are worked on:

#541998: aat_legacy sub-module can't be enabled
#555624: aat_legacy fixes

asb’s picture

Thanks, Evan, for adding the proper linking to the related issues.

Is there any progress on this issue? (still stuck with D5...)

Thanks & greetings, -asb

EvanDonovan’s picture

@asb:

Are you at all comfortable with using MySQL directly?

The DB admin at my organization was able to transfer 100's of nodes from a non-CCK type (Weblink) to a CCK type (Weblink CCK - which is just a node with a Link field) by first migrating the 'extra' node data from the non-CCK table ({links} in this case) to the table that was the CCK field ({content_field_weblink}), and then by changing the node.type field from "weblink" to "weblink_cck".

That migration was actually a lot easier and more effective than the export/re-import which my code in this issue does. So I would recommend that you pursue a similar route if possible.

Let me know if you need more specifics. I don't have a lot of time to devote, but I might be able to put a brief list of steps involved.

Actually, I do have one additional suggestion, which might be easier, though more tedious. You could consider using the Node Convert module. It would at least turn the nodes into the correct type, although it might not preserve the field data. It's worth a test, anyway.

rfay’s picture

Eaton and I were talking tonight about this issue. There's still regret that it never got done.

However, we need to ask: Is this still worth doing at this late point? How many people are still waiting for an AAT upgrade? Or should we just move on?

Drupal 7 beckons to us, with a major upgrade project. It's possible that we should just say "sorry" and move on at this point. So I'm interested in what people have to say.

rfay’s picture

Marked #541998: aat_legacy sub-module can't be enabled as a duplicate of this issue.

asb’s picture

> Is this still worth doing at this late point? How many people are still waiting for an AAT upgrade?

If you really are going to take this road - abandoning legacy users and their data - why even bother with problems like "Drupal 7 beckons to us, with a major upgrade project? Why not "just say "sorry" and move on at this point" once more, meaning: drop "Amazon" module for D6, get rid of the tedious D6-D7 migration and start with an "Amazon2" module for D7 from scratch? That might sound a bit cynic, but if you're willing to lose trust once, wouldn't it be consequent not to bother with a "major upgrade project" (migration to D7) at all?

However, I definitely do need a robust migration path from "Amazontools" for some 50k nodes scattered through different D5 sites. I defintely can't "migrate" this data by hand, and I definitely can't do this without help - CCK is far too complex to be manipulated without coding skills or proper tools.

According to recent stats, there are still a few hundred users of the "Amazontools" module. I doubt that the majority of those users is willing to drop their sites when support for D5 ends... And I guess, that the majority of those users are merely users of Drupal as a Content Management System, not developers using it as a Content Management Framework. So I think most of these installations will die without a migration path, or at least lose a lot of data when they're trying to upgrade.

If you seriously are thinking about dropping the promised migration path and If you seriously want feedback from those stuck with D5 because of the "Amazontools" module, maybe I'd make sense to ask this question over there. MGParisi, are you reading this? Should this be discussed at the "Amazontools" project page that still reads: "For Drupal 6 and above, see Amazon module, which will come bundled with an aat_legacy.module."

Thanks for taking over co-maintainership & greetings, -asb

rfay’s picture

eaton’s picture

If you really are going to take this road - abandoning legacy users and their data - why even bother with problems like "Drupal 7 beckons to us, with a major upgrade project? Why not "just say "sorry" and move on at this point" once more, meaning: drop "Amazon" module for D6, get rid of the tedious D6-D7 migration and start with an "Amazon2" module for D7 from scratch?

Amazon Associate Tools was originally written for Drupal 4.6, a little over 4 years ago. It worked by creating two custom Node Types: "Amazon" and "Amazon Node." Because I needed its functionality for a site I was building, I did most of the heavy lifting upgrading that module for Drupal 4.7, and 5.0. However, I was frustrated with the limitations of its hard-coded content types and wrote another module, 'ASIN Field', which piggybacked on AAT to allow the use of CCK for storing Amazon product information instead of the hard-coded node types.

This module -- Amazon -- was written from scratch for Drupal 6 with the intent of being a sound basis for future development of Amazon API related projects. It uses a different (read, more reliable and more flexible) data storage mechanism and takes advantage of new Drupal tools like Views where the old AAT module was not able to leverage them very well. This module -- 'Amazon' module -- also provides a CCK field to store product information, and it has a tested, working upgrade path from the D5 'ASIN Field' module to its own Drupal 6 CCK field.

What it does not do at present is automatically create its own hard-coded node types to mirror the AAT module's "Amazon" and "Amazon Node" content types. As such, AAT users who used those specific content types in D4.x and 5.x do not have a clean upgrade path. Or at least, they don't have a clean upgrade path without writing some of their own SQL queries.

There is no need to drop Amazon for D6 -- it works, and the Drupal 7 port of Amazon module will involve new code rather than entirely new data storage mechanisms.

asb’s picture

> This module -- 'Amazon' module -- also provides a CCK field to store product information, and it has a tested,
> working upgrade path from the D5 'ASIN Field' module to its own Drupal 6 CCK field.

Should the ASIN field appear on migrated D6 sites indeed? That would be very cool...!

However, on my migration sandbox I'm getting for content types with an ASIN field the following messages (first line is "red", second line is "green"):

This content type has inactive fields. Inactive fields are not included in lists of available fields until their modules are enabled.
ASIN (field_asin) is an inactive Amazon item field that uses a asin widget.

Of course no content from the D5 ASIN field appears (that's what the warning/error suggests, doesn't it?). I'm running "Amazon" module 6.x-1.0-beta7, enabled are all availale sub-modules (Amazon API, Amazon field, Amazon Filter, Amazon legacy support, Amazon media, and Amazon search).

Those "inactive fields" are a neverending nightmare I encountered several times when trying to update sites based on CCK from D5 to D6. My example approach to "fix" this: #501400: Drupal Core/CCK upgrades seem to be not handled properly by Nodereferrer create. To others reading this and having similar issues with "inactive fields" on updated sites: KarenS explicitly warns to "never never never manually alter the database values! Unless you are an expert on the internals of CCK, which I assume you are not, you will probably break everything" (e.g. in #335597: All CCK custom fields are inactive after upgrading or transfering; that's why I try not to touch CCK database tables on production sites). I think, KarenS is right with this, but what options do we have on the other hand?

KarenS suggests the following general procedure to get access to those fields back: "If you have fields that are inactive disable and then re-enable the modules that create those fields.". I already tried this several times on my migration sandbox (disabled, "Amazon" module, ran update.php, re-enabled "Amazon" module, ran update.php again), but the "inactive fields" warnings/errors are still there.

Would it make sense to try something like this (from #500360: This content type has inactive fields...):

UPDATE `content_node_field_instance` SET `widget_type` = 'optionwidgets_select',
`widget_module` = 'optionwidgets' WHERE `field_name` = 'field_[YOUR-FIELD-HERE]';
TRUNCATE `cache`;

Thanks & greetings, -asb

rfay’s picture

@asb, if you get to a stable place and want to, you can make the database available to me and I'll try to understand a way out. You can email me a link to a dump, or just email the db if it's small. Probably the best case situation is for you to try to boil this down to a smaller test case, but I'm willing to take a look at your database if you make it available.

eaton’s picture

asb, have you visited update.php on that site with the ASIN Field module itself enabled? Amazon module dosn't handle the update path for it -- it's specific to the ASIN field.

asb’s picture

Hi,

main and most important question: Have others been following this "tested, working upgrade path" from the D5 'ASIN Field' module to the D6 'Amazon' CCK field? What were their experiences? What was their approach? (Jeff accomplishes lots of things 'normal' people can't accomplish ;)

Until now my understanding was that there's simply absolutely no working path between AAT an Amazon module (left aside home-brewed custom solutions like those Evan succeeded with). If 'ASIN Fields' already should work, we'd IMHO have a completely different situation (meaning: something I could at least try to work with).

@eaton (#45): Yes, several times in different sequences (re-enabled every sub-module one by one or all together).

@rfay (#44): My migration sandbox is a copy of the smallest site of the six installations I currently need to update/migrate (database dump less than 200 MB), and it's one of the newest (about 3 years online), so I thought it should be the easiest (not much more than ~10k nodes). Is this small enough, or how should I "boil this down to a smaller test case" (simply delete a few thousand random nodes? or delete mostly Amazon nodes?) What would be a small enough test case?

Thanks & greetings, -asb

PS: Lots of thanks to you both that things seem to start moving again!

Edit: At least that one didn't work:

mysql> UPDATE `content_node_field_instance` SET `widget_type` = 'text_field', `widget_module` = 'text' WHERE `field_name` = 'field_asin';

Now I'm getting: This content type has inactive fields. Inactive fields are not included in lists of available fields until their modules are enabled. ASIN (field_asin) is an inactive Amazon item field that uses a text_field widget.. I think I'll try to disable and re-enable CCK completely as a next approach...

rfay’s picture

@asb, give me a shot at it. I can't guarantee to get to it right away, but it's a starting point.

Thanks,
-Randy

asb’s picture

Hi,

still nobody that succeeded with the existing upgrade path from the D5 'ASIN Field' module to the D6 'Amazon' CCK field?

I want success stories ;)

Greetings, -asb

rfay’s picture

@asb, I haven't received a demo database from you... Any size you want will be fine. Just put it somewhere I can download it and email me a link, or if you get it small enough, send me it as an attachment to randy at randyfay.com

rfay’s picture

OK, to summarize what I think is going on here:

There are three problems here:

1. CCK update doesn't work between the D5 Amazontools module and D6 Amazon. @asb has come up with a problem doing the update with just CCK ("This content type has inactive fields"). This ought to work out of the box, but didn't work for him.

2. The odd "amazonnode" type has no migration path. It has to first be converted into a CCK type, per something like http://drupal.org/node/595596#comment-2114614.

3. The aat_legacy data type probably led us down a useless path. I don't think we probably want to even think about this.

I'm going to try take a look at the migration of Amazon CCK nodes, since that's the crux of this issue. I think #2 should remain a well-documented hack as provided in the link. And there should not be an aat_legacy module or type.

-Randy

asb’s picture

Version: 6.x-1.0-beta7 » 6.x-1.0-rc2

Thanks again, Randy, for looking into this issue.

For those interested in this issue, I'll try to summarize our chat.

The current status is, that users of the Amazontools (AAT) module can't easily upgrade their sites past Drupal 5, if they want to keep their 'Amazontools' nodes and associated content. There are currently three approaches to migrate AAT data from D5 to the 'Amazon' module on D6 (please correct me if I missed something or got something wrong):

  1. The 'legacy' sub-module, included with the 'Amazon' module for D6, attempts to move data from the proprietary AAT content types (not sure if only 'amazon' or both 'amazon' and 'amazon_node') into a new 'legacy' content type; the Amazon 'legacy' sub-module is currently not working yet, and there is no activity (e.g. with contributed patches) to get it going. As Randy stated above, this approach basically has no future (unless someone jumps in). So the first part of the migration path is de facto dead.
  2. AAT data stored in D5 on content types with an attached ASIN/ISBN CCK field should be migrated automagically during the upgrade of D5 to D6 respectively when installing the 'Amazon' module. However, I wasn't able to reproduce this and couldn't make any CCK ASIN fields appear on a migrated site ("This content type has inactive fields"). That's the second part of the migration path, and that might be the way that might get some dev love.
  3. Besides this two-part "official" migration path, there have been several attemtps to roll custom solutions. At least one of these approaches did work for Evan Donovan; however, Evan's approach from #10 pp. is focused on a certain setup and couldn't be made "generic" without a major effort. Especially it assumes, if I understand it correctly, that all AAT data from D5 can be moved into one content type (amazon_book) in D6, so it's basically the 'legacy' content type approach.

When dissecting the data that would have to be migrated (I'll post our findings in the next comment), we found at least three more or less different scenarios; Randy suggested to distinguish the migration issue into distinct and separate migration "sub-projects".

Last but not least: Work on these migration "sub-projects" will require at least some amount of sponsoring, so it would really help, if those interested in upgrading their D5 sites to D6 could throw in a few bucks.

So far the current status.

asb’s picture

Now to dissecting the data to be imported.

IMHO - at least on my D5 sites - there are three different scenarios how and where the AAT data is stored:

  1. There are two defined content types provided by the 'Amazontools' module, named 'amazon' and 'amazon_node'; those are carrying exactly one (required) ASIN which can't be edited after the node has been created. These two node types and their "integrated" ASIN field haave nothing to do with CCK. Those two AAT content types don't carry any sematic "knowledge" about what they are (be it books, DVDs, audio CDs or whatever Amazon is selling) opposed to custom CCK-based content types that might distinguish between a "Book" and a "CD" and a "Camera" content type with associated Amazon information.
  2. The AAT module allows to add related Amazon product links to any other content type (like "Story", "Page", "Poll", etc.); this feature is simply enabled at ./admin/content/types/{my-content-type} by checking "Allow Amazon.com related product link" and has nothing to do with CCK fields, also. Opposed to the two content types provided by the Amazontools module, these related product links are optional and accept multiple values (= multiple ASINs).
  3. The third scenario are content types with an added 'ISBN/ASIN' CCK field (provided by Jeff's ASIN module) that might allow single or multiple values (= ASINs). Probably these data can be most easily migrated into a D6 site since it's data structure is very close or even identical to the 'Amazon' CCK field provided by the 'Amazon' module on D6

I think that there might be cases where a CCK-based content type has as well an CCK ASIN field as is enriched with "Amazon.com related product links"); however, I'm nor sure if I have such content types - if someone needs a solution for this, now would be the right time to make noise.

All these three scenarios utilize and depend on the 'Amazontools' module's backend to store and manage (= refresh) the data. I don't know enough about the storage mechanism of AAT, so I can't say if it's hard or easy to work with this data.

However, in any attempt to migrate the AAT data a decision has to be made, how the existing data is to be mapped in D6; the decision is,

* if data from 'amazon' and 'amazon_node' content types in D5 are to be moved into one content type in D6; that's basically the approach of the 'legacy' sub-module; or
* if data needs to be differntiated into different content types in D6. If so, the question is (a) which properties allow such an distinction and (b) how the user/site administrator is supposed to handle this.

A "catch all" 'legacy' content type eats all data from 'amazon' (and 'amazon_node' ?) nodes, without looking at them; if we do not follow the 'legacy' approach, the 'amazon' nodes would have to be distinguished into potentially dozends of "semantic" content types (e.g. "film", "book", "camera",
"lens", "accessory" etc.). Doing this the hardcore way (issuing SQL commands at the MySQL prompt and figuring out each term ID manually to distinguish 'amazon' "Book" nodes from 'amazon' "Camera" nodes etc. sounds extremely error-prone and potentially take days per site.

An approach suggested by Jeff in a previous comment was - as I understand it - to put
the ASINs from the 'amazon' content types in a new ASIN field in the 'amazon' nodes with this SQL statement:

UPDATE amazonnode SET ntype = 'field_product' WHERE ntype = 'amazon'

The rest of the procedure I never fully understood (how are 'amazon' nodes "moved" into D6 where thay have no "handler"? What happens with the 'amazon_node' nodes? How are node types with related Amazon product link enabled handled?). At least theoretically all three scenarios could be handled this way: Put an multi value CCK ASIN field on any node type and copy everything stored by AAT into this field. However, somehow the "updated" 'amazon' nodes from AAT would still have to be "converted" somehow so that D6 "sees" them (in my migration attempts, 'amazon' and 'amazon_node' nodes are not visible anymore).

So far with dissecting the data I have to migrate; I might have overssen other modules that also sit on the back of AAT.

asb’s picture

Finally to the last and probably most interesting part: How can these scenarios be approached for a data migration?

As far as I see it, there are basically five options (please correct me if I have missed something):

(a) Resolve remaining issues and finish Amazon 'legacy' sub-module

Pros:
* handles 'amazon' and 'amazon_node' (?) nodes in D6 by moving the data into a 'legacy' content type;
* transparent run-once solution;
* very little or no user interaction requires, should run more or less "automatically";
* easy and straight-forward for the site administrator/user;
* at least some code to start with already exists;
* keeps everything alive, from stored data (ASINs) to URLs and backlinks inside the site pointing to 'amazon_node' nodes etc.

Cons:
* doesn't handle other scenarios: data from node types with related Amazon product link enabled would have to move the 'Amazontools' data into a multi value 'Amazon' CCK field; content types with attached CCK ASIN filed would also have to be handled differently;
* does not fully "upgrade" the AAT data into the CCK and fields-in-core world; especially, by design, wouldn't semantically enhance the mixture collected in 'Amazon' AAT nodes into distinct "Book", "CD" or "Camera" content types.

This approach sounds to me still like the simplest and most straight-foward way to go. In a later and separate follow-up project one could think about ways to make the 'legacy' nodes fully CCK-aware and sort them into matching CCK-based content types. However, this is the maintainer's call.

(b) Drop the 'legacy' approach and develop an utility to migrate 'Amazontools' content into semantically modelled content types

Ideally, site admins/users would get a plug-in (or more precise: one ore more actions) for Views Bulk Operations (VBO) that
* allows to select content according to content type and other criteria (like taxonomy terms), and
* to change the node types from 'amazon' into other content types (might utilize code fragments from the 'nodetype' module)

The VBO action might be a simple wrapper for shell scripts or anything else that can be packaged into such an action.

Pros:
* High degree of flexibility;
* Optional way to fully migrate AAT data into the world of distinct and customized CCK-based content types;
* great "semantical" enhancement of AAT nodes;
* could be a first step into VBO integration which would be an immensely useful addition to the 'Amazon' module;
* not much graphical interface development required, most of it provided by views.

Neutral:
* The VBO plug-in would have to manage database modifications (change node type, update relevant tables, move ASINs accordingly, etc.) in the background, but the user would have to select and apply the matching bulk operation to a set of nodes that he/she would have to choose;
* unclear if this would have to run on D5 or D6.

Cons:
* Probably a major coding effort, even if VBO would carry most of the heavy lifting;
* Pobably would require even more sponsoring, be it with code or cash contributions.

I think this would be my personal favourite, especially considering the possibilities of VBO integration for the 'Amazon' module. However, this is definitely no short route with pragmatic goals but some kind of "optimal" solution (which inherently often are never finished).

(c) Don't develop any (graphical) interface, but provide one or a set of shell/SQL scripts that either populate a 'legacy' content type or try to be a at least bit smarter

This would include - for example - to try to make Evan's script "generic" so it could be used on other sites without rewriting all of the code; this might also include nothing but a set of SQL statements for the different scenarios mentioned above.

Pros:
* Might be well suited to migrate several thousand nodes since it possibly wouldn't have PHP timeouts (similar to VBO actions that utilize the Batch API)

Cons:
* I'm not sure if this would really simplify anything, except if the script(s) would be strictly limited to simplest operations (e.g. put ASIN from field_a into field_b)
* I highly doubt if a script can - with a moderate effort - be made generic enough, so that it could be as well be used by site administrators (opposed to software developers and SQL wizards) as would be up to complex data migrations into different content types based on critera like taxonomy terms (this would be required, except if going for the 'legacy' approach and put everything from D5 content type 'Amazon' into D6 content type 'legacy' (or 'amazon_d6, or whatever it might be called).

(d) Come up with a totally diffent idea

(Insert your innovative ideas here ;-)

(e) Close the migration issue once and for all, and don't offer any migration paths btween 'Amazontools' and 'Amazon' modules.

* remove the 'legacy' content type and associated code from 'Amazon' tools module;
* change the announcement about an upcoming migration path on the 'Amazontools' project page and in the 'Amazon' module issue queue.

The remaining 300 users of AAT would loose thousands of nodes when D5 reaches end of life or have to live with an unsupported Drupal version (including taking the risk of security issues and changing Amazon APIs).

Of course the latter is my least favoured "solution".

So far with the overview about possible approaches to data migration.

clara.raubertas’s picture

FileSize
1.08 KB

Here is a SQL script that worked for me (upload to the root of your drupal install & run in your browser) -- a little more generic than Evan's above though still quick'n'dirty (just titles/images/links are imported, but it shouldn't be too difficult to tailor it to get other available fields too). Obviously, no guarantees, and back up your database first, but hopefully this will be useful to someone.

rfay’s picture

The aat_legacy module now provides an upgrade path for D5 items called "amazonnode" or amazon_node.

http://drupal.org/node/541998#comment-2418786

This is the one thing I said I wasn't going to work on :-)

rfay’s picture

@clara.raubertas, #54, Thanks for sharing your script with us. This script only builds the amazon_item table... It doesn't build any of the nodes that use it (CCK nodes, amazon nodes, or amazon_node nodes). And the amazon_item table can be rebuilt from scratch any time by cron just using ASINs.

rfay’s picture

Version: 6.x-1.x-dev » 6.x-1.0-rc2
Status: Needs work » Active
FileSize
3.2 KB

OK, the next step is now in place. This patch (committed to HEAD) fixes the upgrade path for the D5 ASIN module. There were a number of problems with the upgrade, and it seems to work well now. As far as I can tell, all ASIN (CCK fields with Amazon ASINS) are working in the upgrade.

Committed to HEAD: http://drupal.org/cvs?commit=308560

Now we have two of our work items moved along. The legacy module now handles the former amazonnode, and ASIN nodes survive the upgrade. Maybe we'll get somewhere with this after all.

Here are the upgrade instructions. Major version upgrades in Drupal can be painful, so ALWAYS back up your database, and back it up at good points so you can restart the migration process without doing the whole thing over again.

1. Backup your database. And your files. Switch to a core theme like garland.
2. Disable all CCK modules, including Content module, ASIN, and anything dependent on it.
3. Connect your database to a complete D6 filebase, including Amazon and CCK.
4. Remove all modules from sites/all/modules.
5. Run update.php to update core.
6. Move the CCK module into sites/all/modules
7. Enable Content module.
8. Run update.php
[edit: 8.5] Create the amazon and amazon_node content types. See #62.
9. Move Amazon module into sites/all/modules.
10 Enable Amazon module (NOT ASIN yet)
11. Run update.php. there are some ugly error messages from this update.php. I haven't looked into them. I think they're historical.
12. Enable asin module
13. Run update.php. This may take a long time as it processes your nodes.
14. Move all other modules back into sites/all/modules and enable them as needed
15. update.php
16. Visit your Amazon settings page at admin/settings/amazon. You likely will have to at least set the locale (if it's not 'US').

It might work for you. I look forward to your report. This will be in the dev version within 12 hours.

asb’s picture

Status: Needs work » Active

Hi Randy,

thank you very much for these incredibly fast fixes!

I immediately did a quick test with yesterday's dev version in my sandbox:

  • disabled amazon module
  • disabled cck and everything depending on it (of course)
  • ran update.php
  • enabled content.module, ran updates 6000 to 6010 for content.module (everything that is offered in the pulldown menu)
  • enabled amazon api, ran updates 6001 to 6009 for amazon.module
  • dito with the other amazon sub modules except the asin field, again ran updates 6001 to 6009
  • dito with asin field, again ran updates 6001 to 6009
  • enabled the remaining cck modulesand ran updates 6000 to 6010 again

There were a few error messages during update.php, but not many and nothing really scary:

user warning: Duplicate column name 'mpn' query: ALTER TABLE amazon_item ADD mpn varchar(128) in /var/www/migration/update.php on line 69.
user warning: Duplicate column name 'invalid_asin' query: ALTER TABLE amazon_item ADD `invalid_asin` INT DEFAULT 0 in /var/www/migration/includes/database.mysql-common.inc on line 298.

I'm now getting two new content types ("Amazon data" and "Amazon review"); those appear at ./node/add but not at ./admin/content/types; however, I can edit/configure them at ./admin/content/node-type/amazon-node respectively ./admin/content/node-type/amazon. Other observations:

  • 'amazon' nodes seem to have lost their node bodies completely (haven't looked yet in the database), but they display a small cover (75x51, without a link) and a hyperlinked title, but no other data retrieved from Amazon (like pricing or binding).
  • 'amazon data' nodes look pretty identical.
  • Other content types that had an D5 ISBN/ASIN field attached still have no Amazon data at all, and the content type still claims to have "inactive fields" (ASIN (field_asin) is an inactive Amazon item field that uses a text_field widget.)

As it seems, the sandbox isn't suited well as a testbed; the upgrade precedure seems to offer one shot only.

I'll setup another D5 migration as soon as possible and start from scratch.

Thanks again, and a happy new year!

Greetings, -asb

rfay’s picture

Status: Active » Fixed
FileSize
10.43 KB

OK, this should be the final installment.

In this patch I provide a complete import function that brings in amazonnode data (the related links, and the "amazon" nodes.) It turns them into CCK fields, so there will be no need to support them in the future. Each node which had attached amazon data now has a field_legacy_asin field attached which has the same Amazon data.

The process: is now:

1. Follow the upgrade instructions in #57. That will upgrade your CCK fields from ASIN module. Don't worry about the warnings about the MPN field that asb mentioned. [edit: The amazon and amazon_node content types need to be created between updating the content module and enabling the ASIN module. See #57 and #62.]

2. Enable the Amazon Legacy Importer module

3. Visit Admin->Settings->Amazon Legacy Importer (admin/settings/aat_legacy) to convert the old amazonnodes data.

Your nodes will not look exactly the same, but you should have all the data. You can now change the display of various nodes by configuring them on the "display fields" tab of the content type, or by theming, or by any technique you choose.

Committed to HEAD: http://drupal.org/cvs?commit=308766. It will be in the dev version within 12 hours.

Please report your results.

asb’s picture

Version: 6.x-1.0-rc2 » 6.x-1.x-dev
Status: Active » Fixed

Hi,

indeed, the dev release 6.x-1.x-dev from 2010-Jan-01 is a huge leap; I spent the better part of yesterday to play with several (sandbox) upgrades and the new migration path, which - for the first time - started to work for me. That's a heartly Hooray!

My approachI started with a cloned D5 database and followed the instructions from #57 as far as possible (I don't know exactly what "Connect your database to a complete D6 filebase, including Amazon and CCK" means; I think, neither Amaozon- nor CCK modules should exist in ./sites/all modules then core is updated), but the rest is quite clear: First let Drupal update it's core modules, then upgrade content.module, then the rest of the CCK core modules, then Amazon API, then the rest of the 'Amazon' sub modules except ASIN field, then ASIN field, then run the Legacy importer as instructed in #59. Depending on the site's size this might take some time, in my case it's about four hrs. per migration attempt. Don't try to check if Amazon API is working (e.g. look up an ASIN at ./admin/settings/amazon/test) - yeah, I'm getting a huge array of data for the ASINs used on my D5 site. So the Amazon API works well on an upgraded site, nice!

By following this order (which btw matches the official instructions to upgrade CCK, but are opposed to the recommendations from Viktor Kanes book "Leveraging Drupal") I ran into a bunch of other issues (e.g. image.module seems not to get updated properly, search index is lost completely, etc.), but that will have to be sorted out separately. However, those other errors irritate when browsing through the watchdog logs and, especially the missing images, make the site appear very incomplete. but I'll try to ignore this and focus on the 'Amazon' migration path.

During the CCK/Amazon upgrades, several errors appear, but the overall process finishes. There are no problems anymore to even enable the 'legacy' sub module, and some ASIN field appear on customized content types; that's a "first time", also. All three scenarios i mentioned in "dissecting the data to be imported" (#52) are basically covered now. I did no systematical tests yet, but the migration process appears to be much more robust (opposed to the previous status, as outlined in #51, where upgrade attempts led to random failures).

The approach to convert the "related links" and and the "amazon" nodes from AAT into CCK fields is imho a very good design decision; I think it will allow us to "postprocess" these legacy nodes at a later time with tools like Node Convert into semantic content types; those who don't need this might be even happy with the legacy node types as they are after the upgrade.

So, in short, there is really a lot of progress.

However, when there's much light, there's also some shadow; my observations from yesterday in unsystematic order:

  • Formatting/display of the Amazon data is different from AAT, indeed; I haven't managed to get an ASIN field to display anything but a small cover image and the product's title (no pricing information, no binding, no release date, etc.; the ASIN fields are configured to display "Small image and full info"). So it might be required to refresh/reload this data from Amazon (need to figure out how to do this, seems not to be cron based).
  • At ./admin/content/node I'm getting no legacy content types ("Amazon data" and "Amazon review"), so I can't easily filter out the legacy data for review. The contributed module System information does not know them as well (they're not listed under "Active content types"); it still claims, on a fully migrated D6 site, to have, for example: "amazon: 2047 nodes; amazon_node: 6023 nodes"), both listed under "Deleted content types". Both are content types from AAT/D5 (need to look into the database directly to see what's stored there). When I access a former 'amzon' node on the migrated site, e.g. ./node/13953, I'm getting an empty node with an invalid/empty node type (according to About This Node) that can't be edited (it has an "Edit" tab, but after clicking on it I'm not getting an edit screen). It's similar with former 'amazon_node' content types, e.g. ./node/13925: They display a title, but have no valid node type and no ASIN field attached that could be displayed or edited.
  • When running the Amazon Legacy Importer (from #59), there were a total of about 500 nodes imported. That is a major mismatch; on the D5 site, I have, again according to Systeminfo: 6023 nodes of type 'amazon_node' (on my site called "Produkt"), and 2047 nodes of type 'amazon' (on my site called "Produktrezension"). This matches exactly the figures given by Systeminfo on the migrated D6 site.
  • I think there's a similar mismatch in the converted ASIN fields; during the upgrade/conversion there are figures printed on the screen (but I havn't found them in the watchdog logs), and I have the feeling that these numbers are much too small; however, I have to figure out a way to calculate the incoming/outgoing data on both sides somehow from the database tables to substanciate this "gut feeling". On D5, AAT gives me a figure of "Total registered items: 9024" on it's "Manage" screen (./admin/content/amazon_import/manage)
  • Ratings from the AAT nodes are not migrated. However, that'd be a different upgrade project (and probably involve something like Voting API) and was never discussed, so this is just a hint for other upgraders - the upgrade path discussed here is only about ASIN data attached to proprietary AAT and CCK based nodes, it's not supposed to migrate everything from AAT.

Since this long-standig issue was basically a feature request for an "Upgrade path for Amazon associate tools module" which exists now, I don't want to re-open this issue; I'd suggest to open separate issues for the remaining issues, as far as they are reproducable and can be substanciated. Would this be an acceptable way to continue?

Also I'll cross post this on the relevant 'Amazontools' and 'ISBN/ASIN' issue queues which are (I think there were more, but I can't find them at the moment:

Many thanks again to Randy for his great work and allowing us to close this long-standing issue!

Greetings, -asb

rfay’s picture

@asb: I think I understand what happened to the amazon and amazonnode nodes.... There's no module that defines them since I ripped the guts out of aat_legacy. It shouldn't be any big deal. I'll take a look at it tomorrow.

As far as the display on the ASIN CCK item, that's controlled by
1. Your selection in the "display fields" section of the cck node type configuration
2. Theming. Each of the three defined types is themable with .tpl.php files.

Thanks,
-Randy

rfay’s picture

Status: Fixed » Needs work
FileSize
961 bytes
936 bytes

@asb, I'd like you to try a path and see if it gives you the results you expect. I can probably add this to the code, but I think it's the missing piece.

*After* upgrading the content module and *before* enabling and upgrading the ASIN module, create the amazon and amazon_node content types using the normal CCK process, or using content_copy (I've attached the content types I used).

Other than that, use the entire process as described before.

I'm in the middle of a migration of legacy nodes that says it's going to do 8355 nodes (using your db).

Attached are the node types I created before enabling/updating the ASIN module.

asb’s picture

Version: 6.x-1.0-rc2 » 6.x-1.x-dev
Status: Fixed » Needs work

Hi,

I imported those two content types on a migrated site, re-ran updates 6001 to 6009 from 'amazon' module and am currently re-running "Amazontools Legacy Import"; before having these content types, the importer hadn't any open jobs to finish, now it's processing 8128 nodes.

It currently keeps failing because of one of these CCK nightmare modules (Relevant content):

An error occurred. /batch?id=22&op=do <br /> <b>Fatal error</b>: Call to undefined function relevant_content_cck_content_is_empty() in <b>/var/www/migration/sites/all/modules/cck/content.module</b> on line <b>906</b><br />

I'll have to fix this somehow before proceeding (Edit: "Fixed" by disabling the module; note to self: Get rid of this module).

Edit: It works already in the migrated database, it seems to suffice to import both content types and run "Amazontools Legacy Import".

Nodes of 'amazon' and 'amazon_node' content types do look fine now; they show a cover image and some basic product information, but no pricing or release date. Also, they have a node body, and they're editable now. When replicating AAT's data model, in 'amazon' and 'amazon_node' the field 'Legacy ASIN' could be single value field; it would ony have to be multi value for the "Related links from Amazon". But I think it's fine as it is.

Something still seems to be wrong with the converted ISBN/ASIN fields from D5 ('field_asin' in my db); they are attached to the node and show up on the node editing form, but not in the node display; I triple checked the field's display settings at ./admin/content/node-type/film/fields/field_asin and they em/em> should be visible.

In the meantime I'll set up another D5 test database to try this as instructed in #62.

Thanks & greetings, -asb

asb’s picture

Status: Needs work » Fixed

These are the results of a completely new migration approach with a different site. I'm starting with a cloned D5 database without any contributed modules. After upgrading Drupal core, I'm following #57. In this case I "preprocessed" the files and images folders on D5 to avoid any unnecessary obstructions (moved from ./files to ./sites/default/files and fixed the database records).

Importing the 'amazon' and 'amazon_node' node types before enabling the 'Amazon API' seems to be crucial since I'm now getting a visual on the migrated 'ISBN/ASIN' fields from D5 for the first time. The configuration of the ASIN fields at ./admin/content/node-type/{nodetype}/display still seems to be ignored on some nodes, no matter if working with 'Legacy ASIN' or migrated 'ASIN' fields ("Small image and basic info" and "Small image and full info" seem to output the same amount of data; I'll have to cross-check this with the full array pulled from Amazon (./admin/settings/amazon/test) and the Amazon website, the product data simply might be outdated).

"Amazontools Legacy Import" works smoothly; however there is still a discrepancy between the number of 'amazon' plus 'amazon_node' nodes from D5 and the number of items processed by the importer; the difference might be the "Related links" that seem to be processed by the importer, also.

I'll now upgrade the remaining modules, check sample nodes and then report back. As far as I have seen this looks very good!

Thanks & greetings, -asb

asb’s picture

Third and completely new migration approach, different site, size of db dump ~700 MB; more complex because this site was started on D4. I'm documenting my steps, based on #57, #59, and #62.

What is migrated from D5's 'Amazontools' (AAT) and 'ISBN/ASIN' field modules:

  • 'amazon' and 'amazon_node' nodes from AAT;
  • "Related Amazon links" attached to any other content type;
  • 'ISBN/ASIN' CCK fields, attached to any other node.

What isn't migrated from D5's 'Amazontools' and 'ISBN/ASIN' field modules:

  • Ratings from AAT;
  • Blocks from AAT (e.g. "Amazon: Random review", "Amazon: We recommend", "Amazon: Random book")
  1. (optional) If you have a old site, e.g. where files are located at system/files, fix these old file storage paths in D5; you'll need access to your database and a tool like Search & Replace Scanner (to correct references to the old storage path(s) in your nodes); the recommended path for files in D6 is ./sites/default/files;
  2. Make a full site backup, including database, configuration files, and "files" folder; verify that it can be restored;
  3. Clone the D5 database, because you'll need to have the D5 and D6 site running in parallel; for setup instructions consult the Installation handbook;
  4. Put your site in maintenance mode at ./admin/settings/site-maintenance
  5. (optional, but highly recommended) Switch to a core theme; Garland is recommended;
  6. (optional, but highly recommended) Disable all contributed (non-core) modules on the D5 site; don not uninstall any module you want to have on D6!
  7. Unpack a full D6 archive in your site's root folder;
  8. Copy the "files" folder from the D5 site to sites/default on the new site;
  9. Run update.php to update Drupal core; this might take about an hour on bigger sites;
  10. (optional) Update essential modules, e.g. image.module;
  11. Install CCK module into sites/all/modules (e.g. with Drush: drush dl cck);
  12. Enable content.module;
  13. Run update.php;
  14. (optional) Enable other core modules from CCK (no other CCK modues!);
  15. Run update.php;
  16. Install Amazon module into sites/all/modules.
  17. Create the amazon and amazon_node content types (exports: see #62);
  18. Enable Amazon API from amazon.module (NOT ASIN yet);
  19. Run update.php; ignore errors and/or warnings during database update;
  20. (optional) Enable 'Amazon Filter', 'Amazon media', 'Amazon search'
  21. Run update.php;
  22. Enable 'Amazon field' (= "ASIN field") from 'Amazon' module;
  23. Run update.php. This may take a while as it processes nodes with attached ISBN/ASIN field (from D5); it's displaying a progress bar and the node it is currently processing; ignore warnings like user warning: Unknown column 'field_asin_asin' in 'field list' query: INSERT INTO content_field_asin (vid, nid, field_asin_asin) SELECT vid, nid, field_asin_asin FROM content_type_buch in /var/www/drupal6/sites/all/modules/cck/includes/content.admin.inc on line 1553. or user warning: Can't DROP 'field_asin_asin'; check that column/key exists query: ALTER TABLE content_type_buch DROP field_asin_asin in /var/www/drupal6/includes/database.mysql-common.inc on line 322.
  24. Enable the "Amazon Legacy Import" module from 'Amazon' module;
  25. Visit Admin->Settings->Amazon Legacy Importer (located at ./admin/settings/aat_legacy) to convert the old amazonnodes and amazon related links data; it'll display a screen "Migrating amazonnode field values into field_legacy_asin CCK fields" with progress bar and the node it's currently processing; this might take a long time (depending on the amount of 'Amazontools' nodes and you processor's speed, this might take even several hours);
  26. Run update.php;
  27. Enable other CCK modules (like date);
  28. Run update.php;
  29. Move all other modules back into sites/all/modules and enable them as needed; it's recommended to enable one contibuted module after the other, each followed by running update.php;
  30. Run update.php;
  31. Visit your Amazon settings page at admin/settings/amazon. You'll have to change the locale, if you're not afilliated with Amazon.com ('US')

Following these steps might take half a day or longer, so better do this offline. Also, this might cause heavy cpu and db load; if you're monitoring MySQL, you might encounter 1500-2000 qps or more.

After the import has finished, you might want to verify the success of the migration. Keep in mind, that all migrated nodes are "normal" CCK-based content types with no proprietary properties (as AAT nodes) anymore. You can do anything with them what you can do with any other CCK node type.

  • Check you watchdog log for errors and warnings at ./admin/reports/dblog
  • Compare the number of 'amazon' and 'amazon_nodes' in D5 and D6; that kind of information you can get directly from querying your database (with SQL) or with the contributed Systeminfo module; in my site, I get an exact match (e.g. 'amazon' -> 1626 nodes, 'aazon_node' -> 13782 nodes both on D5 and D6);
  • Filter out 'amazon' nodes at ./admin/content/node; review some nodes; check if the ASIN data has been pulled in correctly; by default, you can expect a small cover image and a linkt to Amazon's product page; for affiliates of Amazon.de (Germany) usually no pricing information is displayed, even for newly published books (needs further investigation); also note, that the cover images by default are not hyperlinked to Amazon's product page; you might want to change this in your theme's template file (is this the way to go for this?);
  • Filter out 'amazon_node' at ./admin/content/node; review some nodes;
  • (optional) Use Views Bulk Operations to filter down your content more selectively;
  • Identify where CCK fields of the type 'Amazon item' are used at ./admin/content/types/fields. They're labeled either something like 'field_asin' (the field's name you provided for 'ISBN/ASIN' fields in D5), or 'field_legacy_asin' ("Related Amazon links" from AAT); the latter will include the content types 'amazon' and 'amazon_node'. Review some nodes of these content types and compare them to their D5 aequivalent.
  • Check the content types with attached ASIN fields at ./admin/content/node-type/{your-content-type}/fields, look out for errors and warnings; check the display settings at ./admin/content/node-type/{your-content-type}/display

The AAT blocks can be manually rebuilt with views:

  • "Amazon: Random review" selects one random 'amazon' node;
  • "Amazon: Random book" selects one random 'amazon_node' node (not necessarily limied to books!)
  • "Amazon: We recommend" selects one 'amazon' node with a freely selectable minimum rating; this requires 'Amazon' module and something like 'Voting API' plus 'Fivestar' in D6

Other cleanup work after a D5->D6 migration:

  • Views: All views based on data from AAT will break, they're not converted/migrated automatically. Also note that D5 had Views 1, D6 has Views 2, so you'll have to rework your views anyway.
  • CCK: Many CCK fields behave differently on D6; you'll want to visit every content type and check the field's configuration as well as the display settings. E.g., on all my migrated sites fieldsets were excluded from display after the upgrade, so neither the fieldsets itself nor the fields contained in it showed up. Also check the weight of the newly created and imported ASIN fields, they might have a value of "10" (on the very bottom).
  • Themeing: ASIN fields from 'Amazon' module might be formatted strangely; especially 'amazon' nodes do look very differently after the migration; fix this in your theme's CSS, if you liked the AAT's formatting better (I'll post suggestions later). Since 'field_legacy_asin' is now a multivalue field, it might be difficult to mimic AAT's (single value) appearance
  • Taxonomy: If your 'Amazon' and 'Amazon_node' nodes had tags, remember to visit ./admin/content/taxonomy and enable the proper categories for those content types (the old settings are not migrated automatically)
  • i10n/i18n: If you are using a non-English (= localized) version of Drupal, the contributed module's translations might disappear after the Upgrade. Re-importing the translation packages can be done automatically with the Localization client.

(to be extended/updated and become a handbook page sometime ;)

Greetings, -asb

asb’s picture

FileSize
5.71 KB

This documentation is now a handbook page.

Additionally you might want to modify your stylesheet with some CSS to remove the additional spacing in the "show full product details" display of the ASIN field, e.g.:

div.amazon-item div {
  height: auto;
  line-height: 1.5em;
  margin-left: 100px;
  padding-left: 1em;
  margin-top: auto;
  margin-bottom: auto;
}

I'm also attaching a very basic view that creates an AAT-like "We read" block. The view filters down to nodes with an attached 'field_legacy_asin' (you might want to change this), and displays a random product with title and product image in a block. It also (hopefully) takes care not to display products with invalid ASINs.

You might want to create a template if you want to style this block properly, e.g. something like views-view--Amazon--block-1.tpl.php. Visit the template suggestings in views for more about this.

This will center the node title and the product cover in the newly created block:

.view-display-id-block_1 .views-row {
display: block;
margin: 0px auto;
text-align: center;
}

.view-display-id-block_1 .views-field-product-image {
margin-top: 3px;
margin-bottom: 3px;
}

If you want to mimic the "We recommend" blocks from AAT, you'll have to add something like 'Fivestar' ratings to your nodes and to the view. That will enable you to display only the best rated products (books, dvds, whatever). The 'Amazon' module currently seems not to pull in user ratings from Amazon, so you'll have to provide your own ratings ;)

Good luck - I'd love to hear about your ideas!

-asb

rfay’s picture

@asb, very nicely done handbook page. Thanks for the excellent effort you put into that.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.