Is there any way to ensure nids will not change when deploying across servers???
To summarize, the problem is that when a node is deployed, there is no guarantee that the node will have the same ID on the destination server.

I've had a lot of trouble with this and really need some help. I understand that this may be a "won't fix" right from the start, but if anyone could help, that would be great. It applies to multi stage environments where any node-related content must go "backwards" from production to beta/staging/dev or where two nodes that reference each other by nid are deployed.

If more detail in required, read this example:
Server1 = dev, Server2 = production. Comments are made on Server2 and should be ported down to Server1 (comments are just an example, this can happen with anything that references nodes). Let's assume that Server1 and Server2 are practically identical in form and data.
On Server1, an editor creates two new articles, getting the autoincrement nids 5 and 6 (these, of course, are arbitrary). The editor then realizes 5 isn't ready for production, but 6 is, so the editor deploys 6 from Server1 to Server2 where 6 will get the nid 5 (because Server2's autoincrement value was never incremented when 5 was created in Server1), then when 5 is deployed, it will get the nid 6, effectively switching their nids. When users comment on each of the articles, the comments reference the new nid assigned on Server2. Then when comments are ported down from Server2 (where they were created) to Server1, 6's comments will reference nid 5, and 5's comments will reference nid 6. As you can see, this is a very simple example and can easily turn into an unrecoverable situation with many articles and comments.

Is there any way to ensure nids will not change across servers???

The deploy module could send the original nid from Server1 to Server2, where Server2 would force the new article to be created with that nid. However, that solution seems like it could cause some mySQL autoincrement problems (and would require hacking node_deploy.module and maybe core). I am utterly lost and would hate to deprive my editors of content deployments, forcing them to only update content when we do a full code & db deployment. We have considered third-party comment systems, but they rely on ids as well, so the problem persists even if you don't have to port comments "backward". Is there a UUID solution?

Any ideas, suggestions, questions, clarifications would be greatly appreciated.

Thanks in advance!

Comments

gdd’s picture

Deploy implements UUIDs for exactly this purpose. You may be interested in reading some of the background posts about this at

http://heyrocker.com/drupal

and

http://palantir.net/blog/bringing-deployment-capability-drupal

You are right about porting the comments down from live, however Deploy works both ways. It can not only map content dev->live but live->dev so you can manage comments that way.

I know you've been in my issue queues so I thought you would realize this. If you are talking about actually replacing the nid with a uuid, no that does not exist. Deploy lays them alongside then nid and uses them as needed. I am hoping to get a real UUID solution into Drupal 8.

The sad fact is that 98% of Drupal sites out there today just do all their content work right on the live site, or they do everything on the back end and do dumps forward (a decent solution for sites with no user-submitted content.)

teastburn85’s picture

Thank you so much for your prompt reply heyrocker! I have read most of the articles on your site already, thanks :)

Haha yes, I realize that's what UUID is for and it is awesome, but unfortunately hardly anyone uses it so I don't get to benefit from it too often. Tired and frustrated I wrote a module that replaced nids with your UUIDs, but too many modules (including, somewhat ironically, the UUID module) rely heavily on the exact current schema definition of nids, thus when they cast the UUIDs to integers, everything crumbles. UUIDs would indeed be an acceptable solution with a third-party comment system such as Disqus.

If only the rest of the world would use your UUID solution!!! The current nid system has only caused us pain.

Your solution of using deploy both directions is intriguing. It had crossed my mind but I shot it down because deploy is a GUI program and, AFAIK, cannot be scripted. With over a thousand articles and (hopefully) multiple comments per article, that would be a lot of manual GUI repetition! Is there a way to script it, even if it's a hack? Some kind of drush integration would be awesome, but sounds quite difficult.

Do you know if forcing the nid to be the same on the destination server is out of the question? It looks like hacking core would have to take place, but I'm relatively new to Drupal so am not positive about that.

Again, I really appreciate your help and work!

gdd’s picture

Yes straight up replacing NIDs with UUIDs simply will not work as Drupal exists currently, for better or worse. Drush integration was committed last week. It needs a little work to work completely without human interaction (it still requires manual entry of username and password) but that would not be hard to add. Here is the issue.

http://drupal.org/node/795408

Forcing the NID to be the same is just not going to work. All the database tables use autoincrements, and there's just no way to control them to that levels.

teastburn85’s picture

Awesome! Drush integration is crucial!

So the idea is I can deploy only comments from live -> dev/beta/staging and not the whole article? Or the article gets deployed and the comments go with it? That would seem to wipe out any changes to the article on dev.

Thanks again!

gdd’s picture

Yes, you can do that. It will require some integration code in a custom module, but perhaps I can write up a handbook page on that since it is bound to be a common request. It will only deploy the comments, not the whole article. I have this system in place for a client right now actually and it works really well.

I also made a patch to allow Drush integration to run unattended. See

#893828: Expand Drush actions to allow unattended deployments

teastburn85’s picture

I updated deploy, downloaded the drush patch, wrote a script to grab the latest comments and deploy them and it worked great! However, my coworkers decided on using the Disqus module for commenting, which nullifies the deploy comments stuff. In order to use it, though, I had to use the node UUID for the comment thread identifiers. Let me know if you'd like me to share either of these approaches.

Thanks again to you and UUID!

dixon_’s picture

Status: Active » Closed (works as designed)

Looks like things got sorted out here :)

john.money’s picture

Status: Closed (works as designed) » Active

I'm going to reopen this to get some thoughts on how it might better work. Here is a (my real-live) scenario:

  • there is a new site section which includes 50 nodes to be published in a month
  • these nodes are created on staging server, nids 1001-1050
  • there is inter-linking between these nodes within the content of the new section, so the node numbers matter
  • before the new site section is ready, there is a need to create a new page to be deployed immediately
  • it is assigned nid 1051 on stage but becomes 1001 on production
  • when the new site section is eventually deployed, all the content links pointing to 1001 are not going to work as intended (e.g. big QA fail which defeats the purpose of using Deploy)

I don't see how I am going to ever reconcile this situation unless I force node IDs on node_save. Thoughts?

*edit: came across a d7 sandbox module node renumber which needs further investigation as an approach to forcing a node ID.

john.money’s picture

As mentioned in my above comment, Deploy is a non-starter if you cannot QA nodes on staging and ensure that any content node ID links will remain valid when pushed to production. I am posting this as proof-of-concept since I needed a solution immediately.

Caveats:

  • no patch is provided since it requires a core hack
  • it has been tested at this point on only my unique D6 configuration and may eat your nodes, render your site inoperable or otherwise cause you a really bad day
  • it has been tested only one direction (stage -> production) but should in theory work bi-directionally; it would have very little benefit with more than one sending server
  • it has been tested only on MySQL and will likely not work on <insert your DB flavor>
  • the core hack is as minimally invasive as possible, but still does not make me particularly happy

Conceptual solution:

  • at node_deploy,
    • if no remote nid exists (new node)
      • then store the current nid as part of the node object (suggested_nid)
  • at node_save,
    • if no current nid exists (new node)
    • and suggested_nid variable exists (propagated across node prepare and presave)
    • and suggested_nid is not already in use
      • then modify drupal_write_record to ignore the serial field and use the suggested_nid

And the code:

In sites/all/modules/mymodule/mymodule.module, add:


/**
 * Implementation of hook_nodeapi().
 */
function mymodule_nodeapi(&$node, $op, $a3 = NULL, $a4 = NULL) {
  // Preserve suggested nid value throughout drupal_execute node->form->node.
  static $suggested_nid = NULL;

  switch ($op) {
    case 'prepare':
      if ($node->suggested_nid) {
        $suggested_nid = $node->suggested_nid;
      }
      break;

    case 'presave':
      if ($suggested_nid) {
        $node->suggested_nid = $suggested_nid;
      }
      break;
  }
}

/**
 * Implements hook_node_deploy
 * 
 * @param object $node A reference to the node that will be deployed
 * @return $node The edited version
 */
function mymodule_node_deploy(&$node) {
  // Set the suggested nid to be handled by drupal_write_record().
  if (!isset($node->remote_nid)) {
    $node->suggested_nid = $node->nid;
  }  

  return $node;
}

(ignore closing php bracket)

In includes/common.inc, modify function drupal_write_record:

  $fields = $defs = $values = $serials = $placeholders = array();

+  // Special handling for Deploy suggested nid. Works in conjunction with 
+  // hook_node_deploy() and hook_nodeapi() prepare/presave ops.
+  if ($table == 'node' && empty($object->nid) && (int)$object->suggested_nid) {
+    // Is suggested nid available?
+    if (!db_result(db_query('SELECT nid FROM {node} WHERE nid=%d', $object->suggested_nid))) {
+      // Set suggested nid and fool schema.
+      $object->nid = $object->suggested_nid;
+      $schema['fields']['nid']['type'] = 'int';
+    }
+    else {
+      watchdog('deploy', 'Could not set suggested nid %nid', array('%nid' => $object->suggested_nid), WATCHDOG_ERROR);
+    }
+  }
+
  // Go through our schema, build SQL, and when inserting, fill in defaults for
  // fields that are not set.

(ignore closing php bracket)