UC PayBox not invoking hook_order case submit

Your hook submit doesn't seem to be in the right place. You call it before being certain there's no error in your Paybox module call.

What I did was call hook_order_submit right before outputting the redirect to the PayBox gateway in uc_paybox.pages.inc

$path_bin = variable_get('uc_paybox_cgi_path', '');
  $chaine = implode($chaine);
  $output = shell_exec(trim($path_bin . $chaine));
  $output = ereg_replace("Content-type: text/html\nCache-Control: no-cache, no-store\nPragma: no-cache", '', $output);
  if ($output) {
  	// insure order is set to "in checkout" before leaving for paybox gateway
  	foreach (module_implements('order') as $module) {
    	module_invoke($module, 'order', 'submit', $order, NULL);
  	}
   	return $output;
  }

The problem with a gateway is that when the user is redirected from your site you no longer have any control over what happens and you're in a state where you might just no longer get any feedback.
What the above code implies is that the order is created and set to "In Checkout", and unless something is done about it, it stays there.
This is why it's now necessary to add code to each of the return links in pages.inc to cancel the order in case of an error. The way your dev version's code is right now, if somebody spams going to the gateway and cancelling, he's gonna create as many 'in_checkout' orders as he wants.

I did this crudely by adding the following lines of code to annule/refuse functions and auto in case of error:

$order = uc_order_load($orderid);
  if ($order) {
    unset($_SESSION['cart_order']);
    // add comment
    uc_order_comment_save($orderid, $user->uid, t('This orders payment has been refused by PayBox.'), 'order');
    // check status
    if (uc_order_update_status($orderid, uc_order_state_default('canceled'))) {
      drupal_set_message(t('Your order payment has been refused.'), 'status');
    }
    else {
      $error = TRUE;
      watchdog('uc_paybox', "Order @order couldn't be marked as 'Canceled'.", array('@order' => $orderid), WATCHDOG_WARNING, NULL);
    }
  }

Now in a nominal scenario you'll always get a return link from PayBox. Even if the user closes the gateway's window manually without returning to the site, you'll always get an "auto" link back from PayBox, which at worst is a 15 minute notice of inactivity.

Now it could be that your site has downtime at the exact moment this auto comes back, which is why, in order to make this completely robust, I added a cron run that sends an email to the administrator that basicly goes "hey, a command has been in checkout for over 24 hours, you should check if its been paid or not and deal with it manually". Should basicly never happen anyway.

Again going in on the problem where your order can be in a state where you have no idea what PayBox is doing (the moment between when you send your user to paybox and when you get any of the return URLs), the user can still act. Let's imagine for example a user that we send off to Paybox and his payment is accepted, but the site has downtime and we don't get the return link.
The user doesn't see his order, figures something went wrong with payment, buys the same product again, pays twice.
This is why I add this code :

if (uc_order_incheckout() || uc_order_pending()){
  	drupal_set_message(t('Vous avez une commande en attente de validation. Cette erreur peut apparaître lorsque vous quittez brusquement la page de paiement PayBox, ou lorsque une commande est encore en attente de validation du paiement. Si cette erreur est toujours présente dans 30 minutes, veuillez contactez un administrateur.'));
    drupal_goto('cart');
  }

function uc_order_incheckout() {
	global $user;
	$count = 0;
	$status = 'in_checkout';
	$result = db_query("SELECT * FROM {uc_orders} WHERE uid = %d AND order_status = '%s'", $user->uid, $status);
	while ($node = db_fetch_object($result)) {
		$count = $count + 1;
  	}
	return ($count>1);
}

as the first thing to do when entering the page to stop users from going through payment while we're still waiting for an update from PayBox.
Again, should never happen, but you never know. I'd rather have them stuck without being able to pay for a day than having them go sheep on me and pay for the same thing 3 times.

Something else I noticed is that after your "auto" return, the status of the order is updated to "payment received" but it doesn't check for payment completion. This is because you assume the "effectue" link will always be called after "auto", when in fact it's not. I added a call to the uc_paybox_effectue() right before the exit() in uc_paybox_auto() to fix this (which is not pretty, I know).

And that pretty much fixed this module's behaviour to go along perfectly with Signup's integration for Ubercart. If you want me to clean this up and propose a patch I'd be glad to give it a shot. If you think this behaviour isn't needed for all users then I'll just stick with my code.