I tried to create a better login or register page when an anonymous user tries to purchase a product, so the login / register message instead of displaying it in the message / warning , IMHO its better to display the message in a page callback.

the code :
add the code in uc_paypal_buttons_menu()

 uc_paypal_buttons_menu() {
  $items['checkout-register'] = array(
   'title' => t('Registration required'),
   'page callback' => 'uc_paypal_buttons_checkout_register',
   'access arguments' => array('access content'),
   'type' => MENU_CALLBACK,
   );
}

new function (need to have theme hook IMHO)

function uc_paypal_buttons_checkout_register() {
if ($user->uid == 0 ) {
    $output .= '<div class="register-wrapper">';
	$output .= '<div class="register">';
	$output .= '<p>To continue the purchase, you will need to register first with us. Without registering the download link will not be provided to you</p>';
	$output .= '</div>';
	$output .= '<div class="login-page">';
	$output .= '<p>If you already is our member but forgot to sign in, please continue to the ' . l('login page' , '/user') . '</p>';
	$output .= '</div>';
	$output .= '<h1>Registration form</h1>';
	$output .= drupal_get_form(user_register);
	$output .= '</div>';
} else {
	drupal_goto('user/');
	}	
	return $output;
	}

Still have problem with this one

 function uc_paypal_buttons_user($op, &$edit, &$account, $category = NULL) {

  if ($op == 'login' && !empty($_SESSION['uc_paypal_buttons_redirect_url']) && $_SESSION['uc_paypal_buttons_redirect_nid']) {
	$redirect = $_SESSION['uc_paypal_buttons_redirect_url'];
    $nid = $_SESSION['uc_paypal_buttons_redirect_nid'];
    $node = node_load($nid);
    //drupal_set_message(t('Click !here to continue to checkout for your purchase of @product.', array('!here' => l(t('here'), $redirect), '@product' => $node->title)));
  }
}

Removing the drupal set message and point to the menu callback


function uc_paypal_buttons_create_order($nid, $quantity, $attributes = NULL) {
  global $user;
	unset($_SESSION['uc_paypal_buttons_redirect_url']);
	unset($_SESSION['uc_paypal_buttons_redirect_nid']);
  // Load product node and product feature settings
  $node = node_load($nid);
  $options = uc_paypal_buttons_product_options($nid);

  // Check if the current user is logged in and
  // if the current product requires a registered user
  if ($options['requireuser'] && !$user->uid) {
    $redirect = 'uc_paypal_buttons/'. intval($nid) .'/'. intval($quantity);
    if (is_array($attributes)) {
      foreach ($attributes as $key => $val) {
        if (!empty($key) && !empty($val)) {
          $redirect .= '/'. urlencode($key) .'/'. urlencode($val) .'/';
        }
      }
    }
    $_SESSION['uc_paypal_buttons_redirect_nid'] = $nid;
    $_SESSION['uc_paypal_buttons_redirect_url'] = $redirect;
	drupal_goto('checkout-register');
//    drupal_set_message(t('Please log in or !register for an account before proceeding to checkout.', array('!register' => l(t('register'), 'user/register'))), 'error');
//    drupal_goto('user');
  }

Current condition:

I got the paypal button to redirect to checkout-complete without problem.
The user can either login or register without problem.
the $_SESSION is forwarded correctly.

The problem:

Its best IMHO to auto redirect user to paypal payment page after the registration / login complete, try that by adding drupal_goto($_SESSION['uc_paypal_buttons_redirect_url']) in the function uc_paypal_buttons_user($op, &$edit, &$account, $category = NULL) but doesn't work

Then to tackle the matter I modified the user-profile.tpl.php to redirect to $_SESSION['uc_paypal_buttons_redirect_url'] which is not really the best solution IMHO.

do you have any better solution for this? because I think its a great feature for the module to have.