After frantically searching for a solution I found out about the l(); function pestering me with plain text.

This already seemed to be a problem in Drupal 5 but no clear workaround was found. I overwrote the theme_pager() function, only for changing the $li_previous and $li_next variables.

Original:

$li_previous = theme('pager_previous', (isset($tags[1]) ? $tags[1] : t('‹ previous')), $limit, $element, 1, $parameters);

Changed to:

$li_previous = theme('pager_previous', (isset($tags[1]) ? $tags[1] : theme_image('image', 'sites/all/themes/landrush/img/vorige.jpg', $alt = 'Vorige', $parameters, FALSE)), $limit, $element, 1, $parameters);

By using the theme_image() function Drupal should nicely format the img tag and output it to your theme. Well, this works, but unfortunately somewhere in this theme_pager process a l(); is telling Drupal to only output plain text here.

The trick here is not to rewrite your whole pager functions. Have a look at http://api.drupal.org/api/function/theme_pager_link/6 .
This function is building all the links which are used by the pager functions. When you look closer to the l(); formatting the output:

return l($text, $_GET['q'], array('attributes' => $attributes, 'query' => count($query) ? implode('&', $query) : NULL));

you'll see that the options aren't setting the html output. So I placed the whole function in my template.php (changing theme_pager_link to landrush_pager_link) with a different output:

return l($text, $_GET['q'], array('attributes' => $attributes, 'html' => TRUE, 'query' => count($query) ? implode('&', $query) : NULL));

Note the 'html' => TRUE.

So now l(); is allowing me to print html tags (images!) within the pager function.

My complete code (in template.php):

<?php
// Override the ending l() function to allow html in the pager links (thanks Kaaskop & Swentel)  :
function landrush_pager_link($text, $page_new, $element, $parameters = array(), $attributes = array()) {
  $page = isset($_GET['page']) ? $_GET['page'] : '';
  if ($new_page = implode(',', pager_load_array($page_new[$element], $element, explode(',', $page)))) {
    $parameters['page'] = $new_page;
  }

  $query = array();
  if (count($parameters)) {
    $query[] = drupal_query_string_encode($parameters, array());
  }
  $querystring = pager_get_querystring();
  if ($querystring != '') {
    $query[] = $querystring;
  }

  // Set each pager link title
  if (!isset($attributes['title'])) {
    static $titles = NULL;
    if (!isset($titles)) {
      $titles = array(
        t('« first') => t('Go to first page'),
        t('‹ previous') => t('Go to previous page'),
        t('next ›') => t('Go to next page'),
        t('last »') => t('Go to last page'),
      );
    }
    if (isset($titles[$text])) {
      $attributes['title'] = $titles[$text];
    }
    else if (is_numeric($text)) {
      $attributes['title'] = t('Go to page @number', array('@number' => $text));
    }
  }

  return l($text, $_GET['q'], array('attributes' => $attributes, 'html' => TRUE, 'query' => count($query) ? implode('&', $query) : NULL));
}

// Function borrowed from http://api.drupal.org/api/function/theme_pager/6 to display less items 
// in our smaller space (needs above function to allow html!)
//
// NOTE: I modified a lot in this function, if you notice a flaw please tell me, haven't had the time to test it yet..
//
function landrush_pager($tags = array(), $limit = 10, $element = 0, $parameters = array()) {
	global $pager_page_array, $pager_total;

	$pager_current = $pager_page_array[$element] + 1;
	$pager_first = $pager_current - $pager_middle + 1;
	$pager_last = $pager_current + $quantity - $pager_middle;
	$pager_max = $pager_total[$element];

	$i = $pager_first;
	if ($pager_last > $pager_max) {
		$i = $i + ($pager_max - $pager_last);
		$pager_last = $pager_max;
	}
	if ($i <= 0) {
		$pager_last = $pager_last + (1 - $i);
		$i = 1;
	}

	$li_previous = theme('pager_previous', (isset($tags[1]) ? $tags[1] : theme_image('sites/default/files/prev.jpg', 'Previous')), $limit, $element, 1, $parameters);
	$li_next = theme('pager_next', (isset($tags[3]) ? $tags[3] : theme_image('sites/default/files/next.jpg', 'Next')), $limit, $element, 1, $parameters);

	if ($pager_total[$element] > 1) {
		if ($li_previous) {
			$items[] = array(
				'class' => 'pager-previous',	
				'data' => $li_previous,
			);
		}

		for (; $i <= $pager_last && $i <= $pager_max; $i++) {
			if ($i > $pager_current) {
				$items[] = array(
					'class' => 'pager-item',
					'data' => theme('pager_next', $i, $limit, $element, ($i - $pager_current), $parameters),
				);
			}
		}
		if ($li_next) {
			$items[] = array(
				'class' => 'pager-next',
				'data' => $li_next,
			);
		}
		return theme('item_list', $items, NULL, 'ul', array('class' => 'pager'));
	}
}
?>

Drupal 5

In Drupal 5, you can override the theme_pager_link function as such to use images as pager links :

<?php
function phptemplate_pager_link($text, $page_new, $element, $parameters = array(), $attributes = array())
{
  $page = isset($_GET['page']) ? $_GET['page'] : '';
  if ($new_page = implode(',', pager_load_array($page_new[$element], $element, explode(',', $page)))) {
    $parameters['page'] = $new_page;
  }

  $query = array();
  if (count($parameters)) {
    $query[] = drupal_query_string_encode($parameters, array());
  }
  $querystring = pager_get_querystring();
  if ($querystring != '') {
    $query[] = $querystring;
  }

  // Set each pager link title
  if (!isset($attributes['title'])) {
    static $titles = NULL;
    if (!isset($titles)) {
      $titles = array(
        t('« first') => t('Go to first page'),
        t('‹ previous') => t('Go to previous page'),
        t('next ›') => t('Go to next page'),
        t('last »') => t('Go to last page'),
      );
    }
    if (isset($titles[$text])) {
      $attributes['title'] = $titles[$text];
    }
    else if (is_numeric($text)) {
      $attributes['title'] = t('Go to page @number', array('@number' => $text));
    }
  }

  return l($text, $_GET['q'], $attributes, count($query) ? implode('&', $query) : NULL, NULL, FALSE, TRUE);
}
?>

The important part is the last TRUE in the returned l() parameters, which lets you print html in your links without getting it escaped.

I was then able to use things like

<?php
$output .= theme('pager_first', theme('image', path_to_theme() . '/images/pager/pager_first.png'), $limit);
?>