I discovered this problem yesterday and spent a couple hours today solving it, and thought I'd share. There seems to be a bug in the way Drupal writes urls when clean urls are enabled, which results in broken pager links. I encountered this with categories which have more than one page, but it probably applies to other uses of the pager as well. The problem is simple: the links are of the form "http://www.mysite.com/taxonomy/term/##?page=##" when the correct form is "http://www.mysite.com/taxonomy/term/##&page=##". The difference is a single character: there is a "?" where there should be a "&". The latter works, the former does not.

I traced the problem to the function "url" in includes/common.inc. Line number 1033 of that file reads

return $base . $path . '?' . $query . $fragment

when it should be

return $base . $path . '&' . $query . $fragment

This is only necessary for clean urls on pages other than the front page. In other cases the code seems to work fine as it is.

The above information applies to Drupal 4.7.4. I checked also the documentation for 4.6.x, and the same problem seems to be there as well, although the line is a different number.

There may of course be a reason why the '?' is there and not the '&', and I have missed it. Can someone verify this?

In case anyone wants to know my exact rewrite rules and such settings, ask away. I doubt they are relevant though, since I used a slight variant of the rules given in the Drupal docs.

Comments

Gurpartap Singh’s picture

Gurpartap Singh commented
Project: Paging » Drupal core
Version: 4.7.x-1.x-dev » 4.7.x-dev
Component: Code » menu system
Status: Needs review » Postponed (maintainer needs more info)

Well, '?' is the right usage on there. '&' is required if there already exists an argument in the url, else '?' is used. However, your case might be different in some sense, maybe.

mantyla’s picture

mantyla commented

You appear to have missed the key words, namely that this problem appears with clean urls active. The rewrite rule I have active, which is unchanged from the one in the .htaccess file that is part of the Drupal distribution, is to add "index.php?q=" in front of everything. This is required for clean urls to work. Note that because of this rewrite rule there is always an argument in the url, and therefore if you add another you should use '&' and not '?'.

Also note that the code I made the change to does not handle the front page, which is handled separately in the url function. In the case of the front page, where the above rewrite rule is not used, there will be no previous argument. However, the function uses '?' correctly in this case.

lccweb’s picture

lccweb commented

i encountered the same problem first mentioned. however, our fix was to modify this part of the php.ini to include "?" as a delimiter.

; List of separator(s) used by PHP to parse input URLs into variables.
; Default is "&".
; NOTE: Every character in this directive is considered as separator!
;arg_separator.input = ";&"
arg_separator.input = "?;&"

it seemed to cure our paging problem with clean url enabled. i hope this helps.

twooten’s picture

twooten commented
Version: 4.7.x-dev » 5.7

Is this problem fixed? I'm using Drupal 5.7 and have clean urls working. I encountered a problem and posted about it here http://drupal.org/node/270956. I stumbled across this post and went to common.inc and on line 1205 changed 

return $base . $path .'?'. $query . $fragment;

to

return $base . $path .'&'. $query . $fragment;

and now my pager is working. I don't know if it's going to break anything else but I thought I'd ask about it.

Tim

Gurpartap Singh’s picture

Gurpartap Singh commented
Version: 5.7 » 7.x-dev
Priority: Minor » Normal
pwolanin’s picture

pwolanin commented

Which browser(s) have problems? I've always had the pager work fine with FF or Safari.

twooten’s picture

twooten commented

I'm using Firefox 2.0.0.14 on linux. Drupal 5.7

mantyla’s picture

mantyla commented

It seems I picked a good time to check back after a long absence...

This issue remains unresolved, even though I made another post with patches, found here: http://drupal.org/node/106793. The patches are now outdated, but the problem is still the same, only the code line number has changed.

In the past, I've tried to explain the issue only to end up misunderstood. Now I will try to be as specific as I can about what makes this bug manifest itself and what does not. I thought I had been clear enough before, but apparently I was mistaken.

So, here's what is required for the bug to appear:

1. Clean URLs must be active
The bug is in a section of code that only applies to clean URLs. If you're not using them, the bug will never appear.

2. You must click on a Drupal-generated pager link not located on the front page
The front page is handled separately in the code and that section is bug free. So you need to find a link elsewhere, such as in a category.

3. PHP has default separators
I've been told twice that there should be a '?' before the first argument and a '&' before the second and all subsequent arguments. I know this, please don't tell me a third time. The problem is that in the code '?' is used twice. Of course, if you include '?' as a separator, the problem never appears - as far as I can tell, this is exactly what the Drupal site itself does. It is still a bug though, since this workaround requires a nonstandard PHP configuration.

4. The default RewriteRule in the root .htaccess file is in effect
The default rule adds "index.php?q=" in front of everything that is not a recognized file or directory - that is, every Drupal path, and every unrecognized path that is not a Drupal path too.

5. Nothing else?
As far as I know, nothing else matters. The browser you use is irrelevant.

And here is a step by step description of what happens:

  1. You click on a Drupal-generated pager link, such as "http://www.mysite.com/taxonomy/term/1?page=1".
  2. A HTTP Request is sent to the server, such as "www.mysite.com" in this example.
  3. The server receives the request, and processes it.
  4. The server finds a ".htaccess" file in the root directory containing the line "RewriteRule ^(.*)$ index.php?q=$1 [L,QSA]". The rewrite module is active, as it must be for clean URLs.
  5. Since "taxonomy" is not an actual directory, the RewriteRule applies. The path "taxonomy/term/1?page=1" is rewritten into "index.php?q=taxonomy/term/1?page=1".
  6. As the new path contains '?' twice, the latter argument is ignored, and the pager will not work. You receive the page 0 instead. Unless of course PHP has been configured to accept '?' as a separator, which it is not by default.

I have been patiently updating my own Drupal files, hoping that this wouldn't be necessary at some point. After this, I hope I've explained my case thoroughly enough.

I'm currently busy migrating to Drupal 5.7, but when I'm done I'll get back to this with patches.

Teemu Mäntylä
Webmaster of HIIT - Helsinki Institute for Information Technology - www.hiit.fi

pwolanin’s picture

pwolanin commented

@mantyla - I understand both your description of the problem and the logic. However, I'm still puzzled since I don't see the bug manifesting.

On my local host, Drupal 7.x, with PHP 5.2.5, Apache 2.2.8 it is set to: arg_separator.input &

With clean URLs enabled I can go to a page like http://localhost/admin/reports/dblog?page=8 and it works as expected.

If I change this to: http://localhost/admin/reports/dblog?foo=bar?page=8

it fails (as expected) and works again when changed to:

http://localhost/admin/reports/dblog?foo=bar&page=8

or when changed to:

http://localhost/admin/reports/dblog&foo=bar&page=8

mantyla’s picture

mantyla commented

I think I have figured it out. I did a few tests, and suddenly after making a clean install of Drupal 6.2 the problem disappeared. The reason was that my RewriteRule was just a little different after all.

The RewriteRule I had was this:
RewriteRule ^(.*)$ index.php?q=$1

And the default RewriteRule is:
RewriteRule ^(.*)$ index.php?q=$1 [L,QSA]

The 'L' flag is irrelevant, but 'QSA' makes the difference. What it does is to preserve any existing arguments after the rewrite, thereby changing the second '?' from my scenario into '&'. I'd forgotten I'd ever changed this.

Having understood this, I don't think this should be considered a bug at all. It is certainly annoying to diagnose - who would suspect anything was wrong with the RewriteRule when the apparent problem is with the pager? It can still hit people like myself who, for whatever reason, remove the 'QSA' flag not knowing why it is there. I kept having the problem throughout the 4.7 branch since I always kept my old .htaccess file unchanged.

Now, if the code is kept as is, it looks better since with clean URLs the only argument in the path is preceded by a '?', and as far as the user sees, this follows PHP standard. On the other hand, this makes the 'QSA' flag mandatory, though it might be needed for other reasons as well.

Perhaps this should be called a case of missing documentation. A notice about the importance of the 'QSA' flag might be added someplace, such as here: http://drupal.org/node/256410.

Who should I turn to about updating that page? Or should I just leave a comment there?

mantyla’s picture

mantyla commented
Priority: Normal » Minor

Never mind - I already found out about the documentation issue reports. I posted this issue there: http://drupal.org/node/275470.

pwolanin’s picture

pwolanin commented
Status: Postponed (maintainer needs more info) » Closed (works as designed)

ok, I just updated that page.