I understand that something similar happened in 2006 but don't know if this is the same problem.

I have my site at /var/www/mysite and I want to import just the "articles" directory of that. My pages are in files like 6R12.html or 11U2.html

Ideally I want the URLs to be http://mysite.com/articles/11U2.html so that it matches the old site, but the system imports them as http://mysite.com/articles//11U2.html - or to be more accurate http://mysite.com/articles/%2F11U2.html

Here is a fragment from an import:

Importing 'articles//11U2.html'
Found a value for 'subtitle' to save as a CCK field value
Found a value for 'factuality' to save as a CCK field value
Found a value for 'author' to save as a CCK field value
Found a value for 'pggnumber' to save as a CCK field value
Found a value for 'pggindex' to save as a CCK field value
Found a value for 'pggdate' to save as a CCK field value
This document (known internally as 'node/963' ) should now be accessible via aliases as both 'articles//11U2' and 'articles//11U2.html'

Is this a problem in my configuration, a bug, or does it always do this?

I appreciate that I can try setting up a mod_rewrite rule to change "/%2F" into just "/" but that would be a really ugly work around.

PS (I got CCK field import working by applying a PHP 5.3 patch. THANKS again for this module)

CommentFileSizeAuthor
#5 import_html-extra_slash_in_node_urls-934716-5.patch998 bytescolan

Comments

alexmc’s picture

alexmc commented

WORKAROUND

OK, I should have tested this first.

I was specifying mysitedir and subdir separately so that the import only did my subdir. If I skip the subdir and specify just the parent dir then it still goes into the subdir and imports the files, but gets the aliases right.

So - a bug, but a minor one

dman’s picture

dman commented

Sounds like an ongoing inconsistency I've hit a few times.
Generally it's just an issue of ensuring that the path to the import directory and the URL alias pattern it produces either both do or both don't have a trailing slash.
There was some validation that tried to correct it for you, can't recall exactly how it did it.
In any case it probably can/should be tidied up with a post-process fix. double slashes are always a mistake.

Yeah, some extra massaging of the parameters could be done. It doesn't cover all the ways in which someone could accidentally tell it to do the wrong thing. URL rewrites are scary, I think I may start outsourcing to PEAR::URL to do the calculations for me from now on. Too many ways to get it wrong with my current regexps and string gluing.

colan’s picture

colan
he/him
Primary language English
Location Toronto 🇨🇦
commented
Version: 6.x-1.x-dev » 7.x-1.x-dev
Issue summary: View changes

Still an issue in D7.

Marking #2003480: Double Slashes in "Clean" url/"Undo" an import as a duplicate of this issue.

colan’s picture

colan
he/him
Primary language English
Location Toronto 🇨🇦
commented

The problem with the workaround in #1 is that I run into #1744606: Batch process won't start if Subsection left blank. The good news is that I get to decide which bug I run into, this one or that one. ;)

colan’s picture

colan
he/him
Primary language English
Location Toronto 🇨🇦
commented
Status: Active » Needs review
StatusFileSize
new import_html-extra_slash_in_node_urls-934716-5.patch998 bytes

Looks like the problem is that the call to file_scan_directory() to get the list of files is passing a trailing slash. According to the API documentation, this shouldn't be happening:

$dir: The base directory or URI to scan, without trailing slash.

The emphasis there is mine.

So it came down to a one-liner where I changed the line by dropping the trailing slash:

$found_files = file_scan_directory($working_path, "/.*/");

to:

$found_files = file_scan_directory(rtrim($working_path, "/"), "/.*/");

See patch attached.

Perhaps the $working_path shouldn't have a trailing slash in the first place, and this should be fixed earlier in the process, but there's code that intentionally adds a trailing slash if it's not there. I'm therefore assuming that messing with that code will cause other problems.

dman’s picture

dman commented

Hm.
From the early days I's used a trailing slash to indicate that the path being passed around was a directory or not.
In URLs, it's necessary for a directory (if not naming index.html directly) to end with a / so that relative URLs work as expected.
The URL
http://example.com/about
is different from
http://example.com/about/
because of what happens when that page has a href="more.htm"

As *most* of the time the key value being passed around between functions is the path, it was useful to know if the path was a file or dir. I'd added the trailing slash to indicate that.

But if current file_scan_directory behaviour doesn't like it, then yep, we can slice it off at the last minute. So that patch looks helpful.