during file uploads, string length of a file's destination path($file->filepath) is not checked. only filename(basename) is validated. since filepath is stored as varchar(255), it needs to be validated or have more db storage space. i dont know which way is the best but filepath validation will eliminate the need for basename validation.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Wim Leers’s picture

Version: 6.x-dev » 7.x-dev
ufku’s picture

Version: 7.x-dev » 6.x-dev

why should we postpone checking a string's length?

mr.baileys’s picture

Version: 6.x-dev » 7.x-dev

It's not postponing. Bugs are fixed against HEAD (7.x-dev) and then backported to previous versions.

drewish’s picture

humm... good point we might need to add another default validator that checks that the file path fits.

codecowboy’s picture

Priority: Normal » Critical

This should be revisited. We still have the 255 character limit on file.uri. The URI in d7 should in general be shorter than earlier versions, but this could still be an issue. Is there any validation in place?

codecowboy’s picture

based on http://www.boutell.com/newfaq/misc/urllength.html
and the fact that the first stable release of mysql 5 was 5.0.3 which supports varchars up to 2^16, postgres not caring about the length of the varchars, sqllite having a billion character limit to varchars by default, mssql supports massive nvarchars. Apache supports up to 8192 bytes in the http header fields and UTF-8 potentially using up to 3 bytes per character.. 2730 is the closest neighor in characters..

length => 2048 with a validate function?

codecowboy’s picture

Status: Active » Needs review
FileSize
3.73 KB

here a first go with a validation function and the schema change.

Status: Needs review » Needs work

The last submitted patch failed testing.

codecowboy’s picture

did some more poking around.. apparently http headers are only supposed to use ascii characters which means only 1 byte per character so the division by 3 for UTF-8 is unnecessary... I'll reroll with the system.install fixed tonight.

sun.core’s picture

+++ includes/common.inc	3 Oct 2009 04:02:16 -0000
@@ -1023,7 +1023,9 @@
-      E_RECOVERABLE_ERROR => 'Recoverable fatal error'
+      E_RECOVERABLE_ERROR => 'Recoverable fatal error',
+      E_DEPRECATED => 'Deprecated function',
+      E_USER_DEPRECATED => 'User deprecated function',

Seems like another patch got mixed in here.

+++ modules/system/system.install	3 Oct 2009 04:02:16 -0000
@@ -755,7 +755,7 @@
         'type' => 'varchar',
-        'length' => 255,
+        'length' => 2048,

I don't think this works.

Powered by Dreditor.

bellHead’s picture

Status: Needs work » Needs review
FileSize
2.07 KB

Rerolled the pertinent bits.

In a public file access model the practical limit is not what the servers can handle (which is configurable anyway), but what clients can handle. The entire IE family up to IE8 have a path length limit of 2048 characters.

Comments updated to reflect this rather than header lengths and encoding sizes as the practical basis for the limitation.

bellHead’s picture

Status: Needs review » Needs work

Bot didn't like it. Will update later today.

bellHead’s picture

Status: Needs work » Needs review
FileSize
2.68 KB

system_update_7040() had been used elsewhere in the meantime. Moved updates to the file table to update_fix_d7_requirements() in keeping with the comments in other system_update_* functions in system.install.

Key length limit for mysql 5.1 is 786 bytes, so the unique constraint is not longer enforcable on uri. Creation of this constraint has been removed from system.install and it is dropped in update_fix_d7_requirements().

bellHead vs bot Round 2 :)

catch’s picture

Status: Needs review » Needs work

I'm sure we have some queries on uri, so if the unique constraint must be removed, then a non-unique index must be added to replace it.

bellHead’s picture

Status: Needs work » Needs review
FileSize
3.21 KB

Index added for uri in all the relevant locations.

eojthebrave’s picture

Status: Needs review » Needs work

This seems like a good idea to me. Just a couple of minor code style issues.

+++ includes/file.inc	28 Jan 2010 18:13:37 -0000
@@ -1281,6 +1281,26 @@ function file_validate(stdClass &$file, 
+ * Check for uris longer than we can store in the database.

Just being picky here but can we change "uri" to "URI" in the documentation and user facing messages.

+++ includes/update.inc	28 Jan 2010 18:13:51 -0000
@@ -533,6 +533,13 @@ function update_fix_d7_requirements() {
+     * Increase the size of the file.uri column. Limit is dictated by the path length limit for IE, 2048 chars.

Documentation should wrap at 80 characters.

Powered by Dreditor.

mr.baileys’s picture

  1. We support all MySQL 5 versions including those < 5.0.3 for Drupal 7 (http://drupal.org/requirements#database). If we claim to support the full range of MySQL 5 versions, we cannot use varchars > 255 bytes in core, no matter how uncommon those minor versions are... A quick glance over core's schema's also didn't yield any varchars over 255 bytes.
  2. +  if (strlen($file->uri) > 2040) {
    +    $errors[] = t("The file's uri exceeds the 2040 character limit.");
    +  }
    

    Shouldn't we be checking against the limit (2048)? If not, we should document why these last 8 bytes are reserved.

bellHead’s picture

Status: Needs work » Needs review
FileSize
3.72 KB

MySQL 5.0.1 and 5.0.2 where alphas, 5.0.3 up to 5.0.14 where betas, it only went production at 5.0.15. I don't think it's too much of a leap to read an implied 'production version of' in the requirements.

I rerolled the patch from codecowboy's efforts, so I left the length check mismatches in place in case there was a reason behind them. There is another weird one in the same vein, filenames are checked at 240 not 255 without any explanation as well. My only guess is that the 240 is based on ideas about maximum filename lengths on Windows, even though I can't find any official support for the number. All the operating systems IIS5 runs on do 255 char filenames.

There doesn't seem to any reason to reserve the space in these fields - data is written into them without any kind of wrapper. So I've bumped both those length checks out to match the field lengths.

EDIT: I've discovered why space is reserved in URI, but no answers on filenames. Updates once I've figured out some more details.

The last submitted patch, filepath_2k_6.patch, failed testing.

catch’s picture

Opened #698902: Make MySQL 5.0.15 requirement official - seems like we might as well bump the version number a bit.

Patch looks fine apart from two things

+++ includes/file.inc	29 Jan 2010 09:44:06 -0000
@@ -1281,6 +1281,26 @@ function file_validate(stdClass &$file, 
+function file_validate_uri_length(stdClass $file) {

Somewhere there was an issue to removing type hinting for stdClass in case people want to do crazy things in contrib making entities real classes. However I can't see it now and not sure if it even went in.

+++ includes/file.inc	29 Jan 2010 09:44:06 -0000
@@ -1281,6 +1281,26 @@ function file_validate(stdClass &$file, 
+  if (strlen($file->uri) > 2048) {
+    $errors[] = t("The file's URI exceeds the 2040icharacter limit.");
+  }

vim typo?

Powered by Dreditor.

bellHead’s picture

Title: {file}_filepath character limitation(255) » {file}_uri and {file}_filename length limitations
Status: Needs review » Needs work

The patch for this issue has gotten deeply confused, and probably serves as an instructive example of what awaits those who try to pick up abandoned issues without doing all the homework.

We now know that ...

The issue was originally raised about truncation or insert failures of the paths in {files} under d6.

Under d7 the issue is somewhat different. The filepath field has been removed and a field called uri has been added. The value stored in uri may be shorter than what was stored in filepath since it is relative to the base of the Drupal installation rather than being an absolute path. It may also be longer since it now includes the filename and a pseudo-protocol 'public' or 'private'.

Truncating values (on a permissive mysql installation) under d7 is as much of a risk as under d6, and will lead to files not being retrievable in either case.

The externally imposed constraints are:

The answer of lengthening the field is not practical under d6 since it is required to support databases which have a size limit of 255 on char fields.

The IE family of browsers up to at least version 8 limit the length of the path portion (the bit between the host and the filename) of a URL to 2048 characters.

The practical limit on the filename is 255 characters imposed by Windows.

The 260 character fully qualified filename (path + name) limit on Windows does appear to apply to at least the earlier versions of IIS supported. It doesn't give an error, it just returns 404 for files that break the limit.

Since this last one is the most restrictive limit, and since by disrupting the local access to file it affects private and public storage equally this is the restriction that matters.

What this means for an answer:

- The lengths of paths for file storage for upload fields need to be checked when the fields are created, these should be limited at 220 so that there are least 40 characters for file naming.
- Filename length checks need to be done against the path and filename limit of 260

Thoughts?

I'll start rolling a patch to this effect later in the week.

catch’s picture

If it's II6 that's the lowest limitation, then I don't think we should worry about that for D7 - it's no longer supported by Microsoft, and there are two more recent releases, or that's what it looks like according to http://support.microsoft.com/lifecycle/?p1=3198

bellHead’s picture

I got the restrictive results on IIS5.1 (Windows XP), tests on IIS7 (Vista) seem ok, but I don't have a Server 2003 to test IIS6 on.

Support for XP is already over, Server 2003 will be supported for around three months after the planned release date of Drupal 7 according to
http://support.microsoft.com/lifecycle/search/default.aspx?sort=PN&alpha...
although this may differ by territory.

There are big hopes for faster uptake of D7 than of D6 - if IIS6 is as restrictive as IIS5 is this time window narrow enough that it can just be ignored?

Some test results from IIS6 would of course be very useful. If it's not as restrictive as IIS5.1 then there is no issue.

Test procedure is basically:

- Create a site which allows directory browsing (It's easier than all the copy and paste)
- Create a directory within that site with any name
- Create a text file in the directory with a 100-120 character name
- You should now be able to browse to the file and open it
- Rename the directory to a name 220 characters long
- You should be able to browse to the directory unless the path to the site is already very long
- If you can't browse to the directory or you get a 404 on opening the file IIS is unable to handle the filename
- You may also get at 414 error (Request URL too long) - I don't think this should happen on IIS6
- Hopefully you can still open the file :)

webchick’s picture

Unless someone can think of a really compelling reason not to do so, I think Drupal 7 not supporting IIS 6 is fine. IIS users make up only a very tiny fraction of our userbase, so they get the same rules applied on them as other minimal technologies, IMO. We don't go out of our way to support Opera 2 either.

It's also fairly consistent, since we also dropped support for older versions of other technologies, including PHP 4, and MySQL 4 and PostgreSQL 7 this release.

bellHead’s picture

Status: Needs work » Needs review
FileSize
4.83 KB

To work under IIS7 and up, and on MySQL 5.0.15 and up. 2048 characters for file path, 255 for the filename and 10 for the pseudo-protocol add up to the 2313 field length for URI.

sun’s picture

Status: Needs review » Needs work
+++ modules/system/system.install	14 Feb 2010 14:20:02 -0000
@@ -748,7 +748,7 @@ function system_schema() {
         'type' => 'varchar',
-        'length' => 255,
+        'length' => 2313,

@@ -2176,7 +2174,7 @@ function system_update_7034() {
         'type' => 'varchar',
-        'length' => 255,
+        'length' => 2313,

errr, varchar > 255? If this is possible at all, then this needs an inline comment, potentially pointing to online resources that prove this.

Powered by Dreditor.

mr.baileys’s picture

varchars > 255 is no problem, see earlier comments in this issue (the MySQL version requirement for D7 was changed as a result of this issue). I don't think this needs an inline comment, but it would probably be a good idea to include a test that exceeds the 255 char limit.

drewish’s picture

I really want to protest the removal of the unique key in #13. Its entire purpose is to prevent the same file from ending up with two file ids. If the unique constraint tops out at 768 bytes then I'd suggest using that as a maximum field length.

codecowboy’s picture

768 characters would be good enough for most cases. It's 3x what is currently available, and we're looking for the maximum theoretically functional length and the unique constraint on file.uri is an important part of the specification for files.

does http://abcdefghi/abcdefghi/abcdefghi/abcdefghi/abcdefghi/abcdefghi/abcde... look long enough for most use cases.. Thats a little over 700 characters...

I think this is one of those cases where protecting the data has to trump potential. At least we're aware of the feasible limits should mysql increase it's unique constraint length.

mr.baileys’s picture

Status: Needs work » Needs review
FileSize
4.23 KB

Re-rolled with the 768 limit imposed by MySQL maximum unique key size, which seems to be the lowest common denominator.

Status: Needs review » Needs work

The last submitted patch, filename_and_uri_length.patch, failed testing.

catch’s picture

Issue tags: +D7 upgrade path

tagging.

marcingy’s picture

After some digging I think I have come to a conclusion as to why this path fails to result in a successful install. The maximum size for a varchar that will serve as an index which uri does is 767 bytes under InnoDB. However as we use UTF-8 each character accounts for 3 bytes, so any varchar set to be greater than 255 will result in a maximum key size violation. See http://bugs.mysql.com/bug.php?id=4541 for more information on the issue.

So unless we are going to store some sort of hashed representation of the uri, remove the unquie index or convert uri to text the maximum length we have to play with is varchar(255).

catch’s picture

Priority: Critical » Normal

So either this is won't fix, or we need to validate the upload length if we're not already. Either way I'm downgrading this issue from critical, since it's just a normal bug.

marcingy’s picture

Status: Needs work » Closed (won't fix)

Setting to won't fix

aliceten’s picture

Issue summary: View changes

Well, you can use Long Path Tool, it works perfectly.

stefan.r’s picture

Version: 7.x-dev » 8.0.x-dev
Priority: Normal » Major
Status: Closed (won't fix) » Active
Issue tags: -D7 upgrade path

People still run into this:

https://drupal.stackexchange.com/questions/36760/overcoming-255-characte...
#2374529: Data too long for column uri
#193954: {file}_uri and {file}_filename length limitations
#70201: Aggregator: Feed URL Length Is A Limitation
#2386539: Varchar(255) too small for some long url

Maybe this was because of http://www.w3.org/2001/tag/doc/get7#myths?

The problem was that the unique constraint didn't allow for more than 255 characters. In utf8mb4 this is even 191 characters, so in #1314214: MySQL driver does not support full UTF-8 (emojis, asian symbols, mathematical symbols) we proposed to save a hash of the file URI and use a unique constraint on that instead. If we go ahead with that, it should allow us to save longer URIs...

stefan.r’s picture

Priority: Major » Normal

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.6 was released on August 1, 2018 and is the final bugfix release for the Drupal 8.5.x series. Drupal 8.5.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.6.0 on September 5, 2018. (Drupal 8.6.0-rc1 is available for testing.)

Bug reports should be targeted against the 8.6.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.8.x-dev

Drupal 8.6.x will not receive any further development aside from security fixes. Bug reports should be targeted against the 8.8.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.9.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.7 was released on June 3, 2020 and is the final full bugfix release for the Drupal 8.8.x series. Drupal 8.8.x will not receive any further development aside from security fixes. Sites should prepare to update to Drupal 8.9.0 or Drupal 9.0.0 for ongoing support.

Bug reports should be targeted against the 8.9.x-dev branch from now on, and new development or disruptive changes should be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.2.x-dev

Drupal 8 is end-of-life as of November 17, 2021. There will not be further changes made to Drupal 8. Bugfixes are now made to the 9.3.x and higher branches only. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.15 was released on June 1st, 2022 and is the final full bugfix release for the Drupal 9.3.x series. Drupal 9.3.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.4.x-dev branch from now on, and new development or disruptive changes should be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

quietone’s picture

This was a bugsmash group triage issue. It was discussed with lendude and myself.

There has been no discussion here for 8 years. Perhaps this has been fixed in the meantime?

Lendude thinks this may have been fixed by #1314214: MySQL driver does not support full UTF-8 (emojis, asian symbols, mathematical symbols) and that #2492171: Provide options to sanitize filenames (transliterate, lowercase, replace whitespace, etc) seems relevant.

Is this issue still relevant?

Since we need more information to move forward with this issue, I am setting the status to Postponed (maintainer needs more info). If we don't receive additional information to help with the issue, it may be closed after three months.

Thanks!

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.9 was released on December 7, 2022 and is the final full bugfix release for the Drupal 9.4.x series. Drupal 9.4.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.5.x-dev branch from now on, and new development or disruptive changes should be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

pameeela’s picture

Status: Postponed (maintainer needs more info) » Closed (outdated)
FileSize
73.19 KB

Seems it has been fixed, if I'm understanding correctly: