drupal_move_uploaded_file() currently attempts to call move_uploaded_file() with an unresolved destination URI. If this fails it will fall back to passing a resolved URI. This is to support safe_mode and open_basedir.

If drupal_move_uploaded_file() is passed an unresolved URI it will perform a file copy followed by an unlink. This produces a considerable amount of I/O. If it is passed a resolved URI (a filepath) it will perform a file rename which produces negligible I/O.

The attached patch means that it always attempts to resolve the destination URI that is passed to move_uploaded_file(). If the URI does not resolve (it is a remote scheme) then move_uploaded_file() will fall back to copying and deleting. This will not cause write I/O because the scheme is remote.

I tested the performance changes of the patch by uploading an Ubuntu .iso file to a file field and measuring the I/O using

sudo iotop -Pao

Without the patch it takes 1390.60 MB of writes.
With the patch it takes 695.30 MB of writes (the file is only written to disk once).

On a loaded server the improvement would be even more. During the testing, my laptop had sufficient free RAM for the written file to be cached in RAM. This meant it could be written to the destination without having to be read. If there was no free RAM there would have to be 695.30 MB of reads.

So - this patch reduces the amount of I/O required for a file upload by 50 - 66 %.

There are already extensive tests for uploading files to local and remote URI schemes.

Issue fork drupal-1395524

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

droplet’s picture

+++ b/core/includes/file.incundefined
@@ -1601,20 +1606,9 @@ function file_save_upload($source, $validators = array(), $destination = FALSE,
+  $destination = drupal_realpath($uri) ?: $uri;

Is there any chance $destination is FALSE ? maybe add an IF check.

-5 days to next Drupal core point release.

jbrown’s picture

The ?: operator ensures that if drupal_realpath($uri) is FALSE then just $uri is returned.

jbrown’s picture

optimize-file-uploading.patch queued for re-testing.

jbrown’s picture

Title: drupal_move_uploaded_file() produces millions of times more I/O than necessary » drupal_move_uploaded_file() produces twice as much I/O than necessary on non-COW filesystems
FileSize
1.85 KB

Reroll.

Note: I retested the performance of this patch, but discovered that the optimization was already being performed. This was because I am now using Btrfs on my laptop and it has copy-on-write. Re-testing on Ext4, the patch performs in line with the results in the issue summary.

See also #1377740: file_unmanaged_move() should issue rename() where possible instead of copy() & unlink().

jbrown’s picture

#5: optimize-file-uploading.patch queued for re-testing.

Dave Reid’s picture

Issue tags: +sprint, +Media Initiative
jbrown’s picture

Issue tags: -sprint, -Media Initiative

#5: optimize-file-uploading.patch queued for re-testing.

jbrown’s picture

#5: optimize-file-uploading.patch queued for re-testing.

jbrown’s picture

Issue tags: +sprint, +Media Initiative

#5: optimize-file-uploading.patch queued for re-testing.

Damien Tournoud’s picture

Title: drupal_move_uploaded_file() produces twice as much I/O than necessary on non-COW filesystems » drupal_move_uploaded_file() produces twice as much I/O than necessary

The claim about COW is dubious, but the patch makes good sense.

Before committing here, could we open a bug report against PHP so that they include trying to resolve URIs first in their filesystem operations?

jbrown’s picture

jbrown’s picture

Issue tags: -sprint, -Media Initiative

#5: optimize-file-uploading.patch queued for re-testing.

jbrown’s picture

#5: optimize-file-uploading.patch queued for re-testing.

jbrown’s picture

#5: optimize-file-uploading.patch queued for re-testing.

jbrown’s picture

#5: optimize-file-uploading.patch queued for re-testing.

Status: Needs review » Needs work

The last submitted patch, optimize-file-uploading.patch, failed testing.

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

voleger’s picture

Issue tags: +Needs reroll
savkaviktor16@gmail.com’s picture

Status: Needs work » Needs review
Issue tags: -Needs reroll
FileSize
1.65 KB

re-rolled

MerryHamster’s picture

Version: 8.5.x-dev » 8.6.x-dev
MerryHamster’s picture

FileSize
1.64 KB

reroll for 8.6.x

voleger’s picture

Status: Needs review » Needs work

Needs code indentation fix.

MerryHamster’s picture

FileSize
1.65 KB

fixed #27

MerryHamster’s picture

Status: Needs work » Needs review
voleger’s picture

Should not this be fixed directly in moveUploadedFile method of file_system service?

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

larowlan’s picture

As voleger points out, this is now part of file system service

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

immaculatexavier’s picture

Rerolled #28 patch against 9.5.x
Attached Rerolled diff

Status: Needs review » Needs work

The last submitted patch, 40: 1395524-40.patch, failed testing. View results

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Ankit.Gupta’s picture

Status: Needs work » Needs review
Issue tags: -Needs reroll
FileSize
856 bytes

Reroll the patch #40 with Drupal 10.1.x

Status: Needs review » Needs work

The last submitted patch, 43: 1395524-43.patch, failed testing. View results

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

dimitriskr made their first commit to this issue’s fork.