Problem/Motivation
Maximum upload size is always passing validation.
Values entered to "maximum upload size" must be able to be parsed by Bytes::toInt()
Values larger than the "max_post_size" setting in php.ini are allowed.
So are random strings like "banana" for that matter.
Proposed resolution
Write tests!!
This issue was opened as "values greater than php.ini's limit are allowed.
Having written tests, it's now clear that validation was fundamentally broken.
Add actual validating code to the validator rather than relying on Bytes::toInt() to not do that for us.
Remaining tasks
[ ] Fix validation
[ ] Deside how to handle vlaues larger than php.ini's max post size.
This is already being discussed in #3142397: Error case missing on file size limit
User interface changes
The maximum upload size field will be more strictly validated and present form errors on invalid data.
Comment | File | Size | Author |
---|---|---|---|
#86 | interdiff-870576-82-86.txt | 6.02 KB | phenaproxima |
#86 | 870576-86.patch | 9.67 KB | phenaproxima |
#82 | 870576-82.patch | 14.35 KB | ravi.shankar |
#81 | 870576-81.patch | 14.45 KB | Hardik_Patel_12 |
#71 | interdiff-69-71.txt | 11.05 KB | thursday_bw |
Comments
Comment #1
jablko CreditAttribution: jablko commentedMy personal thought is that,
My reasoning is that values which can't be parsed into a quantity of bytes are bad data. Values greater than php.ini max size may not be bad data - like when I enter "30M" although my php.ini max size is only 8M, I may actually want to enable files as large as 30M to be uploaded. In this case "30M" is good data and I need to be warned to fix my php.ini.
Refusing to save values greater than the php.ini max size is just a usability frustration? Like if Drupal is smarter than me and thinks I made a mistake, it should be helpful and suggest how I might resolve inconsistent Drupal and php.ini configuration, vs. tying my hands.
A different issue is that the php.ini max size could be mentioned more prominently than in the max size input description, because it affects the file field regardless of whether you configure a Drupal max size.
Maybe the max size could be mentioned in a heading or summary? near the top of the file field edit page?
Comment #2
yched CreditAttribution: yched commentedNot for field_ui.
Comment #3
yched CreditAttribution: yched commentedHowever, to ensure that the field only accepts positive integers, file.module could use field_ui's _element_validate_integer_positive() helper function.
Comment #4
aaron CreditAttribution: aaron commentedthanks, yched. can field_ui be disabled? or is it guaranteed to always be present?
Comment #7
yched CreditAttribution: yched commentedre #4 Field UI can be disabled - but then the form to change the file field definition is not available anymore and nothing can be changed.
Fields and instances definitions can still be updated through API functions, but no validation of the settings happens there (like is often the case in drupal...)
Comment #9
Niklas Fiekas CreditAttribution: Niklas Fiekas commentedMoving to 8.x - to be backported.
Adding Needs tests. The steps to reproduce should be converted to a test case. Note that the image field has the same problem.
The options are:
#type => 'number'
element. The user picture settings use this. That would be:However, the number element type only works with pure numbers. So we'd have to decide for KB, for example.
Comment #17
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedTo move along getting tests for this scenario, small help is that here is a link to an existing test that already does much of what would be
necessary. This tests that drupal observes and generatest the correct error when uploading a file that's too large.
https://git.drupalcode.org/project/drupal/-/blob/9.0.x/core/modules/file...
I think we can add a number field AND a drop down to select the unit KB, MB.
Like so:
Comment #18
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedAs soon as I start looking at testing, I start asking the questions.
We've talked about validation here, however form validation won't cut it. There are plenty of possible scenarios where this value can be set
through other means. A dev setting it directly via a
drush cset
reverting configuration etc.The system kind of handles that now, but the question of how much information to feedback is on the table. If you're importing configuration,
or setting it via code. Perhaps a warning should at least write to the the drupal log.
The form can validate too and present the user with a more friendly message.
The description of the creation field is OK, I think could do with a little improvement for those that don't understand what the php.ini max_upload_size is and the implications. We could dumb that description down and still get the message across.
Comment #19
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedCase in point, in the file module's own tests, there's the file creation trait.
Calling that in a functional test will bypass form validation and result in a false positive test.
Of course we could use the browserTestBase to edit the field configuration form directly.
That might be what needs to happen to test the form validation.
Comment #20
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedA patch that provides a basic demonstration of what I think we want to do here.
This is a webdriver test, so it's bulky for what it does and could be done as a kernel test at least.
But I think it's useful for clarity on direction to take this.. and get it tested.
Comment #21
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedremove duplicate comment
Comment #22
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedSide note, that patch is supposed to fail as it's testing for the desired outcome, not the outcome it actually gets.
Comment #25
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedComment #26
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedI whipped up this simple kernel test
Which suggests that the validation isn't being called in form submission. Because "bananas"
fails the test with
A non-numeric value encountered
Existing function currently:
And it is at least set as an #element_validate callback as suggested by the doc comment.
I'll get that test into a patch, probably tomorrow. Then some investigation is necessary into why the form allows "bananas" as valid input.
Comment #27
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedPatch ahoy.
Quick little kernel patch .. can replace the full web driver test from previous patch.
Comment #28
heddnI wonder if a Unit test is enough for this. Is there specific need for a Kernel?
Usually we would see some asserts. I think for the 100 mb one, we should assertEmpty on
$form_state->getErrors()
. And then on fail, assertNotEmpty.The following is sufficient:
$form_state = new FormState();
Comment #29
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedThanks. Yeah this is not intended as a final path. For now the test just demonstrates the bug.
The JavaScript tests demonstrates that no validation occurs. The kernel test demonstrates that there is a validation function.. but the form does not validate.
That basic test could be a unit test. I'd prefer a test somewhere between these two. I feel a good test needs to submit the form
Comment #30
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedUpdating issue summary.
Comment #31
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedComment #32
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedNew patch!!!
This one has replaced the Kernel test with a unit test. And also provides enhanced validation.
I haven't implemented anything about validating against php.ini as yet.
Comment #33
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedComment #34
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedbad patch path. new patch added
Comment #35
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedAaah.. New patch.. added some more test conditions for negative numbers.. and the fixed the code to validate those too.
Comment #37
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedThe validating against the php.ini max size really needs some consideration.
Each file field gets it's own max upload value, so in the scenarion that php lowered it's value, you'd only be wanred if you edited a field
(and you'd have to adjust the others)..
I don't think it should at all enforce that value to be lower that php's max.. but some feedback to make a user aware of the situation.
Plus there are other modules that upload files too. I like the cron idea in that you could then mark it as an issue on the status page of drupal..
I wonder about performance issues.
Comment #38
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedLooks as though a few other tests fail now.. Because the form fails validation those tests I imagine.
Comment #40
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedI love patches <insert sarcasm emoji here> ..
This patch will fix the first of those broken tests, and hopefully the others. My previous patch was dissallowing empty values as input.
Comment #41
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedComment #42
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedExpect there will be a suggestion to move that data to a data provider.. hmm.
Comment #43
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedComment #44
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedComment #45
heddnAre all these necessary? Can we just parse bytes and see what happens? I'd like to see a test of what happens in that case.
Include edge cases where the string starts or ends with a space.
Should be a data provider.
Comment #46
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedI've notice that test is a false positive too.
I need to add test for 'formError()->shouldNotBeCalled()' I must have lost it along the way.
Comment #47
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedThanks.. agreed I'm proceeding the data provider. from #45.3
I've added some edge cases as per $45.2 locally which showed up some issues (yay tests :) )
I'm having to make some decisions about what data to allow and not.. but suggestions are welcome upon review.
As for #45.1, I'm not quite sure I understand what you mean.
And to cover some more of these edge cases, I'm about to add more checks in that same location.
Comment #48
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedAnd here we go for another round of what will my patch look like today.
#45.2 I have added more edge case test data - and adjusted code to work with those edge cases. Nice catch @heddn
#45.3 I have moved it to a data provider.
You will see I've added yet another check that the data is alphnumeric.
My gut is telling me this is deserving of splitting the storageSettings and fileSettings forms into their own classes.
And this max upload filesize though small, still worthy of extraction.
Comment #50
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedComment #51
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedHad a crack at adding a max filesize units select box with bytes, kilobytes, megabytes, gigabytes, this means an additional field, and probably wrapping the size and units fields in a fieldset. That means merging them and validating the result, and I believe overriding the save method of FieldConfigEditForm.php, with a hook or replacing the form, or making that form handle this specific case. All of which gets complicated in an attempt to simplify things. Better eyes than mine may see a simpler solution.
The idea of splitting the fileSettings and storageSettings out involves refactoring FieldConfigEditForm too, so that worth it's own issue.
Energy spent moving the max_filesize validation to it's own class or service is where the value is.
Comment #52
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedAnd so we have it. Validation mechanics for max file upload field now in it's own helper class. It is much cleaner this way.
I'm also now considering stepping out to a wider view and approaching this from the idea of testing the FieldConfigEditForm
not just this validation part. As it stands now this unit test test the validator directly, as it did before but is was less obvious
when the validator was in the FileItem class.
I'm pondering creating a test that implements formInterface and submitting sample values to the form from there.
Really interested in some input. I've been lucky to have heaps of capacity recently to put time on this, but that well will run dry.
Comment #53
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedComment #54
jungleOne nitpick, Unused use statement.
Chunked file uploading is a topic you might be interested in -- how to proper validating max-size of files uploaded via chunking. So adding two related issues.
Comment #55
jungleContinue with #54
Prefer the following
Or at least, moving operations to the previous line. An example from core/lib/Drupal/Component/Gettext/PoStreamReader.php
[0-9] could be replaced by \d
Comment #56
jungleThink more about #55.1
if
should be called right after the$size = ...
, so that, if the$size
is empty, the 3 lines of code between$size = ...
andif
won't be executed. Or, no matter the$size
is empty or not, those 3 lines got executed.So proposing the following.
Comment #57
jungleAddressing #54, #55 and #56. Changing the target version to 9.1.x.
Comment #58
thursday_bw CreditAttribution: thursday_bw as a volunteer commented"around around around it goes, where it stops nobody knows"
@jungle thanks for those additions. Triggered some further refactoring for readability.
Late in the game to say it now but this validator is looking much less like a 'max filesize upload' setting validator, and more of a generic input validator for the Bytes::toInt() utility. Which means this validator doesn't belong in it's current directory/namespace but rather `Drupal/Component/Utility` or roll it into the `Drupal/Component/Bytes` class itself.
Remembering that the root of this issue is with the form relying on Bytes::toInt() to self validate but toInt() (almost?) always returns a number, and only false in edge cases. So it makes sense that we have ended up with a toInt() validator. (although this validator currently strictly accepts only strings, toInt() accepts integers too. It makes sense to add a public 'validate' input to `Bytes`.. Or modify toInt to return saner output, but that becomes a compatability issue.
Comment #59
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedSelf review, doh.
Rename from stripSpacesFromString to stripWhitespaceFromString
Comment #60
thursday_bw CreditAttribution: thursday_bw as a volunteer commented`tests/Drupal/Tests/Component/Utility/BytesTest.php`
This Bytes class is interesting. I think this test coverage is a little light. And it's uber flexible rather than strict, the `76MRandomStringThatShouldBeIgnoredByParseSize.` is of note.
Ok. So I don't want to put this form validation into Bytes. It's not appropriate because it's much more strict.
We can leave it here associated with FileItems.
Comment #61
thursday_bw CreditAttribution: thursday_bw as a volunteer commented* I started looking at moving this over to a kernel test and test the actual form submission.
* I then discovered `modules/field/tests/src/Functional/FormTest.php` inheriting from BrowserTestBase, which does generic testing for form confg and storage. Is also flagged to be refactored: #3077193: FormTest should be converted (as much as possible) to a kernel test
* There is also `web/core/modules/file/tests/file_test/src/Form/FileTestForm.php` this tests the File form itself, but not the config or storage forms.
I'm resistent to making this conversion now for two reasons.
1) To decide if to create a separate test for config, storage with unique names or should they form part of FormTest class.
2) Given there is a plan to refactor FormTest, I'm hesistant to refactor this into it now. When this unit tests gives the benefits we need
and we can make the required improvements the future after the FormTest conversion. #3077193: FormTest should be converted (as much as possible) to a kernel test
3) Perhaps this can stand alone as it's own Kernel test.
4) It's a valuable solution as it stands now, with the added validator and associated test.
* Let the FormTest refactor issue progress on it's own.
* Add a separate issue for "test coverage for File configration and storage settings forms".
* As there is another ticket also for chunked uploads, we can leave that in it's own ticket and close this off too.
Comment #62
jungleNW for #59 and the following
”Do not append variable name "$size" to the type declaration in a member variable comment“ 1 code standard violation to fix.
Missing
@param
sComment #63
pavnish CreditAttribution: pavnish at Material for Drupal India Association commentedworking on it
Comment #64
pavnish CreditAttribution: pavnish at Material for Drupal India Association commented@jungle Please review this patch All the changes #62 has been implemented.
Thanks
Pavnish
Comment #65
jungle>NW for #59 and the following
Thanks @pavnish. Missed addressing #59
One empty line before @param is expected. Furthermore, the expected comment format is:
So
should be re-written to one-line or
Comment #66
pavnish CreditAttribution: pavnish at Material for Drupal India Association commented@jungle Thanks for pointing addressed #59 .
Please review this patch.
Comment #67
pavnish CreditAttribution: pavnish at Material for Drupal India Association commented@jungle Thanks for pointing #59.
Please ignore #66
The #59 and #65 has been implemented
Please review this patch.
Comment #68
larowlanThis is looking good.
core/modules/file/src/Plugin/Field/FieldType/FileItemMaxFileSizeValidator
is the correct namespace for this new class. That's a plugin folder, where we typically store plugins. I see the merit in splitting this out into it's own class - but given it still relies on FormStateInterface as an argument, it's not completely generic. So I think perhaps we should move it back to where it was - unless we want to make it completely generic as a class in Drupal\Component\FileSystem, which would then require it to simply return a boolean if it were valid, and we'd need to use the existing validation method to call that, and set the error. Thoughts?I don't think we need to use a static property here. We can just use a regular variable and pass it along as an argument to the other methods.
we could
return FALSE
here and that would avoid the need for the$value
local variableThis is some awesome coverage!
Would it be possible to give each of these cases a name.
One of the unfortunate problems with test-providers that aren't named is that PHPUnit will say 'test case 3' failed, which is great if you have 3 cases. But with ~30 odd here, 'Test case 16 failed' means you have to count down to find which one that is.
If we give each of these a name (by making the array associative, with the names as keys) then it will say something like 'Test case "words with spaces" failed' which is much easier to debug.
Comment #69
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedThis patch addresses #68-3 and #68-4
Nice spotting on the unnecssary $valid var.
Coming up with descriptive names for each input in the data provider seemed tedious and unnecessary.
Instead transform the array to also use the input string as a label.
Sample failure output:
Textdox format
Reference to:
#68-1 I will ponder that further.
#68-2 The patch was passing the value around, and I was finding it becomming unwieldy where I was documenting the same variable description over and over.. The main reason I shifted it up to a property.. document it just once and improve readibility. What is the benefit to just passing a variable around versus just setting a property?
Comment #70
larowlanWhat is the benefit to just passing a variable around versus just setting a property?
static properties are a bit of an anti-pattern. Static methods shouldn't have state (because they're static) but we're simulating state with a static property. I don't, might be just me but it feels icky and I don't think its something we regularly do in core, other than for e.g. singletons like Settings. Happy to put that to others for their input.Comment #71
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedOkay. Definitely needs review now.
I attempted to implement something along the lines of #68-1
Which as it unfolded ended up being the same point I'd thought up earlier:
#58
The test is now incorrectly namespaced but I have no clue as to where it should go.
Comment #72
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedI see some doc comments are incorrec now.. but I must put this down and do other things :D
Comment #74
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedBlurgh, well that's really broken..
Comment #75
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedPatch's test succeeds in my local.
BytesValidator
not found.. hmm.Comment #76
quietone CreditAttribution: quietone as a volunteer commentedI was curious about the error so had a look, hope this is helpful and doesn't double up on work. BytesValidator was in the /lib not /core/lib.
Comment #77
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedI know how I did that. Rookie error.
Thanks @quietone for fixing it :fingers_crossed: waiting for the tests.
Comment #78
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedI'm sold on removing that static property.
I would prefer to remove the static from the methods too and instantiate the class rather than calling it statically or passing variables through a chain. Now that we call the validator class from the validation Method in fileItem rather than as an #element_validator we can.
Comment #79
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedComment #80
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedHave literally bean beating around the bush. And having a ball. Back in front of the computer now..
Refactored this back to not having that property as state, while keeping the static class.
Kicking off testbot.
Comment #81
Hardik_Patel_12 CreditAttribution: Hardik_Patel_12 at QED42 for Drupal India Association commentedLast patch failed to apply , follow a new patch.
Comment #82
ravi.shankar CreditAttribution: ravi.shankar at OpenSense Labs commentedFixed lint fails of patch #81.
Comment #83
Hardik_Patel_12 CreditAttribution: Hardik_Patel_12 at QED42 for Drupal India Association commentedComment #84
larowlan8 bytes is the odd one out here with the space. Is that intentional?
What does the trim do here that the preg_replace doesn't?
are the back-reference brackets () required here?
Bytes:toInt doesn't throw any Exceptions - should this be \Throwable or \Error instead?
these changes are out of scope here
the method is static, so we don't need the new call here, just BytesValidator::validateByteSizeString
these changes are out of scope too
love the work here, great coverage
we need to put this file back, I assume it was a bad re-roll that removed it
Comment #85
thursday_bw CreditAttribution: thursday_bw as a volunteer commentedI see there a some conflict markers in my patch..some.how?!?!
This latest patch has pulled in some of the wrong code.
Thanks for the review
Comment #86
phenaproximaTried to address @larowlan's feedback in #84:
I'm not sure if this still needs the "Needs tests" tag, so I'm leaving it for now.
Comment #88
catchThis has just been fixed in #2359675: File field's Maximum upload size always passes validation.