Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.
The file systgem settings at admin/config/media/file-system need to be converted to use the new configuration system.
Tasks
- Create a default config file and add it to the module.
- Convert the admin UI in system.admin.inc to read to/write from the appropriate config.
- Convert any code that currently accesses the variables used to use the config system.
- Write an upgrade path from D7 -> D8 to migrate existing data to the new system and drop the appropriate variables.
This would be a good task for someone wanting to get up to speed on how the new config system works. Feel free to ping me on IRC if you need help.
Comment | File | Size | Author |
---|---|---|---|
#145 | 1496480-file-settings-cmi-145.patch | 56.41 KB | alexpott |
#145 | 143-145-interdiff.txt | 570 bytes | alexpott |
#142 | 1496480-file-settings-cmi-142.patch | 56.53 KB | Berdir |
#142 | 1496480-file-settings-cmi-142-interdiff.txt | 6.48 KB | Berdir |
#143 | 1496480-file-settings-cmi-143.patch | 56.53 KB | Berdir |
Comments
Comment #1
mrf CreditAttribution: mrf commentedComment #2
mrf CreditAttribution: mrf commentedI think I have all the pieces in place, but I'm running into some confusion on how to handle default values that are defined, or partially defined by function calls or other dependencies. I called these out with TODO's in the patch.
Remaining tasks:
Converting current calls throughout the rest of core for these variables
Upgrade path
Comment #3
mrf CreditAttribution: mrf commentedComment #4
mrf CreditAttribution: mrf commentedReminder based on discussion that any changeable defaults at install time can be set in the module init don't have to be done via the file unless static. Value will be available to change after that initial set method is called.
Comment #5
mrf CreditAttribution: mrf commentedGot a lot further along with this one. Still missing the upgrade path, but wanted to see what the testbot has to say about it so far.
Comment #7
mrf CreditAttribution: mrf commentedFixed the syntax errors, based on running test locally this has introduced quite a few failure point for other systems.
Comment #8
mrf CreditAttribution: mrf commented#7: system-filesystemform-cmi-1496480-7.patch queued for re-testing.
Comment #9
Anonymous (not verified) CreditAttribution: Anonymous commented<?
function file_default_scheme() {
- return variable_get('file_default_scheme', 'public');
+ $config = config('system.file_system');
+ return $config->get('file_default_scheme');
}
?>
can be written:
and:
this can be written:
i'm not sure if we have a consensus on chaining yet, but i guess we need one. i'm totally ok with this being decided elsewhere...
Comment #10
marcingy CreditAttribution: marcingy commented#7: system-filesystemform-cmi-1496480-7.patch queued for re-testing.
Comment #11
mrf CreditAttribution: mrf commentedTook another look at this and found an instance where we are looking for file system paths before the config table exists in system.install. I think these missing default values are what was causing the weird test results above.
Not sure what needs to happen to work around this.
Comment #13
gddI know there are places where we special-case install stuff. For instance, on the system performance form, there is config related to JS and CSS caching that is used in the theme system in the installer before the config table gets created. For instance in drupal_aggregate_css() we have
So we could do that, special case maintenance mode (which includes install time) and force it to the default value in that case.
Comment #14
kbasarab CreditAttribution: kbasarab commentedTest will fail as Installation process fails. But rerolls to had and the interdiff shows a config setting I made to avoid a PHP warning for a missing argument.
Comment #15
kbasarab CreditAttribution: kbasarab commentedThis will fail. Simply rerolling against HEAD and merging the conflicts of test files removing into their respective test files.
Comment #16
sunAll config system conversions are API changes, so tagging as such.
Comment #17
alexpottThe patch attached rerolls the patch from #11 as the patches in #14 and #15 have included changes that are not part of this patch.
Currently this patch has issues during upgrade caused due to using config before it's available - which I will hopefully resolve soon. Posting here as a work in progress and to save anyone else who is interested in doing this the reroll.
Additionally the code needs work to improve the comments around what happens during an install.
The patch converts following variables which are present on the file_system_settings form:
It additionally converts the following variables which are part of the file system but can not be configured through the UI.
I've removed the novice tag as the installer issues make this conversion quite tricky.
Comment #19
alexpottFixed the upgrade path failures.
Comment #20
sunWow, great work so far!
(and possibly elsewhere) Admittedly, we have no standard for this kind of text references to config objects + keys within yet, but in day-to-day issue communication, the syntax of:
module.object:key.subkey
(i.e., delimited by a colon)
...gained some "popularity." For now, I think I'd suggest to go with that. (A dot as delimiter definitely seems wrong to me.)
1) Hmmmm... there are some really really concerning user comments on intval()'s behavior on http://de.php.net/manual/en/function.intval.php
2) I think we should keep "chmod" as parent key name, since that clarifies that the value actually must be suitable for chmod().
The "system" part confused me a bit in the new function name; especially when I saw it called from elsewhere -- primarily, because there's a system.module. Would it make sense to use "os" instead? Or do we use "system" in other file functions already?
I almost fear that the old variable was supposed to be configurable on a very low-level (e.g., global $conf or some weird distro-level override). We might have to retain that.
This actually seems to be a File module setting? It doesn't look like it belongs into this conversion?
I LIKE! :)
AFAIK, these belong to File module, too?
btw, the repeated $config on the second line can be skipped. ->set() returns the config object :)
I think you meant
in all of these?
Comment #21
alexpott@sun thanks for the great feedback. Incorporated everything in patch attached.
Do some mode reading on
intval()
and I've replaced it withoctdec()
based on http://www.php.net/manual/en/function.chmod.php#99747Comment #22
alexpottDone some mode reading on
intval()
and I've replaced it withoctdec()
based on http://www.php.net/manual/en/function.chmod.php#99747Comment #23
alexpottFix installer language tests and .yml spacing.
Comment #24
alexpottIn light of #1653026: [META] Use properly typed values in module configuration here's a patch that validates how much nicer this would make our code. The reason this works without the application of a patch is none of the config keys "typed" are maintained by FAPI.
The new YAML File looks like this
as opposed to
Comment #25
disasm CreditAttribution: disasm commentedreroll
Comment #26
sunFor now, we need to go back to #23; i.e., string values.
It's probably easiest to reverse-apply the interdiff from #24; e.g.:
Comment #27
disasm CreditAttribution: disasm commentedI think I did this right. I applied 25, reverse applied 23-24-interdiff, and then diff'd out another patch. We'll see if it passes.
Comment #28
disasm CreditAttribution: disasm commentedComment #30
disasm CreditAttribution: disasm commentedThat patch came out a lot smaller than I expected, and it failed. I'm assuming something went wrong with my process. If I get a chance later this week (maybe office hours on Wednesday) I'll dig into it and see what happened. If someone has free time before then, by all means, have at it.
Comment #31
kbasarab CreditAttribution: kbasarab commentedHere's a test at it. I had a similar result to disasm at first. Started QA'ing from the patch and everything looked right. Realized that I had done my diff against the later version instead of 8.x HEAD. Size looks about right now. Interdiff does show an addition I added back for the verbose settings. When I kicked off tests locally the verbose functions weren't found which I traced back to them being removed in patch. You can see what I added back in the interdiff.
Comment #32
alexpottRerolled patch in #31 looks good - thanks kbasarab!
I think we need a YAML comment in here to explain that this should be php octal notation in a string format. Because if a user did this
directory: 0700
it would break in a very weird way.Compare - passing a string to octdec:
With - passing octal to octdec:
Comment #33
kbasarab CreditAttribution: kbasarab commentedAdded:
to line 2 of the yml.
Comment #34
alexpottLooks great... I can't rtbc this patch cause I wrote a lot of it... but I think its ready :)
Comment #35
Gábor HojtsyDon't think this belongs with D8MI(?).
Comment #36
aspilicious CreditAttribution: aspilicious commentedreviewed this
Comment #37
catchThis really needs a comment as to why we're using octdec() vs. intval() or (int). I would've just slapped an (int) or intval() on there too, so a summary of the php docs pages and/or links to them would help people looking at this.
We should move all magic globals that aren't configuration overrides out of the 'conf' namespace to somewhere else. But that can happen all at once, doesn't need to be here.
hmmm?
Comment #38
pfrenssenThis needs to be rerolled, incorporating the changes made by these issues which have been committed since the patch in #33:
Comment #39
pfrenssenI rerolled the patch against HEAD, converted the newly introduced variables mentioned in #38 and addressed the remarks in #37 (except the $conf issue).
Comment #41
pfrenssenComment #43
pfrenssenI just noticed a duplicate of this issue where a ton of work was done: #1799504: Convert system file related variables to CMI. Is it possible to give these people credit for their work on this issue as well?
Comment #44
pfrenssenInterdiff:
Comment #45
pfrenssenThere were some more, but the tests are still failing.
Comment #47
rvilar#45: 1496480-45-cmi-file_settings.patch queued for re-testing.
Comment #49
Cameron Tod CreditAttribution: Cameron Tod commentedHow about a reroll?
Comment #51
pfrenssenComment #52
Cameron Tod CreditAttribution: Cameron Tod commentedAnd again.
Comment #53
Cameron Tod CreditAttribution: Cameron Tod commentedTags lost, putting back.
Comment #55
Letharion CreditAttribution: Letharion commentedRerolling onto HEAD.
Comment #56
Letharion CreditAttribution: Letharion commentedWow, that was _not_ on HEAD. Sorry.
Comment #57
Letharion CreditAttribution: Letharion commentedNew re-roll.
Comment #59
vijaycs85Just updating hook_update_N
Comment #61
vijaycs85Re-rolling with update change again
Comment #62
vijaycs85Comment #64
vijaycs85#61: 1496480-61-cmi-file_settings.patch queued for re-testing.
Comment #66
vijaycs85Adding system.file.xml
Comment #68
vijaycs85"unable to apply patch"? re-rolling after git pull
Comment #69
vijaycs85Comment #71
Letharion CreditAttribution: Letharion commented*Sigh*. I've tried figuring out the first few errors we're seeing here, and I can't wrap my head around it. Looking back over the issue history, it's clear that a great deal of things have changed during this patches life.
Instead of trying to work backwards with the debugger, I will attempt to "start over" with small parts of the patch, so I can see more exactly what changes cause what fails.
Comment #72
Letharion CreditAttribution: Letharion commentedCutting scope down to just file_chmod_file and file_chmod_directory. I have odd, seemingly unrelated, test-fails locally. Trying to upload to see what happens.
Comment #74
Letharion CreditAttribution: Letharion commentedComment #75
Letharion CreditAttribution: Letharion commentedComment #77
vijaycs85Updating octal view specification as specified in http://www.yaml.org/refcard.html
Comment #79
vijaycs85#77: 1496480-77-cmi-file_settings.patch queued for re-testing.
Comment #81
adammaloneBased on http://drupal.org/node/1653026#comment-6311516 they symfony YAML parser converts octals to the decimal equivalent. Therefore perhaps it would either be better to store the permission in the config as a string or cast as an (int) within drupal_chmod.
If the number from the config file is cast to int - the tests pass. I might be missing something vital and trivialising the issue by a simple (int) cast though.
Comment #82
Letharion CreditAttribution: Letharion commentedWhat in the world? I was certain I fixed the conversion before I uploaded the patch. Here we go again.
@typhonius Isn't casting to int what we already do with octdec?
Comment #83
adammaloneLetharion: I may have misinterpreted PHP docs but octdec() appears to convert an octal to its decimal equivalent whereas casting it as (int) appears to take the octal and merely convert the type to int but leave the actual number as is.
I think one of the big differences between #82 and #77 is the permissions being stored as strings vs octal types in the config file. My comment about casting to int was related to permissions not being stored as strings so we're all good with that now.
Your patch seems to have taken care of drupal_chmod and relevant tests in the first instance - I think I'll start to have a look at expanding the patch to cover the scope of other file system settings.
Comment #84
vijaycs85nice to see it back in green again :). My only concern is calling octdec() in all places where we do config()->get().
Comment #85
vijaycs85Also, wouldn't be quicker if we split them separate as separate task? I made one for file_temporary_path at #1856752: Convert file_temporary_path to the new configuration system
Comment #86
adammaloneAnd here's one for file_public_path: http://drupal.org/node/1856766
Comment #87
BerdirNot sure that makes sense. Writing the actual code is just a part of what needs to be done, the issues also need to be RTBC'd and commited, and if you split it up, then you will need to re-roll them each time one is commited due to conflicts (upgrade path..).
Comment #88
vijaycs85I do agree with not to split, but the number of items are high and failing most cases. Regarding update_N, as it is in system, anyway we need to update when new variable convert. your thoughts?
Comment #89
vijaycs85Now I can see the 'Upgrade path' and I'm totally agree with @Berdir in #87. We need to get this issue committed so that we don't need to common stuff(like http://drupal.org/files/17-19-interdiff.txt) in individual patches.
Comment #90
adammaloneI've had a go at altering almost all of the file_public_path variable_get() calls.
Comment #92
Letharion CreditAttribution: Letharion commentedAs far as I can tell, the fails stem from the default value of the file_public_path is set in $conf prior to test installation in WebTestBase. We probably need to move those to new variable-names into $conf as well, but I fail to do so. I could just be doing it wrong, but my attempt with
$conf['system.file']['path.public'] = $this->public_files_directory;
does not make the value available from config().Comment #93
Letharion CreditAttribution: Letharion commentedVery short patch on top of #82 with debugging code relevant to figure out what's going wrong in #90/#92.
Comment #94
adammaloneThere are a couple of amendments in both TestBase.php and WebTestBase.php that I hoped would address that. I tried doing config()->save on the public files directory location in WebTestBase for example (whilst also leaving it in $conf to try and keep as much working and reduce errors)
When applying the patch locally the affected files pass tests although that's not indicative of a successful patch, rather, that things are roughly on the right track.
Admittedly it took forever for me to realise that to get successful results from config in, say, stream wrappers tests, the correct value for public files location should be saved earlier in the test procedure.
Comment #95
vijaycs85Trying to cover all the changes for this issue. Locally it is failing on Action looping, but it is not related to this patch (hopefully).
Comment #97
Letharion CreditAttribution: Letharion commentedAction looping was one of the things that didn't make any sense to me and I couldn't figure out, so that's why I tried starting from scratch.
Comment #98
vijaycs85Thanks Letharion. I just found this in my local testing:
It make sense that it's failing on this patch :) Looks like default permission for /files/ is not getting set proper?
Comment #99
vijaycs85Updating to get current core changes to start work on this patch.
Comment #101
ACF CreditAttribution: ACF commentedre-roll to update system.install.
Comment #103
vijaycs85Tried to test this locally and found that most of the failing test cases are unit Test Cases and the reason is config.factory is not available on PublicStream:
So this code fixed test cases, Updating this hard coded path to make sure this is the only issue for all test cases. if it is then we need to add a fix to make config.factory available in PublicStream::getDirectoryPath().
Comment #104
vijaycs85Comment #106
vijaycs85updating patch...
Comment #108
gddAs a heads up, there is talk of removing the public file path from CMI and moving it to the (soon to be) new $settings variable. See #1859110-13: Circular dependencies and hidden bugs with configuration overrides.
Comment #109
chx CreditAttribution: chx commentedYes, that's a great idea and absolutely necessary. As pointed out by catch currently the config directory is relative to the public files directory which makes storing the public files directory path inside ... inception.
Comment #110
BerdirIs it? I have not yet looked at this closely but if there really would be a dependency, this patch should fail to install/run tests, which is not the case?
Doesn't it just have the same default value logic? Here is the relevant piece from the config_get_config_directory() function
That said, I'm still +1 for putting the public file path into settings but what about private and temporary?
The advantage of having the public file path in settings.php is IMHO that we could then also simplify the config_directories structure and default to absolute = TRUE and remove the nested array structure. So that these two settings would by default overlap but then they'd be in the same file which would make it easier to change them.
Comment #111
chx CreditAttribution: chx commentedOK, so things do not actually break because we hardwired
conf_path() . '/files
intoconfig_get_config_directory
. I presumed it was usingvariable_get
. If it doesn't, that doesn't really invalidate the argument, it just makes the change from UI a little less devastating but still -- I argue for removing the public files path from the UI because all your uploads still break. So that's not useful. Also, about 99% of the cases the sites/X/files is just fine.As for private and temporary, it's a whole different bag of hurt. private does not really have a good default as you want to set it outside of docroot. temporary has a default but it's not always working. These two (needs fiddling, no good default) make me vote UI for these. I would put the public path there still with an explanation of why it can't be changed.
Comment #112
BerdirAs discussed in IRC, there is still a very high chance that users will break their site despite that.
Because after changing the path in the UI, you will move the files directory over the other new location that you configured. And *then* your site will break, because you will have moved the config directory too and the system won't find it anymore.
And that's why I think the absolute option needs to go.
If we have this in settings.php:
(or similar)
then it will be *much* more obvious to anyone who is changing file_path_public that it does in fact not affect the config path and they will either fix that as well or keep the config files where they are.
The next question is then if config should by default be in files but as discussed with @chx, that is a discussion for another day (Has implications on the installation, needs another writable directory).
Comment #113
chx CreditAttribution: chx commented$settings['config_directory']['active'] = 'sites/default/files/config/active_aserwesgrsewsef';
+10000000000000000000000000 to make the config directory DRUPAL_ROOT relative instead of the mess we have now.
Comment #114
Berdirupdated the patch, replaced path.public with a setting.
Still needs quite a lot of work, just a start.
- There are 25+ definitions of the default value in core, we should either write it into settings.php and require its existence or add a helper function
- Needs a section in settings.php which describes it, similar to other $settings.
- Upgrade path probably needs to write it?
- Fix the failing tests ;)
Comment #116
Berdir#114: file-configuration-cmi-settings-1496480-114.patch queued for re-testing.
Comment #118
BerdirOpened #1887750: Use path relative to DRUPAL_ROOT in configuration directories
Comment #119
sunWith putting the public files path in settings.php, I'm concerned about testing. Web tests in particular.
So far, pretty much everything in settings.php is not testable. And, settings.php force-overrides the environment in which tests are running.
We obviously don't want tests to use, clutter, and destroy the public files path of the parent-site/test-runner. So we'd have to find a discrete way for replacing settings in a web test with settings that are specific to the test environment.
In #1576322-29: page_cache_without_database doesn't return cached pages without the database, @chx explicitly denied writing of a custom settings.php for a test environment that would be used as a replacement for the actual/test-runner's settings.php, due to security reasons; i.e., file paths, authentication credentials, etc. could be exposed to the public.
Comment #120
BerdirHm, yes, I see. That's a problem.
The current patch sets the setting to files/simpletest/... which makes it available within the test class but page requests will just use settings.php, which does not specify it. Had a similar experience with settings.php overrides and tests in 7.x recently.
An idea that I had was that we could *somehow* support a settings.php within the file_storage, which would be a safe thing to write to. That said, I'm not sure how easy it would be to trick Drupal to look there :)
Comment #121
chx CreditAttribution: chx commentedIf we put the whole files url into the agent string instead of just the ID then we could use php storage to read settings.php
Comment #122
sunJust to make sure I understand what you guys are thinking of - do you mean something like this?
Comment #123
BerdirSomething along those lines. the problem is that php_storage needs the hash salt, which is defined in settings.php, so we can either only load the test settings.php additionally to the default one or have to pass through the complete file name (or at least the hashed name), so that it can be loaded directly.
Still, not sure if this is even possible, but I don't have any other ideas on how to make that work at the moment :)
And it is a blocker for using settings() for the public file path. I guess with #1887750: Use path relative to DRUPAL_ROOT in configuration directories, it is not actually required anymore, especially if we can move the config directory out of files.
So I'm not sure if we want to go back to the patch in #106 for the time being, and convert it to CMI to get rid of the variable_get() calls as a first step and look into making it a settings() thing later on... ?
Comment #124
gddI personally would prefer to do the straight conversion first, then look at the public file path issue as a followup. If we want we can leave the public file path as a variable and convert everything else, to make sure we don't forget about it later.
Comment #125
BerdirOk, let's see if we can get that working.
Straight re-roll of the patch from #106 to see if the installer and so on still work. There are some new variable_get() calls that I haven't look into yet.
Comment #127
BerdirFixed the php storage tests, unit test can not use the config system.
Not sure what's up with all those crazy exceptions, let's try again.
Comment #129
BerdirFixed DrupalKernelTest. Still hoping that I'll get to see the results even if there are some weird fatal errors remaining, but those look random to me and usually don't prevent from displaying the results...
Comment #131
xjmstdClass being used for an entity instead of an Entity, looks like?
Comment #132
gddWell, I couldn't reproduce any of these fails locally. Gonna give it another kick but not sure how much it will help :/
Comment #133
gdd#129: 1496480-cmi-file_settings-drupal-129.patch queued for re-testing.
Comment #135
alexpottMaybe the patch attached will fix the testbot errors...
I think we should go with heyrocker's suggestion from #124 and remove the public path conversion from this patch... especially since...
I think makes it moot...
Comment #136
alexpottComment #137
gddI've got a patch almost ready that does #124, just doing some testing. Will upload in a bit,
Comment #139
gddOK attached patch removes all conversions of the public file path to config, and reverts them back to variables. This is quite a bit of the patch, but if we can get the rest of the variables converted and done with, I think it makes sense. Lets see how badly I broke everything.
Comment #141
gddSomething is going wrong with the translations:// stream, spent a bit of time trying to figure out what but its it's too late.
Comment #142
BerdirSo there were
twothree bugs in WebTestBase.- The public file path was set to the translation directory. That caused the file field RSS failure.
- The path configuration was written before the modules were installed and then overwritten with the default configuration. So it worked within the test but blow up in the test page.
- The config overrides were wrong, as nested keys need to be an array, not using the . syntax.
Also converted some missing translation path settings and emptied the default path setting and made sure that it's set in locale.install.
Not yet sure what to do about the @todo in the installer but this should pass the tests I think.
EDIT: Forgot one bug, there were three.
Comment #143
BerdirThat set needs a save.
Comment #145
alexpottMissing a call to
parent::setUp()
in a test.Comment #146
alexpottDang forgot to change the status...
Comment #147
gddI think we are good to go here. Will be great to see this get in. I re-opened #1856766: Convert file_public_path to the new settings system (make it not configurable via UI or YAML) to track that progress. Thanks everyone who has worked on this throughout the long issue.
Comment #148
webchickCommitted and pushed to 8.x. Thanks!
Comment #149
hass CreditAttribution: hass commentedFollowup #1914040: Private file system path is "0" after installation
Comment #151
ianthomas_ukReopening since there is still a call to variable_get('stream_public_path', 'sites/default/files'); in core/modules/file/tests/file_test/lib/Drupal/file_test/DummyReadOnlyStreamWrapper.php
Comment #152
BerdirThat's just a test variable, let's open a new issue instead of re-opening an old issue with 150 comments.
Comment #153
ianthomas_ukOK, opened #2109833: Finish converting stream_public_path to CMI