Make sure TestSitePath exist before creating .htkey

       // Consumed by drupal_valid_test_ua() before settings.php is loaded.
+      file_prepare_directory($key_file_directory);

file_prepare_directory() cannot be used, because it requires a fully booted Drupal environment.

The comment in the line directly above clearly states that Settings and settings.php is not available yet.

Curious, could you repeat + redo your steps to reproduce, but remove Drush entirely from the equation, please?

Unless your installation profile causes the regular Drupal installation procedure to prematurely end, Drush is really the only aspect here that could cause a misbehavior. There are many similar problems with Drush on D8, so that wouldn't be a surprise.

To eliminate other variables, I was able to follow @jrbeeman's steps to reproduce on Ubuntu 14.04 with a standard, current AMP stack. So the problem seems dependent neither on any of the environment details listed in the summary nor on Drush.

fwiw in #2272879: Can not run a WebTestBase or KernelTest base tests from inside a simpletest the .htkey file is written without checking for the existing of the directory.

@jrbeeman what I think @sun is trying to say is calling file_prepare_directory() assumes a fully bootstrapped Drupal which is normally the case but simpletest runs its tests in a 'child site'.

You could try to test from the commandline like

cd ~/Sites/drupal/d8/www/core/scripts
php run-tests.sh --list
# choose a group name like simpletest.
php run-tests.sh --url http://drupal.d8 simpletest

Hope that helps?

This is at needs work.

I'm also seeing this issue with a custom install profile. My error message is slightly different, but seems to be the same basic idea. I am able to run other tests locally (such as Drupal\standard\Tests\StandardTest) without issues.

System details:
OSX 10.10.3
Drupal 8.0.x-beta10

file_put_contents(/Users/Swan/Sites/locals/d8-profile.local/sites/simpletest/353164/.htkey): failed to open stream: Permission denied

file_put_contents('/Users/Swan/Sites/locals/d8-profile.local/sites/simpletest/353164/.htkey', '7pvNs8HEcBkyRzYErDvMnFkGciDQmemOHVRxDohKBGwTTCfUC3ZlE2a3BSzK3yJOfi198Y6bOw')
drupal_generate_test_ua('simpletest353164')
Drupal\Core\Test\EventSubscriber\HttpRequestSubscriber->onBeforeSendRequest(Object, 'before')
GuzzleHttp\Event\Emitter->emit('before', Object)
GuzzleHttp\RequestFsm->__invoke(Object)
GuzzleHttp\Client->send(Object)
GuzzleHttp\Client->get('http://updates.drupal.org/release-history/drupal/8.x?site_key=dn7hyUzY-tTG5jQ1vzEXRoImFx7Z-LIkO43kdrkLaOQ&version=8.0.0-beta10&list=block%2Cblock_content%2Cbreakpoint%2Cckeditor%2Cconfig%2Ccontextual%2Cdblog%2Ceditor%2Centity_reference%2Cfield%2Cfield_ui%2Cfile%2Cfilter%2Cimage%2Clink%2Cmenu_link_content%2Cmenu_ui%2Cnode%2Coptions%2Cpage_cache%2Cpath%2Cquickedit%2Crdf%2Cshortcut%2Csystem%2Ctext%2Ctoolbar%2Cupdate%2Cuser%2Cviews%2Cviews_ui%2Cclassy%2Cseven%2Cbartik', Array)
Drupal\update\UpdateFetcher->fetchProjectData(Array, 'dn7hyUzY-tTG5jQ1vzEXRoImFx7Z-LIkO43kdrkLaOQ')
Drupal\update\UpdateProcessor->processFetchTask(Array)
Drupal\update\UpdateProcessor->fetchData()
update_fetch_data()
update_cron()
call_user_func_array('update_cron', Array)
Drupal\Core\Extension\ModuleHandler->invoke('update', 'cron')
Drupal\Core\Cron->invokeCronHandlers()
Drupal\Core\Cron->run()
install_finished(Array)
install_run_task(Array, Array)
install_run_tasks(Array)
install_drupal(Object, Array)
Drupal\simpletest\WebTestBase->setUp()
Drupal\simpletest\TestBase->run()
_simpletest_batch_operation(Array, '9', Array)
call_user_func_array('_simpletest_batch_operation', Array)
_batch_process()
_batch_do()
_batch_page(Object)
Drupal\system\Controller\BatchController->batchPage(Object)
call_user_func_array(Array, Array)
Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object, 1)
Symfony\Component\HttpKernel\HttpKernel->handle(Object, 1, 1)
Drupal\Core\StackMiddleware\Session->handle(Object, 1, 1)
Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object, 1, 1)
Drupal\page_cache\StackMiddleware\PageCache->pass(Object, 1, 1)
Drupal\page_cache\StackMiddleware\PageCache->handle(Object, 1, 1)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object, 1, 1)
Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object, 1, 1)
Stack\StackedHttpKernel->handle(Object, 1, 1)
Drupal\Core\DrupalKernel->handle(Object)

Looking into this a bit more. When I copy the "Standard" install profile, and rename everything to "Standard2", and run the test, and everything comes up green. So that's curious.

What I've found from investigating my install profile is that adding the Update Manager module (update) to the dependencies causes the exception in the test (even when the test has nothing to do with updates).

FWIW I am also running into this issue on an install profile. The tests run and once the simpletest browser is clicking around the website, a http request on the target page/thread is made which triggers the exact same permission denied error.

Possible solution without bootstrapping?

Patch works as advertised, probably needs a test though.

Re-rolled for 8.2.x

This is no longer an issue with BrowserTestBase as it sets the sites directory to be 777 after the installer completes. Test to demo the issue with the 777 commented out, but I think this can be closed.

This is no longer an issue with BrowserTestBase as it sets the sites directory to be 777 after the installer completes

This issue still crops up using #2793445: Allow BTB to test an existing, already installed Drupal site instead of installing from scratch but you won't notice it unless you're sending PHP errors from the requests back to the tests :)

+++ b/core/includes/bootstrap.inc
@@ -681,6 +682,7 @@ function drupal_generate_test_ua($prefix) {
+      chmod($key_file_directory, 0775);

I actually had to make this mkdir($key_file_directory, 0775);

A proper fix should probably be solved in the BTB issue but here it is for now anyway.

The problem is when the tests are run on already installed site the TestSitePath doesn't exist so this can cause an issue. See #2793445-86: Allow BTB to test an existing, already installed Drupal site instead of installing from scratch or https://gitlab.com/weitzman/drupal-test-traits/merge_requests/36/diffs#3... so this is not a bug but a task.

Hmmm.... so the use case is to run tests against an existing database. We're not installing a site all right? So the files directory that's being used is the same as the site too?

I'm worried because in \Drupal\Core\DrupalKernel::findSitePath() we use

    // Check for a simpletest override.
    if ($test_prefix = drupal_valid_test_ua()) {
      $test_db = new TestDatabase($test_prefix);
      return $test_db->getTestSitePath();
    }

to get the site path. Even if there should be a separate test path I'm not sure here is the right place to create it. Atm the directory is created in \Drupal\Core\Test\FunctionalTestSetupTrait::prepareEnvironment() and I'm not sure we should change that.

We're not installing a site all right?

Yes, that is right.

So the files directory that's being used is the same as the site too?

This is a really good question. If we go by this logic then we should need to update \Drupal\Core\Test\TestDatabase to accommodate that case.

Even if there should be a separate test path I'm not sure here is the right place to create it.

I think if we accommodate the case that testing DB is default DB in \Drupal\Core\Test\TestDatabase we can address that.

I'm +1 to make \Drupal\Core\Test\TestDatabase more usable when site is already installed. What do you think?

I don't think we should change \Drupal\Core\Test\TestDatabase at all. But maybe we should make it easy to override.

Yeah, that makes sense.

Any ideas how to proceed?

It is tricky because the class is used both by the test runner and the site under test so it has to be overridable in both places. This is made even harder by the fact that in the site under test it is used before settings.php is loaded so overriding it there is not an option either. So we're going to need to be wrap drupal_valid_test_ua() to return one of these objects and to communicate the class to use in the SIMPLTEST_USER_AGENT.

Re-rolled to apply to 8.6.x.

In Drupal testing trait library, we have an open merge request https://gitlab.com/weitzman/drupal-test-traits/merge_requests/62 to fix this issue. Please try it or similar approach on your projects.

For me the patch was not sufficient because it was not able to create the sites/simpletest folder. So for that I created a merge request that creates the directory recursively. Not sure why the tests are failing though...

This issue is being reviewed by the kind folks in Slack, #needs-review-queue-initiative. We are working to keep the size of Needs Review queue [2700+ issues] to around 400 (1 month or less), following Review a patch or merge request as a guide.

MR should be updated for 10.1

Also IS should be updated as it references acquia dev desktop which support stopped in June 2021.

I've actually recently run into this issue, and I have some more details that affected me here I wanted to share...

The fix here still solves my problem, and I can explain why as well.

So I ran into this issue with the following scenario...
* I am installing a site that uses a custom cron job to make a call out to an RSS feed, download the file, and store it locally for processing by the feeds module.
* Drupal core runs through the INSTALL steps... but the install_configure_form step actually hardens the simpletest site directory, and sets the folder to 555 (not writeable).
* Drupal core at the end of the installation then triggers a cron. My job calls out to the internet. Drupal core has a custom testing middleware that intercepts that calls, and tries to add testing headers to the call, and ultimately calls drupal_generate_test_ua(), which tries to write the .htkey file to a hardened sites directory, and that fails... so my first cron run fails silently.

I actually came across this b/c by enabling the "locale" module, the call to drupal_generate_test_ua() was made earlier in the stack, before the hardening, and the .htkey file gets placed successfully, and there is no more failure.