There are a number of issues that exist with the way SimpleTest invokes tests and the environment surrounding the test. These issues have slowly manifested themselves over time and 65 line setup method the performs a series of tricks to clear particular variables and statics in order to simulate a clean environment. In addition the actually way in which tests are invoked varies between the webrunner and the run-tests.sh script. I will summarize the problems as goals.

Goals

  • Provide a shared test running API that can be invoked via batch API, script and makes use of the queue API.
  • Rework the test hardness in such a way that the environment is completely clean and all related issues due to left overs from the host environment are removed.
  • Maintain the current error reporting features and leave open the possibility for further debugging tools.

I have spent a fair amount of time thinking about this and have made several attempts at this in other issues, but usually resulted in overly large patches that were thus ignored. The hope with this issue is to come up with a plan and create a number of sub-issues branching out of this issue in order to accomplish the goals in bite size pieces.

Approaches
I have narrowed it down to three major approaches the latter two should provide a perfectly clean environment, but require a bit more processing time. The processing time should not be the major concern if it solves all our environment issues. I also have a plan for lowering the processing time more dramatically then this will increase it.

  • Continue on the current path and make use of the new drupal_static() API to clear all statics and hope for the best.
  • Re-bootstrap Drupal after every major step in the setup process.
    • cURL requests to a script that causes Drupal to bootstrap like index.php, but performs test operations instead.
    • Invoke Drupal bootstrap in-line after each major operation.

Approach 1
The first approach is by far the simplest, but is not much different then what we are currently doing. It should eliminate a few issues, but will not deal with globals like $user.

Approach 2
Using cURL to call a script should work perfectly and allow Drupal to boot in the prefixed environment as it does with requests made through drupalPost(). It also means that any fatal errors and such need to be recorded in such a way that they can be passed up the chain a requests or accessed by the upper most. In addition any timeouts that tests can set need to be applied to all processes.

Although it provides the perfectly clean environment it is the most complicated and may cause issues with debugging and error capturing.

Approach 3
This approach should provide the same result as the second, but I am not sure it will actually work nor that Drupal will even be happy bootstrapping multiple times in the same process. It would seem if require_once is used everywhere that at least that part is ok and any variables loaded should just override.

If the host environment has additional modules enabled the module files will be loaded when they normally would not in the test environment (if the modules are not enabled there). I am not sure if that is an issue to worry about as and code should be using module_exists() would should function properly.

This approach does not break the debugging and error catching code as it all happens in one process like it does not. It also does not require the timeouts to be applied to multiple processes.

Conclusion
I think approach 3 is preferable, but as I wrote I am not sure it will work. I will begin work on a number of the item that need to be completed regardless of which approach is taken, but in the mean time I would like to get others thoughts.

CommentFileSizeAuthor
#6 488182-test-runner.patch14.93 KBboombatower

Comments

damien tournoud’s picture

damien tournoud commented

Approach 3 is the same as approach 1 (ie. you already loaded files, you have initialized static variables, and PHP will not let you go back in time).

The only sane approach is 2.

chx’s picture

chx commented

The question is, if we have all statics and then we reset em, save $GLOBALS in the beginning of running a test and resetting that too, what else consists as a state of Drupal that needs cleaning given that the DB is already covered?

damien tournoud’s picture

damien tournoud commented

@chx: functions cannot be unloaded, constants cannot be undefined.

boombatower’s picture

boombatower commented

Even if we can save globals how can we load them for a basic state? anything from the base environment cannot be assumed to be the basic state.

#3 points out constants which is a final nail in the coffin for the other approaches. Looks like #2 it is, I was kinda thinking it would come down to that, but was hoping for the more solid evidence which has been provided.

I will attempt an attack plan and create a set of small, bite-size issues soon.

boombatower’s picture

boombatower commented
Assigned: Unassigned » boombatower

Initial thoughts on major API functions. I'll get started and see how this works out.

I think we should rename $test_id to $batch_id as it represents a set (or batch) of tests.

function simpletest_test_get_all();
function simpletest_test_queue_all(array $tests);
function simpletest_test_run_next($batch_id); // To be called in already booted environment.
function simpletest_result_get($batch_id);
function simpletest_result_format($type, $batch_id);
boombatower’s picture

boombatower commented
StatusFileSize
new 488182-test-runner.patch14.93 KB

Extremely early code, but I'm done for the night.

moshe weitzman’s picture

moshe weitzman commented

I have to agree that #2 is the cleanest solution. Damien is the master of error passing between tested and testing side. surely he can help here.

andypost’s picture

andypost
he/him
Primary language Russian
commented

Suppose reseting of php is possible in one process but require to dig into php internals. As I know some of this way persistent db-connections are work.

There's a big probability in memory leakage not only in php but environment also.

The most slower thing is install process itself, why not separate tests as requiring shared or clean environment, once drupal was installed correctly - just another key in getInfo() array.

moshe weitzman’s picture

moshe weitzman commented

We have that. tests that extend DrupalWebTestCase need clean. DrupalUnitTestCase or somesuch indicates non-clean. There are very few of those.

Status: Active » Closed (outdated)

Automatically closed because Drupal 7 security and bugfix support has ended as of 5 January 2025. If the issue verifiably applies to later versions, please reopen with details and update the version.