Provide php 5.4 testing on testbots for D8 code without breaking everything else

This is probably the proper queue.

The main thing to consider is that testbots need to simultaneously support testing of D6, D7, and D8 tests right now ... and simply moving testbots to PHP 5.4 will break D6 and D7 testing.

Just pointing out that, despite the title, this isn't simply an upgrade task. :)

No, we currently test with a single version of php on the testbots. Testing with multiple versions will require qa/PIFR to determine what version to use with each test and send it over to the testbots, and the testbots will have to have the capability to choose. Or something like that.

It occurs to me that there's no reason the testbot *machines* should not have support for a variety of PHP environments, database environments, and even webserver environments, although that's never been called for.

@jthorson you probably already have a full plan for this, but here's my quick think from the testbot side (not PIFR):

1. Rework the testbot PHP config so that it uses php-fpm (Separate daemon running to do php work)
2. Install all needed PHP versions
3. Run php-fpm daemons with the versions required (Alternately start them as needed.)
4. PIFR specifies the php to use (and alternately database)
5. Use the appropriate php from the command line, and use an alternate vhost configured with the matching php.

If I'm not mistaken the exact same thing can be done even more easily with databases, since they don't conflict on ports or anything.

I had always thought of this problem as requiring exponentiation of testbots, but it's obviously not the case. This doesn't solve the problem of exponentiation of *tests* of course (If you want to know about PHP5.3 and PHP5.5 you have to test on both), but it does have give us a leg up.

If we fix up PIFR to be able to do this (or do environments already help?) let's make sure we can send both php version and database type/version.

Yeah, if we can get multiple versions of PHP to run simultaneously, we can add a PHP_Version test parameter quite easily, and just swap out which php_cli binary we call on the exec() calls.

Before this, however, we need to upgrade to wheezy, as there are no valid (trusted) repositories offering PHP 5.5 for debian squeeze.

To answer #3, we have a single PHP 5.4 testbot, with tests for Drupal 6, 7, and 8 that can be run on demand by anyone with PIFR admin rights on qa.d.o.

Wheezy isn't really the issue here. At least two PHP versions have to be built from source for this approach, because the pre-built packages conflict no matter what distro they're deployed on. There's nothing wrong with upgrading, but I disagree that it's a blocker.

Maybe this will help.. here's a recent guide put together by a former colleague of mine about compiling multiple PHP versions on debian: http://haza.github.io/install-php-fpm-apache/

I'm only suggesting we go with wheezy because, based on the google research earlier this year, it appears that everyone else in the debian world is following dotdeb's lead, and has abandoned any attempts at combining squeeze and PHP 5.5.

For those new to this thread, we've seen this coming for a while, and have even made a couple attempts to compile packages for the testbots earlier this year ... #1867192: Testbots need to run on 5.4, 5.5, 5.6 and 7 and #2057263: Test deploy of multiple custom PHP packages on a testbot.

...copying my comment from #1498574-176: [policy, no patch] Require PHP 5.4:

How many testbots do we have? I don't want to derail this, but wouldn't it be a good idea as a follow-up to leave a bot running on 5.3 (the one currently recommended for D7) and another on 5.2 (the one currently recommended for D6)? Then we could figure a way to send tests to these bots specifically (perhaps a token in the patch filename) in order to test php-version-specific issues. If there is no specific php version defined in the uploaded patch filename, then the test should default to run against the php version that is set as the branch's recommended.

Of course if we can get testbots to be able to run tests against multiple/various versions of php, that would be even better than having dedicated bots per php version. In order to save on resources we should stick on testing each patch against specific branch + the recommended php version for that branch by default. If we happen to suspect an issue is related to a specific version of php, then we should be able to override the default recommended version of php with any supported version for that branch.

...adding related issues.

Just a couple things:

1) https://github.com/CHH/php-build :-)

2) There's already a specification for which PHP version to use with which Drupal, and that's the minimum requirement here: https://drupal.org/requirements

So for D6 it's 4.4.0... I feel old. :-)

Also, with D8 the PHP version requirement could be gleaned from /composer.json, if someone were to put it there.

The minimum requirement and the recommended version are very different things. We should either test with the recommended version, or the very latest version we're hoping to support (i.e. Drupal 6 ought really to work fine with at least the oldest supported release of PHP which at the moment is 5.3, if we only had one version to test 8.x against it might as well be 5.5 as anything else).

OK, if the testbot runs 5.3 for D6, then change the requirements page, because no one cares how it runs under 4.4.0.

@Mile23: see #2140113: [Policy] Stop supporting PHP 4 where that's being discussed.

Unless I'm mistaken, this blocking us from #2152073: Bump Drupal core's PHP requirement to 5.4.2, which I think puts this into at least "major" priority, if not critical.

FYI, we have *HEAD* testing available at any time (not patch testing), and the results are available to all at https://qa.drupal.org/pifr/test/600303

Those who have pifr privs on qa, which should include all committers (and really anybody else who wants them) can click one of the tabs there and request a retest. Testing on 5.4 and 5.5 is also available for D6 and D7 (see the left sidebar).

We also have the ability now to run all the testbots on 5.4 and can change at any time. This requires some D7 bugfixes, as seen at the above link.

k, I didn't quite get what was being asked for here, but after talking to rfay and jthorson tonight, here it is in picture form. :)

Basically, until PHP 5.4 is passing on Drupal 6, 7, *and* 8, we can't move the PHP version. The links in the sidebar of qa.d.o will take you to the things that need troubleshooting. Instructions on getting started are at https://drupal.org/node/2144475.

We'll also put a general call to action about this and other issues in next week's This week in Drupal core.

PHP 5.4, 5.5, SQLite, and PostgreSQL environments are all available for on-demand testing of Drupal Core (D6, 7, and 8 HEAD).

AWESOME! :D

Aside, what is broken with sqlite3 and why ist postgre still missing?

Sqlite tests, but has failures. Postgres is also there on qa.d.o, but the D8 install on postgres fails, so there are no testing results. D7 on postgres does complete testing (with failures).

Is there an overview of how this is working somewhere? I ask because I'd love to contribute, but there's no point of entry to find out what's going on, since it's in about a hundred million different issues.

@Mile23 thanks! #5 is probably the most complete plan. I hope to get puppet going for php-fpm shortly. The other side of this is making PIFR handle it somehow (perhaps just environment) but somehow we're going to have to limit the resource cost of this (prefer a php version for a drupal version?)

So it seems like we could build different PHP versions with php-build and then symlink or otherwise call the proper binary within PIFR.

php-build is a nifty multi-version PHP tool used by the Travis-CI folks. "Builds PHP so that multiple versions can be used side by side." http://chh.github.com/php-build

The provisioning phase would build these binaries automatically.

We'd add a PHP version option in the test request from PIFT. PIFR then does some versioning magic and spawns the process. This would also allow re-running with different PHP versions, because they're all binaries on the same machine, or to ping two 'bots at the same time with different version requirements.

There could also be some architecture to put the PHP version(s) (and other testbot requirements) in a file at the root level of Drupal and other projects. Something akin to the .travis.yml file, but maybe .drupaltestbot.json. This would allow PIFT to be stupider, and put the onus on PIFR.

My $.02. Larger denominations available on request.

Putting a control file into the root of the repo for each branch a la .travis.yml makes a ton of sense. That lets us control very easily what version of Drupal we care about testing on what versions, and then scale the testbot servers however we need.

@Mile23 php-build sounds great. I'd hope to have packages, and not be building at provisioning time, but great stuff.

@Crell, I *love* the idea of running this via the actual project branch. That's delightful! It's possible we could just not tell PIFT *or* PIFR what version to test on and just let the testbot follow the instructions in the branch.

From #13

We should either test with the recommended version, or the very latest version we're hoping to support (i.e. ... if we only had one version to test 8.x against it might as well be 5.5 as anything else).

All things being equal it would make sense to test the recommended PHP version most thoroughly, but thinking about the changes that happen between PHP versions I think it's much more likely that someone will accidentally attempt to use a PHP 5.5 feature in D8 code than that they will accidentally attempt to use a feature that was removed in PHP 5.5. Therefore, I suggest our main test bots should be running the minimum version of PHP that we support.

It would be good to get test coverage on other versions of PHP. Perhaps we could do a daily test on HEAD on a range of versions?

#29: That describes our current state ... we currently have continuous testing on PHP 5.3.27 (Not quite the 'minimum', but comparable), and the links in the left column of qa.d.o can be used to test HEAD against PHP 5.4 and PHP 5.5.

I think the MVP is to get 7.x branches to use 5.3.27, and make 8.x branch default to the PHP 5.4 testbot(s). If we can get that, we can start putting 5.4 code into Drupal 8 without breaking everything else.

jthorson: How hard would it be to modify the test runner to check a file in the docroot to determine which branch/bot/environment(s) to test against, a la #26/#27?

Try this: #2155639: Design a .drupaltestbot.json file to be at the top level of project/branch/etc.

#31 @Crell:

How hard would it be? It would be as easy as strapping aluminum tire rims and spinners on a rusted-out 1974 Datsun pickup truck.

In other words, sometimes the extra bling simply isn't worth the investment.

Currently, PIFR uses a constant to store the PHP binary path. Once php-fpm is in place, we can convert this to a variable, and have PIFR set it based on the Drupal core version for the test (already available in our test paramters), as per #6 above.

To be clear, I'm not saying it's a bad idea. In the long run, having a file in the repository makes a lot of sense ... it's how every other modern CI system out there currently works. In fact, I'd love for our testing infrastructure to be able to re-use .travis.yml, phpci.yml, or other test definition formats.

But, like that rusted-out 1974 Datsun pickup, there comes a point where it's better to start limiting the amount of effort you put into keeping it running, and start turning your attention to saving up for a new vehicle. Because if you don't plan ahead, and insist on driving that truck into the ground until it quits, you're eventually going to find yourself with *no* transportation (and not able to get to the dealership to pick up a new one!)

While I agree with you, sometimes you need to replace the breaks now because the car's unsafe to drive otherwise. You may be planning to replace the car, but at least for the next 2 weeks you need breaks that won't kill you.

(Really, that happened to me with my last car; the garage wouldn't give me the car back to drive home until they replaced the breaks, even though my next stop was the dealership to start looking for a new car.)

Just an FYI to those interested: This issue is blocked by three core issues, https://drupal.org/project/issues/search?issue_tags=phpfpm-testbot

In D7 we have a single test that cannot run on php-fpm - it makes an assumption that is invalid.

In D8 we seem to have two bugs related to shutdown functions that show up in php-fpm.

I've done some work to see if we could use phpbrew or similar to switch between Apache modules. It is definitely not possible to load multiple php5_module modules at the same time.

A hack would be to use phpbrew switch <php-version> and service apache2 reloadat the beginning of each test. It would probably work. It's not pretty.

IMO it's important to get the 3 core issues above resolved, not least because two of the three are critical to Drupal's functionality on php-fpm systems.

I've gone about as far as I can go on this, so will leave it now until there's progress or something.

As @rfay said, this is postponed on:
#2157053: Ensure register_shutdown_function() works with php-fpm (blocks testbot php-fpm)
#2157061: Run tests on php-fpm
#2157017: D7: DrupalHTTPRequestTestCase breaks on php-fpm [testbot PHP 5.4 blocker]

And yes, FPM is the way to go(or possibly fast cgi), mod_php won't work with multiple versions of PHP.

Since @sun has been working away on the blocking issues, I want to mention where we're at on this.

I'm gone until late February, assuming I can find time to work on it then. For the next month I'm bike touring with my wife Nancy in Hawaii, without a computer. I'm way fine with anybody picking it up and running with it in the meantime.

The last work on this was done in the drupaltestbot-puppet project on the 20131211_php_fpm branch. It's been a long time, but I think we were close, and just the three issues were in the way.

As @rfay mentioned, I proposed one very dirty hack and one simple resolution for:

#2157053: Ensure register_shutdown_function() works with php-fpm (blocks testbot php-fpm)
#2157017: D7: DrupalHTTPRequestTestCase breaks on php-fpm [testbot PHP 5.4 blocker]

Both patches are passing against D8 and D7 respectively (on current testbots).

So if anyone wants to seriously help, the next best step is to run (just) the affected tests on a local machine that uses php-fpm. (I don't have that locally. [yet, sadly.])

→ If they pass, we can move forward and commit them.

Doing so will allow testbot maintainers to focus on the necessary testbot changes only. Multiple verifications from people on php-fpm/fastcgi for above patches certainly won't hurt, so I'd like to encourage you to test locally, even if someone else already said that he/she is going to do it.

Lastly, I want to make sure that #670454: Support HTTP Authorization in CGI environment is on everyone's radar:

That's a trivial backport for D6 to enable the $_SERVER['HTTP_AUTHORIZATION'] header processing for CGI environments. Without this patch, tests of contributed modules that are currently relying on that HTTP header information will blow up as soon as the testbot switches to php-fpm/FastCGI. Hence I've bumped that to critical as well.

That said, I provided the (literally identical) backport a long time ago already; it just needs... an RTBC status change. (Drupal core development is weird at times.)

Let's make sure that our testbot maintainers are able to focus on the testbots themselves.

It's actually super easy to set up a testbot as a vagrant machine, so people can easily test and work through this. https://drupal.org/node/2144475

1. Tagging with phpfpm-testbot, so that this master/parent issue appears in the tagged list of issues, too.

   → https://drupal.org/project/issues/search?issue_tags=phpfpm-testbot

2. Bumping to critical, because

   1. #2152073: Bump Drupal core's PHP requirement to 5.4.2 is required before release.
   2. #2028109-127: Convert hook_stream_wrappers() to tagged services. elaborately explains that there are major differences between stream wrappers in PHP 5.3 and PHP 5.4, and we're not able to clean up and fix Drupal core's Stream Wrapper API without bumping requirements to PHP 5.4. Required before beta.
   3. #2134513: [policy, no patch] Trait strategy and #2079797: Provide a trait for $this->t() and $this->formatPlural() are blocked on PHP 5.4. Required before beta.
   4. #2159845: Upgrade to PHPUnit 4 is blocked on PHP 5.4.

3. IMO, we should put a deadline on this effort; e.g.:

   If we do not manage to get testbots to run on multiple PHP versions until 2014-MM-DD, then we will go back to #2135197: Upgrade testbots to PHP 5.4.

   In other words: We're shooting for perfection here, which is fine, unless perfection blocks progress.

   We will survive if testbots are simply bumped from PHP 5.3 to PHP 5.4. Maintainers of pre-D8 code just need to be careful to not commit code that requires PHP 5.4. To my knowledge, there are only a few differences in the first place; the majority of changes in PHP 5.4 are new language features that are easy to spot. → Perfection of running tests on different PHP versions can still come later.

   To complete this proposal with a concrete deadline, March 1st (2014-03-01) would sound reasonable to me. That's ~30 days from now.

Thoughts?

Hm. No, I don't think so. If we were to move testbots to 5.4.x as a whole, we'd break D7 and D6 core testing. See https://qa.drupal.org/pifr/test/600308#tabset-tab-4 (d7) and https://qa.drupal.org/pifr/test/600313#tabset-tab-4 (d6).

Using this approach is an attempt to allow D8 to move to 5.4 independently so we don't also need to add "Fix all PHP 5.4-related test fails in D7 and D6 core" to the list of blockers here.

webchick: Well, we have a conundrum, then. Because IMO we can't even go to beta n D8 until at least a month after we support 5.4 for realsies, because we want to refactor some code to use traits instead of crufty utility base classes. That will take time, but will qualify as a BC break. So whatever we're going to do here has to be done quickly, because it's a beta-blocker-with-padding.

At a glance, it seems like fixing 3 core issues would be way easier than fixing N unknown core issues (plus likely N unknown compatibility issues as well in the top, say, 30 contrib modules). But if someone wants to take on "Make ALL the code PHP 5.4 compatible^&%!@" then obviously I wouldn't stand in the way.

One thing we can't do though is make it impossible to put out bug fix/security releases of the stable releases of Drupal core, which all of the 900K+ Drupal sites in existence are running, because we have no idea if their test harnesses are passing or not. That's simply non-negotiable.

If it means D8 doesn't end up being able to leverage traits, then that would certainly suck, but these issues have literally been open for months, promoted in TWIDC, etc. so I don't feel like placing a deadline on the testbot maintainers is fair here. The php-fpm initiative was started in the first place because the testbot maintainers are trying to help core developers out and not make them fix all the things, esp. in D6.

However, a question to the testbot maintainers: it seems like we have a sudden rash of core developers passionate about solving this issue for D8 (hooray!). What's the best way for them to help get access and involved in testbot scripts themselves, so instead of them demanding timelines from you, they can help make it happen directly?

#38 explains where we're at. The code is checked in on a branch. Anyone who does not fear puppet could work with it.

The automated testing docs explain how to develop using vagrant. One can use a Windows, macos, or Linux host to do this. It does not require a separate in cloud testbot.

http://Drupal.org/automated-testing

No magic.

#36 also explains that we could do this by swapping the Apache config for each test, and restart apache. That's ugly, but simple. And it does not require fixing the fpm problems just to make testing with multiple versions work.

So, there are a ton of referenced issues here, and the issue is postponed, but there's no clear explanation as to what's blocking it. So:

1. What's blocking this issue from the core side? (The 8.x issues tagged phpfpm-testbot are all closed.)
2. What's blocking this issue from the PIFR/PIFT/infra side?
3. Can we update the issue summary to clearly list both those things, and the next steps?

Thanks!

Hmm. I'd say there is now *one* fixed issue and 4 core issues not committed yet (one on 8.x). Two are RTBC.

What we need from the testbot side (assuming we'll do this with php-fpm) is for D7 and D8 to pass all tests under php-fpm.

The 8.x issue got reopened after I posted my comment. :) So we also need the 7.x issues fixed even to only do this for D8? @jthorson seemed to imply it might be possible to dedicate specific bots to D8 to avoid that, but maybe I misunderstood.

Did some work on #205969: drupal_http_request() assumes presence of Reason-Phrase in response Status-Line. Anyone care to review?

Should there be a testbot-blocker tag?

Re: 48
The option to dedicate specific bots to D8 is an alternative to the php-fpm option (though it requires the same work on the PIFT side of things).

The list of issues that need attention is:

https://drupal.org/project/issues/search/drupal?issue_tags=phpfpm-testbot

The "phpfpm-testbot" issue tag is the "PHP 5.4 Testbot Blocker" tag already; we do not need a new one.

We need some attention from D7 and D6 core committers/maintainers.

The additional benefit of fixing these issues is that we will make Drupal core 100% compatible with FastCGI/php-fpm.

There are a few different approaches we can use to tackle this ... if the phpfpm approach timelines aren't working, it should be feasible to dedicate a few bots to D8 core testing, and make these "PHP 5.4 only" testbots.

I'll be updating this issue with the actual tasks required to pull this off in the next couple days.

As an alternative, I'm wondering if we couldn't also just have multiple php version installed in parallel and switch the used command and apache module before we start a test? Wouldn't we have to do that for php-fpm anyway?

Yes, that would require that we manually compile different php versions and keep them in different directories, but that's not different for php-fpm is it?

What php-fpm allows is to use different PHP versions *at the same time*. We don't need that?

@Berdir, in #36 I experimented with switching PHP version between tests and using Apache as a module. It would work fine. It got no support at the time.

I was for PHP-fpm because it was shiny and I get distracted by shiny things. But Berdir's words rang very true.

What php-fpm allows is to use different PHP versions *at the same time*. We
don't need that?

The only thing I could think of is with mod_php we would have to restart apache on version switch where as fpm they aren't tied at the hip. Will that reload/restart cause any issues, session or other?

The PHP binary used to execute tests is already a constant in the PIFR code, and switching this to a variable should be rather trivial. Adding apache restarts to the mix would be a bit more complex; and thus comes with a higher risk of breaking things; when I'm trying to prioritize stability above everything else right now.

That isn't to say I don't support the approach ... there are a number of different approaches which may help solve the multi-php question. I don't really care which one we use in the end ... I'm happy to settle on "whichever is working first". To me, it feels like the easiest approach for *right now* is to dedicate some bots to 5.4, and only send D8 tests to those bots ... but that's solely based on the fact that I haven't been involved in the php-fpm work to date, and don't really know where it stands.

My plan at the moment is to add an environmental parameter to the testbots which can be used to indicate which versions of PHP they are able to support. After that will be adding a 'target-php-version' parameter to the test array, and populating it for each test arriving from Drupal.org. I see both of these as being required in any case, whether using php-fpm, dedicated bots, or multiple binaries and apache restarts.

Once that's in place, we can see where we are with any of these three alternatives, and move forward from there.

Progress report:
Branches now have a 'test.php.version' parameter being passed to qa.d.o on all tests, with a value of '53' for D6/7 tests, and a value of '54' for D8 tests.

This will be used as a key within PIFR ... it should appear within the 'plugin_argument' array for all tests.

Wow ... I think I may have stumbled on an elegant hack solution which requires a very minimal amount of code changes, and will support dedicated PHP 5.4 environments without too much disruption.

This will need some testing, but I believe it will:

• allow all existing tests to continue to test normally on the existing bots,
• allow us to uniquely flag a PHP 5.4 environment by adding a 'phpversion' plugin argument to the existing environment configuration (no new config fields needed!),
• support dedicated PHP 5.4 testbots *AND* php-fpm testbots (which would just have multiple environments they can support), and
• prevent a test from being queued against the 5.4 environment AND a legacy 5.3 environment; ensuring each test is only ran once before we get results.

We'll need to do a couple days of thorough testing before we can roll anything out, but I wanted to provide another update here (along with the patch).

Premature celebration. "=" versus "==", every time.

Looks good. Can't really comment on the change, because I'm not familiar with the server environment/plugin part of PIFR.

The only remark I have is to perhaps use "5.3" instead of "53". For one, because that enables you to use version_compare(). And also, because I could very well see us moving from minor versions to patch/point versions in the future, at which point "5327" would look a bit insane ;)

Closely related, I just filed: #2204253: Output PHP version of testbot environment in review log

Testing uncovered that I missed an array index.

Here's the new patch.

Testing confirmed this is working as desired.

The next step will be to get our testbotmaster puppet scripts to be able to build and maintain a php 5.4 bot OR a php 5.3 bot (I assume using environments?). At this point, I'd default it to 5.4 and only enable the current php pin if looking specifically for a php 5.3 environment.

Once that's done, we can:

i) create a PHP 5.4 environment entry (or repurpose the existing 'core only' environment entry),
ii) Add the "phpversion:5.4" as a plugin argument to the environment,
iii) Update 3/4 of the bots to PHP 5.4 (i.e. remove the php pin, but doing so in a way we can still manage them with puppet), and
iv) Move those bots over from the 'mysql' environment to the PHP 5.4 one.

@jthorson We've been able to do 5.4 and 5.5 for some time now, as you know, using the puppet build. Am I missing something? You set the fact in /etc/php_version or something like that, right? Or are you looking for something else?

Possibly ... in building the existing 5.4 and 5.5 testbots, I disabled puppet and manually adjusted the apt pinning; so there could be other capabilities I'm not aware of.

I think I understand puppet well enough now that I'll go take a look at lunch. :)

@jthorson, it's controlled by /etc/php_major_version, which can contain 5.3/5.4/5.5. If php_major_version doesn't exist, it defaults to 5.3, which of course is easy to change.

This is done with a plugin, which affects what facter reports (it adds a new fact which puppet then uses to set up the right one.)

https://twitter.com/jeremythorson/status/438570220081266688

Woot!

The puppet build change wasn't quite enough ... I had to manually trigger a few things. Here's the sequence:

i) disable the testbot
ii) Add /etc/php_major_version
iii) puppet agent --test
iv) apt-get update && apt-get upgrade
v) rm /etc/php5/conf.d/20-apc.ini (was causing conflicts in the next step)
vi) apt-get install php5
vii) Move the testbot to the php 5.4 mysql environment (on qa.d.o)
viii) enable the testbot

And ...

... deployed.

Thank you SO much, jthorson!!!!!! :D

I noticed these PHP warnings in every test review log:

PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php5/20100525/sqlite.so' - /usr/lib/php5/20100525/sqlite.so: cannot open shared object file: No such file or directory in Unknown on line 0

Does not appear to affect tests negatively right now.