PHP version affects success of parsing

Reproducible. If the file has been recently touched, or if the job queue has been reset, running drush cron also triggers the segmentation fault. If the job queue hasn't been reset, or the file hasn't been touched, there's an expired parsing job in the {queue} table that keeps it from being parsed again. Which is how I knew which file it was, by the way.

This is crashing in grammar_parser, so moving over there.

This bit of code is enough to crash it:

// I'm not sure if it needs all these includes, but anyway...
module_load_include('inc', 'grammar_parser', 'engine/parser');
module_load_include('inc', 'grammar_parser', 'engine/reader');
module_load_include('inc', 'grammar_parser', 'engine/writer');
module_load_include('inc', 'grammar_parser', 'engine/editor');
module_load_include('inc', 'grammar_parser', 'engine/list');
module_load_include('inc', 'grammar_parser', 'engine/object');

  // You'll need to grab the attachment in the original report and substitute your path here...
  // Note that the API module normally does some \r \n replacements, but they didn't change the outcome
  // so I simplified and got rid of them.
 $source = file_get_contents('/opt/www/apitoindex/drupal-8.x/core/vendor/Symfony/Component/Routing/RouteCollection.php');
  
  $editor = PGPEditor::getInstance();
  $reader = $editor->getReader();
  $reader->setSnippet($source);
  $reader->addTokenNames();
  // Up to the previous line it's OK; crash happens in the next line.
  $reader->buildGrammar();

As I mentioned above, this crash causes a segmentation fault when I run cron on the command line. When I put the above code into a PHP-format block (eek!) on my test site, I got a WSOD. Normally I don't get WSOD, since I have xdebug running, and it will give me a stack trace for PHP problems, but this is a bit more drastic I guess.

Anyway... since I don't know much about the inner workings of grammar_parser, I'm moving it over to that issue queue in hopes that you know more than I do. :)

Meanwhile I will also try deleting parts of the offending file to see if I can narrow down where it is crashing. I had originally suspected some of the lines with backslashes in them, but there are plenty of other files that are parsing with similar lines in my repository without problems, so I don't think that's it. Anyway I'll report back shortly with a smaller test file if possible.

Here's a much smaller test file that also crashes the parser.

Hm. But why would it crash, and why on just that one file? There are a lot of other Symfony files (and also core 8.x Drupal database class files now) that have namespaces in them, and they don't cause crashes. Also, this crash is a segmentation fault, not just a mere PHP error... very odd!

Yes, it is just that namespace stuff. If I take out the

implements \IteratorAggregate

it parses OK.

Interestingly, if I leave that part in, but take a few of the lines from the method out, it also doesn't crash. Also, the other namespace stuff at the top of the file doesn't cause problems, and the rest of the Symfony and other Drupal code with namespaces doesn't cause crashes either. Just this one file.

Anyway... This is probably going to be impacting api.drupal.org soon if it isn't already (seg faults during the cron job are not a happy situation, although perhaps it is a quirk of my OS/etc. that it is segfaulting? I don't really know).

So do you think this can be fixed sometime soon, and/or do we need to recruit help in order to get it fixed?

Awesome! I'll test it out later today. Thanks!

With this patch applied, I cannot get the regular (no namespaces) tests for the API module to run with our sample code. For instance, the test "Branch and object creation" in api/tests/api_objects.test (which basically just parses the files in api/tests/sample and verifies they were all created correctly) crashes with a PHP error, indicating not everything was parsed. Also before it got to that PHP error, it failed a bunch of the tests that see if the class inheritance was found correctly.

So, I didn't get to the point of testing the namespaced code, since non-namespaced code isn't working.

By the way... We have 15 tests right now in the API project. But from a Grammar Parser perspective, they are pretty much the same as far as what they are parsing. That Branch and Object Creation test is fairly quick, and is probably a good one to adopt if you want to just have a reality check of "grammar parser still works with the API module".

I'm still getting the same test failures as with the previous patch applied. See #9.

I am running Grammar Parser and API module off of git, and then I applied the patch in #10. For API I am on branch 6.x, and for Grammar Parser, I seem to be on branch "master" (is that correct?).

With that setup, when I run the API module's tests, specifically the Branch and Object Creation test, quite a few of the test assertions fail. The ones that fail seem to be related to testing that the "class A extends/implements class/interface B" things are working.

Do I need to make any changes to the API module to use the new code? In other words, are there any changes to the output?

So I just tried applying the patch again (I had removed the patch), and reparsed the "sample" project that I have on my test API site (which parses the code in api/tests/sample).

It is not working correctly. For instance, we have in classes.php:

class Sample extends ClassNotDefinedHere implements SampleInterface {

But when I go to the page for this class, it doesn't show me the class hierarchy. That indicates to me that this information is not being provided any more by Grammar Parser (which is the same problem that happened in the test). The hierarchy is also missing from other classes in the sample.

The relevant code from api/parser.inc is:

  $editor = PGPEditor::getInstance();
  $reader = $editor->getReader();
  $reader->setSnippet($docblock['source']);
  $reader->addTokenNames();
  $reader->buildGrammar();
  $statements = $reader->getStatements();
... [pass this into a different function]

 $current = $statements->first();
  while ($current != NULL) {
    $statement = $current->data;

...
        elseif ($statement->type == T_CLASS || $statement->type == T_INTERFACE) {
          $docblock['extends'] = $statement->extends;
          $docblock['implements'] = $statement->implements;
        }

I haven't run this through a debugger, but based on the output I'm seeing, I would say that $statement->type is correct, but $statement->extends and $statement->implements have either changed or are not present.

Ah. Well that explains it. I will try that type of thing out. I'll need to make a change that is consistent with the old or new code I guess.

I am now running the 7.x branch of the API module, using Grammar Parser installed as a library... and I'd like to get back to this issue.

Basically, we need to be able to support namespaces in the API module eventually. The support I think I would ideally want from Grammar parser would be:

a) Support of use/namespace declarations - it should return those as individual statements in $reader->getStatements(), so that I can recognize that a file contains them, and act appropriately.

b) Obviously, not crash when encountering use/namespace declarations, or ad-hoc uses of namespaces in code (see comment #3 above for an example of a file that crashes the current, unpatched Grammar Parser, using the code snippet in comment #2 to parse it).

c) Do something appropriate in the PGPEditor::statementOperandToText() function if there is a namespace involved (or have another method that deals with the namespace?). I'm not sure what "appropriate" would be, but here are some examples of namespace-using code, and I would need to be able to tease apart the namespace and the class in these cases:
http://us.php.net/manual/en/language.namespaces.basics.php

I think that's all the API module would need... I've outlined what I want the API module to do with namespaces in the issue summary of this issue
#1507476: Support namespaces in API module
and I can't think of anything else I would want/need Grammar parser to do to support this.

So... Does the patch above do all of this? If so, I'll go back to testing it and see what I can figure out.

And by the way, I believe that the namespace and use declarations need to be at the top of files:
http://us.php.net/manual/en/language.namespaces.definition.php
http://us.php.net/manual/en/language.namespaces.importing.php

There is a *discouraged* allowance for multiple namespaces in a file, but in any case they have to be outside of any other PHP code (not embedded in any other code):
http://us.php.net/manual/en/language.namespaces.definitionmultiple.php

RE #19 - Which specific files trigger the problem change over time, as Symfony files checked into Drupal 8.x change. That is why I attached a few specific files to comments above, so we could have something definite to test against.

This issue is on Grammar Parser, so let's keep the discussion pertinent. I'll file a feature request on inheritance graphs on the API module though, good idea.

Thanks for the patch! I'll do some testing on this later in the week (possibly tomorrow).

I tested the patch in #23 a bit today.

For the first pass, I just patched Grammar Parser and didn't change the API module code (so it is not making use of any of the new functionality directly).

The good news is that at least in API version 7.x, cron/parsing is still fine (no syntax changes, and the output still looks the same, although I haven't yet run all the tests to make sure).

The bad news is that Grammar Parser is still crashing. I am consistently getting a segmentation fault when it tries to parse file:
core/vendor/symfony/http-foundation/Symfony/Component/HttpFoundation/HeaderBag.php
from the latest 8.x (I'm attaching the particular file it is trying to parse, since 8.x changes rapidly).

I will try excluding that directory and see what other files it still crashes on.

Another file that crashes with this patch applied:
core/vendor/symfony/dependency-injection/Symfony/Component/DependencyInjection/SimpleXMLElement.php
(attaching this file too)

Additional files that crash during parsing with the patch in #23 and the latest 8.x code base:

core/vendor/symfony/routing/Symfony/Component/Routing/RouteCollection.php
core/vendor/symfony/http-kernel/Symfony/Component/HttpKernel/Tests/KernelTest.php
core/vendor/symfony/yaml/Symfony/Component/Yaml/Tests/DumperTest.php

Note that after each crashing file I found, I disabled parsing for that whole directory. So there could be other files in the same directory that also crash and I wouldn't know about them.

And another note: in IRC solotandem told me that he has some code that can successfully parse the file noted in comment #25, so he's planning to try posting a new patch soon.

With the new patch, parsing still crashed for me with file:
drupal-8.x/core/vendor/symfony/http-foundation/Symfony/Component/HttpFoundation/HeaderBag.php
(comment #25).

The crash (which is a segmentation fault, not just a PHP error) is definitely happening in the call to $reader->buildGrammar(). Here is the code that the API module is using:

  $editor = PGPEditor::getInstance();
  $reader = $editor->getReader();
  $reader->setSnippet($docblock['source']);
  $reader->addTokenNames();
  $reader->buildGrammar();

$docblock['source'] here has been set to the entire contents of core/vendor/symfony/http-foundation/Symfony/Component/HttpFoundation/HeaderBag.php (the same file that was attached to comment #25 above).

Some settings that could be of interest:
Grammar Parser memory limit: 512M
PHP version: 5.3.10-1ubuntu3.2

Hm. It must be something to do with the version of PHP I am running or some setting, because you are able to run this on your machine, and I cannot.

In my previous tests in this original report, I had completely eliminated the API module -- did you ever get a segmentation fault in PHP at that time (see comment #2 above) running purely Grammar Parser code?

So... Following what I did in comment #2, I just did the following:

a) Created a block with the PHP input format, and the following code (with an opening &lt?php tag):

$library = libraries_load('grammar_parser');

// You'll need to grab the attachment in comment #25 and substitute your path here...
$source = file_get_contents('/opt/www/apitoindex/drupal-8.x/core/vendor/symfony/http-foundation/Symfony/Component/HttpFoundation/HeaderBag.php');
 
$editor = PGPEditor::getInstance();
$reader = $editor->getReader();
$reader->setSnippet($source);
$reader->addTokenNames();
// Up to the previous line it's OK; crash happens in the next line.
$reader->buildGrammar();

return "It is OK";

b) Enabled this block on a particular page.

c) If I move the "return" line before the buildGrammar call, the block is fine and shows "It is OK". If I have the block as shown here, I get a white screen of death. (In my cron runs, PHP was having a segmentation fault.)

Apparently you do not have the same problem, but it is not an API module issue (this test does not use the API module)... I guess it is some kind of an internal PHP bug with my version of PHP and not with your version of PHP? Very odd.

I'm on 5.3.10. It could be the PHP modules that I have installed -- I did a Google search and it looks like some of the PHP modules have memory issues that can cause the PHP process to crash. Anyway, it isn't really a Grammar Parser problem at this point. I might try to narrow it down, and I might try to disable some PHP modules temporarily to see if the crashing goes away.

Meanwhile, I can parse almost all of the D8 source code, and I have the 4 Symfony directories that crash on my machine excluded, so I should be able to try out the new grammar parser code and see how to integrate it into the API module in the best way. Thanks!

Good idea. I guess that's
#1704294: Add support for use and namespace statements and expressions
and I think we should just close this one as "fixed", since it's obviously a PHP bug and not a Grammar Parser bug that is causing crashes on my system.