Parser gets OOM when parsing Drupal 6 source tree

subscribe. yarrrr

Let me know if this observation holds with the latest code. Late yesterday I committed a change to turn off the debug flag that had accidentally been turned on in one function. My observation is this seems to prevent the unbounded growth in memory usage, although the memory usage is still significant.

I had the same or a similar looking problem the latest. Maxing out 256Mb trying to parse civicrm code base.

<b>Fatal error</b>: Allowed memory size of 268435456 bytes exhausted (tried to allocate 35 bytes) in <b>/var/www/.../htdocs/sites/default/modules/api/parser.inc</b> on line <b>620</b><br />

Line 620 says $tokens = token_get_all($code); , which is a php function... so I assume this isn't (at least in my case) a grammar parser issue

Marked #819928: Out of memory even with 512MB duplicate

Please test the attached patch and report your results.

My test results are as follows:

Using the grammar parser ui module to invoke the parsing of the D7 includes and modules directories (in separate passes), the memory usage did not exceed 117.4 million bytes (mB) for the includes directory files and 92.5mB for the modules directory files. The starting memory usage was 19.9mB for the Drupal bootstrap. [The initial memory usage (i.e. to load the parser code but before the parsing begins) is 3.2mB when invoked from the command line without a Drupal bootstrap, thus providing 16mB of cushion.]

The execution time exceeds 60 seconds for the includes file and 120 seconds for the modules files.

Tested w/ grammar_parser_ui against the includes/ directory in D7:

w/o patch:
Max memory usage: 197MB

w/ patch:
Max memory usage: 51MB

So the patch definitely helps with memory usage. Still having a problem with timeouts though. I couldn't parse the modules/ directory without it timing out.

An interesting thing happens in the memory log, though: when parsing common.inc, the memory spikes during the build grammar phase, then levels off for the rest of the parsing:

====================
create reader for file ../includes/common.inc
Peak: 20,709,376 bytes
Curr: 20,447,232 bytes

====================
set snippet
Peak: 20,971,520 bytes
Curr: 20,709,376 bytes

====================
unset contents
Peak: 20,971,520 bytes
Curr: 20,709,376 bytes

====================
add token names
Peak: 20,971,520 bytes
Curr: 20,709,376 bytes

====================
build grammar
Peak: 50,331,648 bytes
Curr: 50,331,648 bytes

#7: gp_779528_1.patch queued for re-testing.

Wow, a very impressive improvement.

I loaded up API module with grammar_parser and then ran cron using

drush -d cron

which reports the ongoing memory usage after each job. The results, before and after, are attached. But before the patch, each API job in the queue added one or two or as much as 40MB to usage. After the patch, each job in the API queue, resulting in a memory-limit fatal on my machine (set to memory_limit=512M) at simpletest.module, with 506MB allocated.

With the patch applied, memory usage still grows gradually, but it's like a 500K per job. The entire cron run succeeds.

Note that there is still a memory leak somewhere, as you can see it steadily climb.

Note also that the peak memory usage of 142MB is certainly higher than we'd hope for this.

But this is such a massive improvement in memory usage that it's definitely a +1 from me.

Committed in next dev release. Thanks to everyone who reported test results.

Still experiencing Out of memory even with 1 million bytes

PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted (tried to allocate 16 bytes) in path/to/modules/grammar_parser/engine/grammar_parser.list.inc on line 65

I am using 7.x.1.x-dev of grammar parser. I have also applied the patch just to satisfy my curiosity.

Three questions:
- what are you parsing? Can you attach the memory.txt file?
- how are you invoking the parsing? (what other modules are involved, e.g. grammar_parser_ui or coder_upgrade?)
- what is the date of the dev version you are using?

I'm running into this too, even at 700M memory limit.

My memory.txt is attached. I get the problem with both API module & gpui module. I'm using the dev version of grammar_parser from 2011-May-06 and the 1.0 release of grammar_parser_ui, both patched for Drupal 6.

Please contact me privately if you'd like a copy of the last file attempted. I can't post it publicly, but can share confidentially.

Marking fixed as Matt's issue is not with Drupal 6 code, but a custom module.

Oh dear, I still have this just with API module. I don't have an unlimited amount of memory to use on the server, and it's burning through the 256+ that I've given to php-cli.

I

Is this a duplicate of #1383890: Parser fails in D8 (Symfony) code? Are you trying to parse code that has namespace constructs?

It's D6 with API module... so I doubt it.

So, are you saying it won't parse the code in the API module itself? If not, please be specific (or contact me in IRC with details).

It was trying to build an API site. It failed due to memory #fail, as this issue has. I ended up building the whole thing on another server so that it would have a GB or so to work with, then moved the database back. Sad panda.

So, again, if you are "trying to build an API site," then the issue is with what you are attempting to parse. Any chance you could provide some details on the latter?

The very basic API site that's trying to be indexed is in http://dl.dropbox.com/u/7350603/tmp/indexed_files.tgz

The dropbox files no longer exist. If you can provide another sample of files that hits the memory limit, then reference the same and reopen this.

As no concrete evidence has been submitted, changing status.

If trying to parse code with namespace and use syntax, please try the new 7.x-2.x branch. Parsing should be successful. However, functions used by other projects (API, Coder Upgrade) to interface with the parser have not been updated for structural changes in the grammar storage. See #1708334: Update functions used by other projects to interface with the parser. Also, the grammar_parser_lib make file has not been updated to grab the 2.x branch from this project.