I've got a fairly straight-forward function in a module I'm developing that's running out of memory. It was working fine for quite a while in production, but as the number of website users grew it reached the point where it exhausted the memory available to it. It now consistently fails with errors like: "Fatal error: Out of memory (allocated 33292288) (tried to allocate 207 bytes)".
I'm in a shared hosting environment so I can't increase the memory limit. Nor do I think this is really a solution. The current limit is 90MB and this gets used up pretty quickly. We only have 300 users and not all of these are processed by the function. So I'm looking for some tips on managing memory a bit better.
I've reproduced the code below. It's edited a little. I've added comments so I can refer to sections. I've snipped some irrelevant stuff to keep it short. I've been using the memory_get_usage() function to help me monitor where memory is being used up. I've also got a test site with only a small number of users, on which the function does run to completion.
Section A initialises the function's main variables. At the completion of the execution of this section, some 17MB of memory is in use. This will include all the Drupal stuff as well as my module. So we're well within the memory limit at this point.
Section B is where the memory usage grows and eventually runs out. Very roughly, every five users processed seems to use up an extra 1MB of memory. 270 users can be safely processed. By 290, we run out of memory every time. As far as I can see, this is due to the step at B1, where a user's CiviCRM contact record is loaded and an array with selected user data from this contact record is returned to my function. However, the two variables that hold this data: $contact_data and $result go out of scope with each loop iteration and are set again in the next one. I imagine that any memory they were using could be free'd after each iteration. (Note: this is based on a Java/C++ idea of scope and garbage collection, not PHP - a language in which I'm not an expert.) Yet the memory usage just seems to grow.
Section C which build the actual HTML output, based on the processed data, adds hardly anything to the memory usage.
Questions:
1) Should PHP be garbage collecting (in any way)? It seems to me that when the memory is used up, a lot could then be free'd up as it's no longer required.
2) Is there a way I can force memory to be released after each iteration of the loop in Section B.
3) Can you see any memory leaks in my code?
4) Is there a better way to do this sort of processing? This function doesn't need to be fast, but it does need to use less memory.
Any suggestions much appreciated.
function _rdata_report($full) {
$output = '';
// A: initialisation - configure a set of arrays
$comps = _rdata_comps(); // Map of competition codes to names
$teams = _rego_teams(); // Map of team codes to names
$counts = array();
$players = array();
$counts['All'] = 0;
foreach ($comps as $comp => $name) {
$counts[$comp] = 0;
}
foreach ($teams as $team => $name) {
$counts[$team] = 0;
$players[$team] = array();
}
$uids = _rdata_uids(); // Returns an array of the user id's of all active users
// B: Load and process user data
foreach ($uids as $uid) {
if ($uid == 1) {
continue;
}
$contact_data = _rego_get_contact_data($uid); // B2: Loads user's contact data from CiviCRM (if any)
$result = _rego_validate($contact_data);
if ($result['valid']) {
$counts['All']++;
$counts[$contact_data['Competition']]++;
$counts[$contact_data['Team']]++;
$players[$contact_data['Team']][] = $contact_data['Full Name'];
}
}
// C: Produce output based on user data.
$output .= '<h3>';
$output .= 'Total Valid Registrations - ' . $counts['All'] . ' players';
$output .= '</h3>';
$output .= '<ul>';
foreach ($comps as $comp => $compName) {
$output .= '<li>' . $compName . ' (' . $counts[$comp] . ')';
$output .= '<ul>';
foreach ($teams as $team => $teamName) {
if (substr($team, 0, strlen($comp)) != $comp) {
continue;
}
$output .= '<li>' . $teamName . ' (' . $counts[$team] . ')';
if ($full && $players[$team]) {
$output .= '<ul>';
foreach ($players[$team] as $player) {
$output .= '<li>' . $player . '</li>';
}
$output .= '</ul>';
}
$output .= '</li>';
}
$output .= '</ul></li>';
}
$output .= '</ul>';
$output .= '<br />';
$output .= '<br />';
return $output;
}
Comments
B1 == B2
NB: Small typo above. I refer to "B1" in the text and "B2" in the code. They're meant to be the same thing.
It's tough to say without
It's tough to say without seeing all of the code, but here are a few things I'd try...
Is anything returning references that might be hanging around?
You might try unsetting the $result and $contact_data variables after each iteration.
Are the calculations anything that could be done with straight SQL rather than iterative PHP?
Try a variation that calls _rego_get_contact_data and _rego_validate but doesn't assign the results.
Try commenting out the if ($result['valid']) block.
http://www.trailheadinteractive.com
References
Can you explain "returning references" in a PHP context? Is that different to an ordinary return value? Some of the CRM stuff does use an & in some places - does that indicate a reference? (Apologies: still learning PHP.)
A function that returns a
A function that returns a reference is defined like this...
But looking at the php 5 docs, it looks like you have to call it like this...
So maybe that's not relevant.
http://www.trailheadinteractive.com
CiviCRM
But some of the underlying CiviCRM code uses this notation. So maybe it has a few references that are left lying around. I'll have a look at its code.
More CiviCRM
So the underlying API call I make is to crm_get_contact(). It is defined as
function &crm_get_contact(...). I was calling it using a normal$var = crm_get_contact();. However, changing this to$var =& crm_get_contact();did not make any difference.With the ampersand was what
With the ampersand was what I was thinking about. That would be returning a reference that may not be cleaned up immediately.
http://www.trailheadinteractive.com
Some results
OK, I tried the following:
- commenting out the if ($result['valid']) block
- unsetting the $result and $contact_data variables after each iteration
- calling _rego_get_contact_data and not setting the result
None of these made any significant difference. Memory is used up just as quickly and as much.
Which makes me think the call to
_rego_get_contact_data()is where the memory is being used up.Can you post that function?
Can you post that function?
http://www.trailheadinteractive.com
That function
So _rego_get_contact_data() essentially builds a keyed array of about 20 elements for a single user. There are two callouts to the CiviCRM API. One per user to crm_get_contact() in _rego_get_contact_data(), and one per custom field (of which there are 10) to crm_get_custom_field() in _rego_test_elt().
Can u call this function ..
after
unset( $contact );
and
unset( $field );
add the following line:
CRM_Core_DAO::freeResult( );
this is pretty destructive, but free's all the mysql resources associaed with civicrm (which i think is ok in your case). We use a pear library (DB_DataObject) which has a nasty habit of storing a lot of stuff in static arrays including memory expensive resources.
lobo
http://civicrm.org/
http://civicrm.org/blog/
http://wiki.civicrm.org/
http://lobostravel.blogspot.com/
We have a winner!
Thanks lobo, that was spot on. Memory usage stayed constant for the entire run of the script. I'll have a closer look as to where I need to put this, but it's fixed the memory problem in this script. Is there a "working with CiviCRM" FAQ or something that this is in? Or can be added to?
posting it on the civicrm forums and the solution will help ..
at http://forum.civicrm.org/ in the developer section. someone will find it someday :)
lobo
http://civicrm.org/
http://civicrm.org/blog/
http://wiki.civicrm.org/
http://lobostravel.blogspot.com/
Posted
See this CiviCRM forum topic.