Long running "thread/process" in drupal
I have a background process that needs to run periodically that imports a bunch of data into drupal nodes. At present, I do this by "posting" to one of my module's functions. The problem is that this process/thread may need to run for 10 minutes or so during a maintenance window and currently, it times out after 60 seconds. I get the following error and the execution halts:
Fatal error: Maximum execution time of 60 seconds exceeded in ...
I know about setting the time period to a larger value using the php settings in the .htaccess files (e.g. php_value max_execution_time 120), but I'm looking for a way to run a maintenance process that may run for minutes, if not an hour in some cases in the background and I'd hate to lock up other things because I set the timeout period too long. In addition, some hosting sites I've looked at have pretty strict controls over these values and don't let you override them.
So has anyone come up with a scheme for long running php/drupal threads/processes?
Shell access?
I know if you have shell access and your host allows/provides a cli php parser, you can do some things that way. That is how we run an import process we use.
solved it different way...
I solved it by adding a call to php's set_time_limit() function inside my loop that was importing records. I set it to a time period that I think the query/update will take + some slop. It updates the timeout each time through the loop.
It should work for me unless I find myself in an infinite loop or gobbling up memory at a high rate.