By j.b on

I am working on a website which has around 150k users.
I have to create a csv file which contains the details of user (drupal fields and profile2 fields ) and export the csv to another server daily.

I have implemented by export function in a hook_cron and in the function i do a user_load function call and two profile2_load_by_user and some other custom mysql queries.

The problem which i have is that my export_function does not seem to terminate, it may be a timeout or a memory problem.

I have tested my function my limiting the number of users to around 1000 and it works fine.

I am wondering whether it is possible to export huge volume of data through hook_cron.

⋅ Categories: Drupal 7.x

Comments

coreyp_1’s picture

coreyp_1 commented

Anything is possible, it's just not always easy. :)

I had a similar problem in which I needed cron runs to last hours without timing out (transferring LARGE files to an S3-like provider).

My solution was similar to this one http://www.failover.co/blog/drupal-7-running-cron-command-line but with a few modifications.

I run my php cron from a perl script which can check to see if it is already running. I also added these two lines at the beginning, because of my long-running processes:

set_time_limit(0);
ini_set("mysqli.reconnect", "On");

I also made a few changes specific to my server needs, such as custom memory limits for php run from CLI.

A few suggestions: If you are needing to load a lot of info, but are only pulling a few values, don't use user_load, etc. They will use up too much memory and be too slow. It is better to pull all the info from a custom SQL query. If you set no timeout, do not use any load_ functions, and write to disk incrementally (as opposed to saving everything into one giant string and writing it at once) then it shouldn't matter if you have 1k or 1M records, it should just work.