customizable peak time max link checker add-in thing

i thought of running cron every 5 minutes but i think that would put a bit too much stress on our already groaning servers. LOL

as far as not seeing many new links, well, that's not the case with us.
long story short, we get a lot of new links added to pages on a consistent basis.

basically i was just trying to throttle the link checker during peak time and then use some off peak time to churn through the links instead of just sitting idle.
we're paying for the CPU, might as well use it, eh?

and the numbers in the vars are just starters for tweaking. i put in 500, not thinking that it would check 500, but just to make it max out the timer and give me a place to start with tweaks. i still have a time-out override, i just modify it a bit during off-peak hours. so far it's looking like it checks about 200 per cron with those settings. but my goal is to tweak those until i get it as high as possible without crashing cron. obviously, everyone would have to tweak it to fit their site/server/situation.

so far i haven't had cron fail once with my new code inserted. (except for the issues brought up in the time out thread here: http://drupal.org/node/739524 but that was happening before i inserted this code, so i don't think the new code is the cause of that)

well, it was failing with the original code.
and unless i'm reading my own code wrong, it still has the check in it to make sure it stops checking after half max execution time, i'm just changing the divisor from your default 1000 to something else which shortens the reported time it took to check the link, in effect giving it more time to check links.

now that i look at i should change it to instead of changing the 1000 divisor to changing the 2, so you would change half max execution time to 1/4 or 3/4 max execution time or whatever you deem best for your circumstance

modified to change the cron timeout fail stopper on line 147 (of the original .module file. line 171 of the .module after you insert the modified code)

  // --- set peak time vars (in 24hour time) //
  $startPeaktime = 6;
  $endPeakTime = 23;
  $check_links_max_per_cron_run_offPeak = 500;
  // --- set max execution multiplier to say after how much of the max time it will stop itself to prevent cron from failing
  //     for example: setting to 3/4 will stop the linkchecker after 3/4 of the max time has passed.
  //     use fractions or decimals, either should work
  $maxTimerMultiplier_offPeak = 3/4;
  $maxTimerMultiplier_default = 1/2;
  // --- //
  
  // get current time
  $currentTimeHour = date("H");
  // check if peak time
  $peaktime = (($startPeaktime<=$currentTimeHour)&&($currentTimeHour<=$endPeakTime)?true:false);
  $check_links_max_per_cron_run = ($peaktime?$max_execution_time:$check_links_max_per_cron_run_offPeak);
  $maxTimerMultiplier = ($peaktime?$maxTimerMultiplier_default:$maxTimerMultiplier_offPeak);
  //$check_links_max_per_cron_run = variable_get('linkchecker_check_links_max', 10);

  $check_links_interval = variable_get('linkchecker_check_links_interval', 2419200);
  $useragent = variable_get('linkchecker_check_useragent', 'Drupal (+http://drupal.org/)');
  watchdog('linkchecker', '$max_execution_time:'.$max_execution_time, array(), WATCHDOG_NOTICE);
  watchdog('linkchecker', '$check_links_max_per_cron_run:'.$check_links_max_per_cron_run, array(), WATCHDOG_NOTICE);
  // Get URLs for checking.
  $result = db_query_range("SELECT * FROM {linkchecker_links} WHERE last_checked < %d AND status = %d ORDER BY last_checked, lid ASC", time() - $check_links_interval, 1, 0, $check_links_max_per_cron_run);
  watchdog('linkchecker', 'num pulled from db to check:'.mysql_num_rows($result), array(), WATCHDOG_NOTICE);
  watchdog('linkchecker', 'peaktime?:'.($peaktime?'yes':'no'), array(), WATCHDOG_NOTICE);
  $linkcheckercountertally=0;
  while ($link = db_fetch_object($result)) {
    // Fetch URL.
    watchdog('linkchecker', 'checking: '.$link->url, array(), WATCHDOG_NOTICE);
    $response = drupal_http_request($link->url, array('User-Agent' => 'User-Agent: ' . $useragent), $link->method, NULL, 1);
    //watchdog('linkchecker', 'done checking: '.$link->url, array(), WATCHDOG_NOTICE);
    _linkchecker_status_handling($link, $response);    
    
    $linkcheckercountertally++;
    if ((timer_read('page') / 1000) > ($max_execution_time * $maxTimerMultiplier)) {
      watchdog('linkchecker', 'stop linkchecking, we hit the max: '.$linkcheckercountertally, array(), WATCHDOG_NOTICE);
      break; // Stop once we have used over half of the maximum execution time.
    }
  }
  watchdog('linkchecker', 'ran out of links to check', array(), WATCHDOG_NOTICE);
}

[i'm guessing that #6 is meant as a reply to my other thread about specific links failing cron http://drupal.org/node/739524 ? ]

FYI: this code addition worked great over the weekend. cron never failed.
checking 180+ links per cron during off-peak traffic times and tuning down to check the linkchecker default during peak times.

really, all it's doing is checking what hour of the day it is and if it is "off-peak" it raises the max links that can be checked and raises the amount of time it will take to check links before breaking and exiting cron.

our servers are highly loaded enough during the day that i don't want to chance running every 5-15 min during peak traffic times.

i guess i could just run cron every 15 min during non-peak hours, or i could do it this way... same end result i suppose.

i don't catch your meaning...

sorry to resurrect such an old thread, but i realize i never really answered your final question. and tbh, i really can't. we have many other modules that are triggered by cron and run fine during peak times, but after much extensive testing, using supercron (before it was abandoned) to specify cron module trigger order, etc, linkchecker was the only one that seemed to cause cron to crash.

using supercron, we put linkchecker as the last module that cron triggered, cron still crashed, but at least everything else ran successfully before the crash.

using the above peak time hack solved those crashes.

i've been using it for almost 2 years now in 6.x-2.4, and porting it now to 2.5 without any issues. now with supercron gone, we still have no crashes of cron.

just wanted to post this as an FYI for anyone else having issues or wanting to have the ability to limit linkchecker during specific times.

crash == cron hangs and never finishes, it's been a while, so i can't exactly remember exactly what happened

either way, i like having the ability to set limits during specific hours of the day.

looking back, i think the crashing was more related to this #156582: drupal_http_request() should support timeout setting and i had a different thread to try and debug the cron hanging that i was experiencing.

but, as i said, i like having the ability to set limits based on hour of the day. that's why i started this thread in the first place as a possible feature request and providing the code that could be the start of that feature addition, or allow people to do it themselves if desired.

i apologize for re-opening this thread, i just wanted to let people know that the peak time code that i provided earlier still works with the 6.x-2.5 version of linkchecker