Killer Feature Request: Schedule one portion one time, another at another time

I'm having some significant issues with Search as well; it's an area of Drupal that could benefit tremendously from some new engineering, I like the idea a lot though; a few options come to mind:

1. We could define sets of calls and let supercron.php start any set individually; or
2. We could just make it so that supercron.php can take a module= parameter (it does now but a changing safety token is present to avoid an outside user, say, call up search indexing repeatedly until the site dies; we could just publish a magic URL with a task-specific constant safety token though, and the whole URL could be listed in crontab as is even when the cron handler is disabled in SuperCron's main job list)

I like the first idea to some extent on a conceptual level, but #2 might be more realistic as there are few "clusters of cron tasks" that one could think of defining at this stage in Drupal 6 history (why do I have a feeling I will regret having said that ?) and implementing that would be a lot simpler. What do you think ?

Ok, we'll fast-track that; I could use that too.

Just to whet your appetite: the next project we'll likely publish is internally called SuperIndexer; we are looking at why indexing takes so long and needs to be redone so gratuitously often. It'll take a while though to get something out.

Looks like we are facing the same problems !

+1 subscribe
Looking for this feature so I can recommend it with boost.

It's in the CVS; pls. try it and let me know how it goes ! I've also added a configuration page to help users figure out the proper format for the crontab command lines. I think this could look a lot better though.

That would be great.

Elysia cron (http://drupal.org/project/elysia_cron) does that but lacks some supercron features (output capture).

Yup; I'll wrap in some ideas to get rid of that ugly new tab in there and post something, hopefully next week

Actually, I like #1, and I had the indexing scenario in mind; for really complex and recursive forms of indexing, bundles or sets of cronjobs could be very useful. For example, I might want to schedule a series of lucene tasks that have dependencies.

So, while I think better indexing, via SuperIndexer, is always good, I also think that the type of index (hence indexer) chosen will depend on the content and the use cases to be approached.

Yup; we are finishing work on SuperSearch* then merge all existing changes (including a cool self-updating view of timings and stats so you don`t have to reload the page) right after that is done.

* Check it out, it will resolve a lot of search indexing issues. For one it indexes much faster -- we get metrics of 11x to 17x indexing speed-ups. It also solves a fundamental flaw in search indexing that we discovered on customer sites. Search asks you to take a wild guess on something you should never have to think about: how many items should be indexed between cron runs ? That number is almost always wrong. Aim too low and indexing slows to a crawl. Aim too high and a huge problem occurs: as one instance of a cron run is finishing its indexing and overshooting the cron call period, another starts and messes up the first one's work, crashing in the best cases, or, in the very worst case, silently messing up your index. So SuperSearch does this: it asks you how long it should try to index content for (e.g., what's your cron call periodicity) and tries to fit as much work as possible within that period. On the off chance that it tries to index one node too many, the next cron call will get this message: hold your horses and come back in a minute, we`re not done here yet. So at the very worst you lose one cron call's worth of time where not as much work ends up being done, instead of wasting your full index.

(Plus it caches results, allows theming, has reordering and search-enrichment hooks, an SEO magic wand, and all sorts of goodies; we need testers and new ideas so if you care to pls join the effort)

Yup, here it is:

http://drupal.org/project/supersearch

We are still battling some strange drupal.org CVS gremlins, but the first package should be up shortly.

This feature is now implemented; pls. try 1.4-beta1 !