It seems that conversions fail erratically.
Cron was totally blocked for 5 weeks on one site because of conversions problems.
I spent hours to convert all un-converted files manually by re-running conversions: I had to re-upload some files that were apparently corrupted in the upload process and to reboot the server several times as failed conversions end up killing the converters.
Then, I could re-enable cron for some time.
Wiithout uploading any new file, cron went down agin. FF kept trying to convert already converted files, and already successful conversions failed during cron.
For example, cron gets exhausted by generating dozens of errors on each failed conversion - for a given file:
File ISA-95_2_revised-B2MML_alignement.xls, uri=bitcache://d971091aa43073c0d9ded8398bd0d740484bb2f9 was not converted to text/csv by executing a pipeline {localhost:8100}{mv} "[in_file]" "[in_file].xls"; {java} -jar {{/opt/jodconverter/lib/jodconverter-cli.jar}} -f csv "[in_file].xls"; {mv} "[in_file].xls" "[in_file]"; {mv} "[in_file].csv" "[out_file]".
Re-running cron on this file succeeds:
File ISA-95_2_revised-B2MML_alignement.xls, uri=bitcache://f8349f790d4fcbecadd4d1257a9f7ce0231a0053 was converted by executing a pipeline {localhost:8100}{mv} "[in_file]" "[in_file].xls"; {java} -jar {{/opt/jodconverter/lib/jodconverter-cli.jar}} -f csv "[in_file].xls"; {mv} "[in_file].xls" "[in_file]"; {mv} "[in_file].csv" "[out_file]".
thanks for help
Comments
Comment #1
jvieille commentedApparently, conversions fail for specific files. In my case, one XSL and one doc/docx file fail systematically in background conversion (they are successfully processed by the front-end "re-run" command).
FF seems trouble by this and keeps activating the conversion a dozen of times at cron time, thus crashing cron.
I don't know if the failure to convert might come from the converters (outside FF responsibiity) ot FF, but I would suggest that FF handles properly this kind of error: if a conversion appears to fail, instead of looping forever on trying to make it, flag the corresponding file and do not try to process again this conversion unless the user un-flags it.
Comment #2
jvieille commentedUpdate:
It seems that if the number of files processed per cron is limited, conversion stop failing.
- 10, 20 => OK
- 100, 0 (unlimited) => failure
Any explanation?
All files being converted and OK, unlimited processing shall not hang even if all files are checked
Why conversions start to occur on already converted files, while nothing happens with the lower setting?
Comment #3
jvieille commentedAny hint about this?
I was again hit by this misbehavior. Setting the maximum number of files per cron run to 10 seems to be appropriate, above is risky
Comment #4
johanneshahn commentedi have some problems with file_cron too.
i have about 6000 files and every file is about 70MB.
first problem was that the variable "cron_last" was never set, because
the cronjob breaks. then i get the problem
in the first sql selection.
.....
// Inspect recemtly updated files which were changed since
// an hour before last cron run.
$recent = 60*60;
$result = db_query("SELECT f.nid, f.v.....
the result was always all files from db cuz cron_last is zero.
My solution
first:
set cron_last before starting conversion or something else
....
if (FILE_CRON) {
$time = time();
// ++++++++++++ add this ++++++++++++
// first set cron_last before something goes wrong
$lastcronrun = (int)variable_get('cron_last', 0);
variable_set('cron_last', $time);
// ++++++++++++ add this ++++++++++++
// If PHP is not in 'safe mode', increase the maximum execution time:
// Setting time limit to zero stops feedapi cron processing.
//if (!ini_get('safe_mode
......
second:
set a limit with constant FILE_CRON_LIMIT_TOTAL
i think that select always all updated files isnt good,
because cron_last can be zero. and then file_convert
is starting always from the beginning of all files.
// Inspect recemtly updated files which were changed since
// an hour before last cron run.
$recent = 60*60;
//++++++ change this +++++
//added Johannes Hahn set max limit her to FILE_CRON_LIMIT_TOTAL
if(FILE_CRON_LIMIT_TOTAL > 0){
$result = db_query("SELECT f.nid, f.vid, f.uri, f.size, f.type AS filemime, n.title AS filename FROM {file_nodes} f INNER JOIN {node} n ON f.nid = n.nid WHERE n.changed > %d ORDER BY n.changed LIMIT %d", $lastcronrun - $recent, FILE_CRON_LIMIT_TOTAL);
}else{
$result = db_query("SELECT f.nid, f.vid, f.uri, f.size, f.type AS filemime, n.title AS filename FROM {file_nodes} f INNER JOIN {node} n ON f.nid = n.nid WHERE n.changed > %d ORDER BY n.changed", $lastcronrun - $recent);
}
//++++++ change this +++++
now my cron job is running well.
Comment #5
jvieille commentedWhere to change that?
Is it something handled by FF or Drupal core?
If FF, is the suggested change committed?
Thanks
Comment #6
johanneshahn commentedrewritten file_cron
please use latest dev release.
dont forget to run update.php
New cron options (/admin/settings/file) are:
may we need better description?