Large video files keep loading to Youtube as the orphaned video and didn't link to the original video node
| Project: | Video Upload |
| Version: | 6.x-1.x-dev |
| Component: | Code |
| Category: | bug report |
| Priority: | critical |
| Assigned: | Unassigned |
| Status: | active |
Hi,
I'd upload video successfully using Video Upload Field 6.x-1.x-dev. But those files is small like 10MB or so.
Today I tried to upload a large video file more than 70MB. This file has been uploaded to Drupal in the [token] directory successfully before I clicked save of the video node. Then the video will be uploaded from Drupal to YouTube during the cron job. Since the file was taking too long to upload, then the browser got the Error: 1460 time-out.
After the time out, this video file seemed to loose connection with the video node. Cron job kept uploading video file to YouTube again and again, and lot of Orphaned videos with YouTube referece ID existed on the YouTube. But the Video Node keep showing "Video is being processed" without the YouTube referece ID . Please see the first attachment.
And this uncessfull large video file will block the whole upload process if I created another video node. The above video node will be purged/disappeared with the new video node in the video upload under http://mysite/admin/content/video-upload. However, these 2 nodes were all existed in Drupal without the YouTube referece ID. Check the other 2 attachment.
Hope this help in debug......................
| Attachment | Size |
|---|---|
| WeatherForcast196.jpg | 116.29 KB |
| WeatherForcast197.jpg | 106.37 KB |
| WeatherForcast198.jpg | 13.14 KB |
#1
Oh....more update
Wait for a little bit longer. The cron will still get
Error: 1460 time-out, however the new video will be successfully uploaded to YouTube and the node will be updated with the YouTube reference ID. But the original unsuccessfully large video file keep loading and loading again. Gee..............Looks like the original unsuccessfully large video file broken the link between Cron and the video node.Hope this helps!!
#2
In order to make sure the time-out is the real issue, I try the other experiment without firewall connection limits.
This time, I upload another big file more than 70MB at home without Firewall connection limits. I didn't get
Error: 1460 time-outthis time during the node saving and cron job running. And the file is in the directory for sure. However, I still got the Video Node keep showing "Video is being processed" without the YouTube referece ID. And more Orphaned videos with YouTube referece ID existed on the YouTube.So....it seems that the
Error: 1460 time-outis not the real issue to break the video link instead of the file size matter?Any clue, again??
#3
The Issue title suppose to change seems the issue is large video file instead of time-out issue....... ^^|||
#4
ah, typo error ......correct again
#5
Any suggestion for the big video file? Thanks in advance.
#6
Sounds like your script is taking too long since it's a large file. Check your php.ini settings and see what your max_execution_time is set to.
#7
Make sense....... However YouTube is under site maintenance. I'll check it later. Thanks in advance.
BTW, did you find the "default keyword" fields anyway?
#8
Gee......... A lot try and error
My environment
Drupal 6.6
File Field 6.x-3.0
Video Upload 6.x-1.x-dev (May-19)
Thanks nikefido for the suggestion.
I did change the max_execution_time from 30 to 2,000 but still get unacceptable results for the large video exceed than 50MB since the upload traffic is very unpredictable. Every video can be uploaded to YouTube finally under 200MB as I tested. However, the link between Video Node and YouTube may not be correct. Around 80% of videos under 100MB get the correct YouTube link to video nodes successfully. No video beyond 100MB can get the correct YouTube link.
For those unsuccessfully linked video, the Cron run by poormanscron had been interrupted right at "video_upload_cron", and the rest cron job behind "video_upload_cron" can not be fired. However, the Video is still under uploading to YouTube without the control of "video_upload_cron". After a while another Cron run by poormanscron will be fired when reaching the cron run interval, then cron job interrupted again, there's the second instance of the same Video under uploading without control. And on and on................ Then YouTube has a lot of duplicate Videos instances uploaded with different YouTube links into 2 groups, some of the instances are Live!, some are tagged with Rejected(Duplicated work).
For those Live! video instances, they can be uploaded repeatedly before the first instance had been uploaded for duplication check. The rest will be tagged "Rejected(Duplicated work) once there was one successfully instance existed at YouTube.
The strange thing here is Video Upload was not so stable enough to re-link the node to any successfully video instance. Sometime you have to remove the video node and the video files on Drupal and YouTube to stop the repeated upload. The other time, Video Upload will find a instance(sometime link to Live!, sometime to Rejected) to link and stop the repeated upload.
I think the above bug all rise from the upload traffic takes too long to upload the video from Drupal to YouTube.
Hope this helps in debug and find the solution to solve. Thanks in advance.
#10
I'd love the see the process the node goes through from the time it first gets uploaded to youtube to the time the id comes back and it updates the node again. I'm trying to use this for a video contest where users will be submitting videos to the website. Would my best bet be to limit the video size to 30MB or so? what if cron needs to upload 10 videos every time it runs?
#11
@mellenger, the process is fairly straight-forward: during cron, any videos with a status of 'upload pending' are cycled through and sent to YouTube, which immediately returns an ID. This ID is stored and the status is set to 'unknown'. The next time cron is run, it cycles through all videos having an unknown status and checks with youtube. If the video has completed processing, the status is set to something like 'ok'.
@happydrupal, how frequently is poormanscron running cron? If it's too frequent, I can easily see a scenario where the uploads overlap. Perhaps a solution would be to toggle the status bit prior to upload so they don't overlap. However, you should also keep in mind that YouTube has an unpublished limit to how frequently one can use the service (I hit this limit when I had cron set to run every 5 minutes, but things worked again when I backed it off to 30 minutes).
#12
@jhedstrom
PHP setting
max_execution_time 2000
max_input_time 1200
memory_limit 96M
Poormanscron
Cron runs interval: 60 min
Retry interval: 30 min
The video beyond 30MB can be uploaded to YouTube as an orphaned video but lost link with the original video node. I still think the above bug all rise from the upload traffic takes too long to upload the video from Drupal to YouTube.
#13
Hi
here's an extra thing to try!
I'm running into the same "error". When uploading files larger than a certain amount the video is actually uploaded correct to youtube, but loses its connection to the node. I hope the following can help diagnose the origin of the problem. What's happening? I looked into the function that performs the actual upload
function _video_upload_gdata_insert_video(&$yt, &$yt_video) {// Try to upload the video, catching a Zend_Gdata_App_HttpException
// if available or just a regular Zend_Gdata_App_Exception.
try {
$new_video = $yt->insertEntry($yt_video, VIDEO_UPLOAD_YOUTUBE_UPLOAD_URL, 'Zend_Gdata_YouTube_VideoEntry');
// Set API version to 2.0.
$new_video->setMajorProtocolVersion(2);
dpm($new_video->getVideoId());
if ($new_video) {
// Translate zend result back into Drupal/FileField.
$result['video_id'] = $new_video->getVideoId();
}
return $result;
} catch (Zend_Gdata_App_HttpException $e) {
dpm(var_export($e,TRUE));
watchdog('video_upload', 'Video Upload: error %error while attempting to upload a video.', array('%error' => $e->getRawResponseBody()));
} catch (Zend_Gdata_App_Exception $e) {
watchdog('video_upload', 'Video Upload: error %error while attempting to upload a video.', array('%error' => $e->getMessage()));
}
}
(dpm's inserted)
Scenario A - small file
1. When the video is small enough insertEntry returns a YouTube id - and the result is returned to the node. Every things perfect
Scenario B - big file
1. When the video is to big, I get a "Read timed out after 10 seconds" error from Zend_Gdata_App_HttpException.
Zend_Gdata_App_HttpException::__set_state(array(
'_httpClientException' =>
Zend_Http_Client_Adapter_Exception::__set_state(array(
'message' => 'Read timed out after 10 seconds',
'string' => '',
'code' => 1000,
'file' => '/var/www/sites/xxx.xxx.xxx/sites/all/modules/video_upload/providers/youtube/Zend/Http/Client/Adapter/Socket.php',
'line' => 444,
'trace' =>
This error message is not shown in the watchdog because $e->getRawResponseBody() is empty, but a dump of $e-object does show it.
2. BUT the video is actually still uploaded to YouTube behind the scenes. So I guess the try-catch is somehow "premature" when saying something is wrong.
3. No $result is returned to the node
4. And because of this upload_pending is still the status of the node's video-field. So on next cron-run, it uploads it again - and it all starts over again.
(The cron runs IS completed - according to watchdog.)
I've also changed max_execution_limits, max_post_size and so on, without solving the problem. BUT wait, there is more ....
After googling, I've found out that when using the Zend HTTP_Client library with large files, it might be a good idea to raise the timeout limit. See http://code.google.com/intl/da/apis/gdata/articles/php_client_lib.html#a...
So I've tried to do this before uploading the video to YouTube. This was done in the youtube.inc file by inserting 2 lines of code
function video_upload_upload_youtube($video, $node, $field) {
static $http_client;
if (!$http_client) {
$http_client = _video_upload_authenticate_youtube();
}
///////////////////
// RAISE TIMEOUT to 1000 seconds
$configs = array('timeout' => 1000);
$http_client->setConfig($configs);
///////////////////
$yt = _video_upload_youtube($http_client);
if (!$yt) {
return FALSE;
}
$yt_video = _video_upload_video_entry();
_video_upload_construct_media_group($yt, $yt_video, $node, $field);
// Attach the file.
_video_upload_attach_video($yt, $yt_video, $video);
// Send the video.
return _video_upload_youtube_insert_video($yt, $yt_video);
}
This actually worked for me.
(But when the files got big enough (around 100mb), I ran into a new problem: "Mysql server has gone away" error fills the screen when cron was running. I guess because the connection to the database somehow timed out. I'm looking into this now - but I think that's a different problem.)
Still the most strange part is that insertEntry throws an error, when the video is still uploading to YouTube. I'm not a developer, so I don't know if and where this timeout setting should be included, but for me it was a supplement to all the other "limits".
#14
Thanks for looking into that issue mazaza. I'll try to add that timeout limit as a configurable option in the next release.
#15
Hi jhedstrom,
thanks for a very nice module.
I looked into the "Mysql Server Gone Away" issue, and found that mysql WAIT_TIMEOUT setting is important. I've raised the timeout setting through http://drupal.org/project/drupal_tweaks module with success. Setting the WAIT_TIMEOUT (2000 seconds) above php execution (2200 seconds) time out did the job for me so far.
In my testing today, I've been able to upload videos above 250gb and also upload for instance three videos totalling 350mb in the same cron run. I'll do some further tests to find the limit.
(I got my WAIT_TIMEOUT inspiration from http://drupal.org/node/259580#comment-1511512, but Drupal Tweak was an easy way to try it)
#16
Hi,
I also ran into this problem... I did manage to show the (big, 30mb) video in the node after multiple upload errors though, by deleting the duplicate uploads on youtube, and changing the youtube id field in the database itself... then at least the original upload shows up in the node on my drupal site.
I originally had my cron job run every 5 minutes, and have now changed it to every 30 minutes. I hope that solves a lot of this problem.
I now also set
max_execution_time = 2000
max_input_time = 1200
in php.ini.
Thanks all for the good work, saved me a lot of time!