Audio files not being associated with the node

Update: It seems that sometimes (only sometimes) the audio files are not uploaded either. I am even more confused. Will continue to try and pin this down.

The recording must be sent by clicking the "Send" button before the CCK node is saved. If the recording has not finished uploading then the upload may get interrupted. This more likely with a long recording. I suspect this may be what is happening.

We are working on a way to upload the recording and then save the CCK node as a single step for the user. This can be done by capturing the FORM submit button, calling sendRecordedMessage() on the Applet, then POSTing the actual Drupal FORM.

Simon, can you try to reproduce the problem by recording a long message, clicking Send in the Applet then immediately saving the Applet. If you see the same problem then we will have a good indication that this is the cause.

Mmm do you mean saving before "Upload finished" appears on the applet? I'm not sure that is the cause, the recordings are being lost even after waiting for the audio to be successfully uploaded and then saving the node.

The single step upload/save form would be a very welcome addition!

Simon

the recordings are being lost even after waiting for the audio to be successfully uploaded and then saving the node.

Thanks, that is very helpful data. That implies that the file is reaching the server but then getting lost in the PHP code.

I notice you reference "field_estudio_dictado". That is not in the original AudioRecordingField module. I'm glad you have modified the module.

But I wonder, has anyone seen this dropped file problem with the unmodified AudioRecordingField? (I'm just trying to isolate the problem.)

Actually, "field_estudio_dictado" is just the name of our CCK Audio Recording field. I haven't modified the module apart from your code correction in http://drupal.org/node/324886#comment-1188374 to get rid of the "Missing argument" problem (which works fine, by the way).

I have performed a few tests under the suspicion that the problem might have to do (at least in some ocurrences) to the issue with Safari resetting the Applet when viewing another tab/window. I have another node in my drupal site that is frequently kept open in another tab by my users, and refreshes itself every 60 seconds. I tried the following in both Safari and Firefox:

1- Opened the node with auto-refresh in one tab
2- Opened a node with the audio recording field in another tab
3- Started recording. I did not view the other node, but allowed time for it to refresh in the background
4- Stopped the recording and pressed "Send"
5- Saved the node

On both browsers the applet reported "Upload finished", however, the compression step in safari was almost instantaneous and the audio was not saved. On Firefox I could follow the progress of the compression and upload, and the audio was sent successfully, and was correctly associated with the node.

I performed exactly the same test again but this time I sent the recording and saved the node before the node in the background refreshed itself. Sure enough, both browsers saved the audio correctly with the node.

Does this make any sense to you? I'm sorry if I can't be of help but I hope at least I can provide useful debugging info.

Thanks for your help!

Simon

On both browsers the applet reported "Upload finished", however, the compression step in safari was almost instantaneous and the audio was not saved. On Firefox I could follow the progress of the compression and upload, and the audio was sent successfully, and was correctly associated with the node.

Ahah! That is interesting. The released version has a problem. It prints some debug information that does not end in a line feed. That causes the following result code to be stuck at the end of a line. It must be at the beginning of the line to be seen by the Applet.

The fix is to put a newline before the SUCCESS result code, like this:

// Start SUCCESS on a new line so it is clear of earlier debug dumps.
print("\nSUCCESS - File has been uploaded.\n");

Note the \n before SUCCESS.

The "Upload finished" just means the file was sent. It does not mean it was received. When the file is received in the current version you should see "SUCCESS - File has been uploaded.". I will change these messages to be more clear. How about something like:

"Waiting for response..." then
"File saved."

Please let me know if it solves the problem.

This was discussed in another Issue but it is hard to find. More info here:

http://drupal.org/node/348717

I'm sorry to report that even after modifying the code as you mentioned, and migrating my users to firefox (just in case), I am still losing some audio files.

After the code modification, the message "SUCCESS - File has been uploaded" appears and the waveform dissapears, which I understand is the expected behavior. Using the applet on firefox also apparently avoids the problem of losing the audio when switching webpages/tabs.

My users are still losing some of the audio files they upload. It happens on about 4-5 of every 100 files they upload, but it is obviously frustrating. Since I can't reproduce this consistently, and rely on their input, I was wondering if there was any way for the whole process to be more verbose in order to debug, for example having the applet log to the Drupal watchdog as soon as the file is uploaded, or some sort of popup confirmation that includes the filename of the successful upload.

This issue has been specially difficult to pin down because it would seem to be a composite of two problems: Sometimes the audio file gets uploaded but not associated with the node, and sometimes the file doesn't get uploaded at all. I thought perhaps moving to firefox and modifying the code would do it, but unfortunately audios keep getting lost.

Please let me know if there are any tests or anything else I can perform to help you debug this. Thanks.

Simon

PS. I have also been unsuccessful in trying to modify the module to set the upload filename to something with the user and/or node ID (this would help in finding the files that do get uploaded but not associated). Any thoughts?

After the code modification, the message "SUCCESS - File has been uploaded" appears and the waveform dissapears, which I understand is the expected behavior.

At this point the file_save_upload() function has been called and passed.

Looking for info on file_save_upload() I found a Drupal issue with it failing silently if the file is too large! You mentioned it was mostly large files that were dropping. Could this be the problem? See link to issue report:

http://drupal.org/node/30520

Sometimes the audio file gets uploaded but not associated with the node, and sometimes the file doesn't get uploaded at all.

Is it possible that sometimes your users upload the recording but then neglect to submit the Drupal message that it is associated with?

You can pass back info like file name to the browser using CALLJS. You can see an example at the end of the function AudioRecordingField_file_receive(). You could for example try adding:

print("\nCALLJS Uploaded file named $new_name\n");

Also look in the Java Console for debugging info printed by the server.

I'll look at this more later.

From what I can see in http://drupal.org/node/30520 , this happens when the file is larger than the php.ini's 'post_max_size' . Mine is currently set at 8MB.

I'm using Speex compression, so of approximately 700 audios so far, the largest one is 1.3MB. Perhaps this sends it lower on my suspects list, although it is true that it seems to happen predominantly with longer dictations.

Is it possible that sometimes your users upload the recording but then neglect to submit the Drupal message that it is associated with?

It is always possible, but I have made a point of being specific about the 2 step process, send and submit, and have at least one of my users who is very methodical, but is still having lost dictations. Unfortunately, since it happens so sporadically, I haven't been able to reproduce the situation (on Firefox. Safari still loses the audio on tab change).

Thanks for your help. I'll attempt to implement your debugging suggestions and report back.

Reading a little more into the issue you pointed out regarding file upload (http://drupal.org/node/30520), 'post_max_size' is set at 8MB but 'upload_max_filesize' is set at 2M , which does make it at least plausible that it's running into this limitation and failing silently.

Since it's a production server, I'll try to increase the 'upload_max_filesize' limit to about 5MB late tonight and see if that has any effect...

I wanted to run this by you as a separate possible cause, and see what you think.

Interviewing my users, they seem to think that even though the lost audios are somewhat long, it's not so much the length/duration of the actual audio but perhaps the time they spend with the node editing page open, since some short audios in which they had to pause repeatedly or for a prolonged period also seem to get lost.

Is it possible that the page/java applet is timing out somehow without it being apparent? I don't want to add more confounding variables to the debugging process but I thought it might be relevant.

Note that POSTed files are UUEncoded. UUEncoded files are about 33% bigger than the original file.

it's not so much the length/duration of the actual audio but perhaps the time they spend with the node editing page open,

Thanks for this info. It is important to find out what user activity is associated with the dropped audio. Hopefully it is not completely random.

A few things come to mind. Drupal may be timing out and doing an auto-save of the edited page. This protects you when you edit for hours and then the power goes out. Are you using the AutoSave module?

http://drupal.org/project/autosave

People who edit a long time may use the Preview mode. The recording will probably get erased when switching to that Preview page and coming back. Changing the behavior of init() and start() in ListenUp, as we discussed before, will address this issue. is this possible?

I am not using autosave. I am using the 'saveguard' module, which will 'Warn user when they are leaving a page with unsaved changes' .
Also, the preview is not an issue since I'm using the 'Preview' module which 'Hides preview button on nodes.', so my users can't preview the nodes.

Ok , I performed a couple more tests..

1. Increased the 'upload_max_filesize' in php.ini to 4M, I´m still getting both lost audio files and unassociated audio files.
2. I was able to reproduce the problem of audio files not being associated with the node by leaving the node editing page open for an extended period. Since I have Trim enabled, the resulting audio file was small (around 100K), however the editing page was open for about 2-3 minutes. The applet accurately reported the file was uploaded (the file arrived on the server), but after saving the node and returning to it there was no audio loaded.
3. I activated the stashBeforeSend option and repeated the test on #2 . The file was again not associated with the node. But if I return to the node, load the previously stashed recording, send the audio and save the node, the audio IS associated with the node, which is consistent with my editing duration hypothesis, since loading the stashed recording/sending the audio/saving node only takes a few seconds.

I am still consistently losing about 5% of all my dictations, specially the more complex ones and my users are understandably frustrated. Short of asking my users to reenter each node they dictate and perform #3 if the audio does not load, I can't think of anything else I can do to solve this in the short term.

Please let me know any other tests I could run to help debug. Thanks!

I found a very interesting note about file_save_upload()

http://api.drupal.org/api/function/file_save_upload/6

The docs say "The file will be added to the files table as a temporary file. Temporary files are periodically cleaned. To make the file permanent file call file_set_status() to change its status."

ALso on this page: http://drupal.org/node/357711
it says "In D6, files now have a status. By default, the uploaded file is considered temporary and will be deleted on the next cron run after 24 hrs. So, if there is a file_save_upload and the same function does not include file_set_status($file, FILE_STATUS_PERMANENT);, it would be nice to issue a warning that the file may be deleted as a temporary file."

I noticed in my Java console that the file status below is 0, which means it is temporary.

stdClass::__set_state(array(
'filename' => '_1243542651.spx',
'filepath' => 'sites/default/files/_1243542651.spx',
'filemime' => 'audio/ogg',
'source' => 0,
'destination' => 'sites/default/files/_1243542651.spx',
'filesize' => 7315,
'uid' => 0,
'status' => 0,
'timestamp' => 1243542651,
'fid' => '530',
))

But looking at it with Dev Loader it is 1 so something may be making it permanent. But maybe it is temporary long enough to get deleted.

Note the code I added at the bottom:

if(!$file = file_save_upload(0,NULL,$upload_dir)){
print("\nERROR - file_save_upload failed\n");
$fid = "NULL";
} else{
print("\n_FILE ----- BEGIN\n");
var_export($file);
print("\n_FILE ----- END\n");
$fid = $file->fid;

// Set the file to permanent status so it won't get deleted by a cron job.
$tempfile = (object)$file;
file_set_status($tempfile, FILE_STATUS_PERMANENT);
print("\n_FILE after setting FILE_STATUS_PERMANENT ----- BEGIN\n");
var_export($file);
print("\n_FILE ----- END\n");

When I looked in the Java Console the status changed from 0 to 1.

This might be the solution.