When working with a Migrate into a mysql db for files from mssql, the uri had a special character (em dash in my case) that caused Migrate to not be able to validate the file. This is fine, except when it generated the message with the uri included, the special character was not filtered out causing saveMessage in sqlmap.inc to fail making the root cause difficulty to find.

Comments

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented
Status: Active » Postponed (maintainer needs more info)

What precisely was the failure message in sqlmap.inc?

reujwils’s picture

reujwils commented

The message that pop's up via the UI is:

An AJAX HTTP error occurred. HTTP Result Code: 500 Debugging information follows. Path: /batch?render=overlay&id=88&op=do StatusText: Service unavailable (with message) ResponseText: PDOException: in MigrateSQLMap->saveMessage() (line 303 of /var/www/html/sites/all/modules/migrate/plugins/sources/sqlmap.inc).

I'll try to replicate the error in my dev box to get the exact detail, but from what I remember, when I debugged the source and intercepted the saveMessage call, the message had the full uri (i.e. public://pictures/.....), I think it was 'Unable to copy file from public://pictures/....' and the resulting uri had the \x96 character causing the message insert to fail.

Of course I modified my migration to replace the special characters in the uri, but it just slowed me down with the more unsual service unavailable message from above.

I'll track down the exact line for you, but let me know in the meantime if this is or isn't clear.

Thanks!

reujwils’s picture

reujwils commented
Status: Postponed (maintainer needs more info) » Active

Forgot to update the status after I supplied more info as requested.

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented
Status: Active » Postponed (maintainer needs more info)

Running in drush with -d may be more useful (like, maybe we would be able to see the specific PDOException being generated).

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented
Status: Postponed (maintainer needs more info) » Closed (cannot reproduce)

No further information provided.

msbrar’s picture

msbrar commented
Priority: Minor » Normal
Status: Closed (cannot reproduce) » Active

"Unable to copy files from" with special characters in the file name still occurs. The problem seems with the copyFile function in plugins/destinations/file.inc. The method assumes that the sourcePath is an unencoded URI. However if the migrated source path is already a well encoded URI, the method encodes it again causing the special characters like % to be reencoded as %25 breaking the path.

There should be an option to tell if the source URI is already a well encoded URI or not. I added the marked lines in this method and it started working for me. Could you please validate the approach.

protected function copyFile($destination) {
    // Perform the copy operation, with a cleaned-up path.
    $preencodedsource = TRUE;  /* added this */
    if (!$preencodedsource) {  /* added this */
    	$this->sourcePath = self::urlencode($this->sourcePath);  /* existing code ignored if the URIs are pre-encoded */
    } /* added this */
    else { /* added this */
    	$destination = rawurldecode($destination);  /* added this .... decode the destination file name from an already well encoded URI*/
    } /* added this */
    if (!@copy($this->sourcePath, $destination)) {
      $migration = Migration::currentMigration();
      $migration->saveMessage(t('The specified file %file could not be copied to ' .
              '%destination.',
              array('%file' => $this->sourcePath, '%destination' => $destination)));
      return FALSE;
    }
    else {
      return TRUE;
    }
  }
mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented
Priority: Normal » Minor
Status: Active » Closed (cannot reproduce)

Please don't reopen long-closed issues. And please do check the issue queue before adding new issues. See #1856694: Local files and rawurlencode() don't work correctly in MigrateFileUri.