Error messages not generated in UTF8 in some situations

Thanks for your prompt reply. Here is a snapshot of the tax creation screen where you'll find all the information.

Regarding to the installation it was achieved by using the webmatrix package. Commerce and dependencies was manually installed one module after the other. I tried this using both the release of Drupal 7 and the last dev version. I didn't have any error message during the installation neither with webmatrix or with Commerce.

I don't know if this could be of some usefulness, but I noticed that when trying to delete some records directly in the database (for testing purposes of course!!), specially the one having a varbinarymax field type, I recieve the following message from mssql 'Invalid Cast from System.Stream to System.Byte[] '.

Yes this works perfectly on mysql with the exact same settings and all last dev versions.

Webmatrix is a package to install an Express Web platform on windows with IIS 7.5, PHP 5.3.6, SQL server 2008 (version 10.50), the sql/Php driver (Microsoft 11/10/2010, last version), the sqlsrv module from Commerce Guys, and to create an empty SQL database. The install of Drupal and all modules are done exactly the same way than for a classic install with mysql.

My first try was to use all release and beta versions, but then the error occured earlier when I installed the tax ui module. That the reason why I went with all last dev versions (but I'm not sure if it was not due to the order I followed when activating the different modules that was different from the one I followed later on with my other attemps). With the dev versions the install completed without any warning, and I didn't notice anything wrong in drupal behavior.

What do you think about my remark on the varbinarymax field type ?

Thanks

Could you tell me which version of MS SQL you are using, and give me some other usefull details you can think of regarding to the install you ran the test ( do you use a webmatrix install, a full server 2008 and MS SQL...), therefore I will be able to rebuild the environment and run some test again.

Thanks a lot

I guess I have some new tracks to solve this issue. I didn't get any occasion yet to work again on mssql server, but I got a very similar behavior on mysql, even though the error message is slightly different (I got there an infinite loop with rules, but all dev got so much updates since I encountered the problem, that it could be the same issue).

This issue happends when you create a new VAT type tax (there was only this one and one product setup), with no condition setup, and you set your product price with the option none to Include tax in this price. You get then an infinite loop when you try to add your product to the cart. As soon as you setup a condition for the tax rules everything comes back to normal.

Current install - Drupal 7.2 - all last dev to today for Commerce and dependencies

After long hours trying to understand what was the story about this invalid multibyte error, I finally found out that it was a two drawers issue :

First : The invalid multibyte error was in fact the message sent by SQL server (set up in french in this case) after an exception which was not UTF8 encoded, therefore when the message was sent to check_plain it threw this error...

It could be a good idea to validate all strings against UTF8 and replace all characters in the string with something like '?', for any character that doesn't comply to UTF8, before calling check_plain(). That way we could have a chance to get on display the message. (That's what I did to solve this part)

Second part : Now the underlying sql server exception was due to a Data Field that was not serialized before reaching the function execute().

In the case of sqlsrv query.inc, we should check for arrays before binding values to the sql statement and serialize them, as we don't expect at this point to get anything else than strings. This was not revealed when using mysql, because, I presume, this is taking care of at some point by serializing everything before building the final query.

Nevertheless it seems that there is a lack of consistency in the way data fields are prepared, even if most of the time the are properly formatted, there are cases where arrays are sent as it to the save function. That is the case when managing VAT via rules, which was my initial issue.

Point #1 : yes, at least some Sql Exception messages are passing through check_plain before being output to the page (see the attached screen shot of the page ). Further more, check plain has only one instruction htmlspecialchars($text, ENT_QUOTES, 'UTF-8') which will throw an invalid databyte exception if the string is not UTF8 encoded. That's the reason why I suggest ( at least every time the string presents some risks as for Database's exceptions) to pre-check the string with drupal_validate_utf8 and replace all invalid characters before calling check_plain as a fall back. This could be something like this :

if ( !drupal_validate_utf8( $text)) {
  $chars = str_split($text);
  $text = '';
  foreach($chars as $char) {
    if ( !drupal_validate_utf8( $char)) {
      $text .= "?";
    } else {
      $text .= $char;
    }
  }
}

Point #2. I totally agree on the fact that Passing an array to an insert query should fail. Plain and simple, what I said was not that it is something to do, but and specially with Commerce which is in some cases passing unprocessed Data fields as Arrays, my suggestion is to give a fall back once more by converting the array to serialized strings and avoid a database exception ( This could be restricted to data Arrays that have a 'components' key). I tested that Drupal's mysql interface doesn't encoutered the same issue because I suppose it is fixing the case before passing to the actual query.

I'm probably the only one at this point to claim about this issue, but I'm not sure about the number of commerce install on sql server. I made several fresh installs on different platforms (Webmatrix on XP and windows server 2008 + sql server 2008) and I got always the same result. Set up commerce with commerce_kickstart, add a new VAT tax with a condition, create a new product without including the tax you created, then go to your cart and add a product.... you should get it.