Problem/Motivation

When typing in Chinese a comma is represented as '，' rather than ','. The first character is Unicode U+FF0C ‘Fullwidth Comma', the second character used in the West is U+002C. Separating terms in free tagging vocabularies currently works with U+002C but doesn't work with U+FF0C. Users typing in a Chinese language IME have little to no awareness of this and no ability to type the ',' Unicode character U+002C other than exiting the IME.

What are the steps required to reproduce the bug?

Try and separate terms in a free tagging vocabulary with U+FF0C.

What behavior were you expecting?

Terms to be separated.

What happened instead?

Terms were not separated.

Also note:

Lists in Chinese are sometimes separated with '，' U+FF0C and at other times separated by '、' Unicode U+3001 'Ideographic Comma' - depending on the type of list/list context. In a multilingual or Chinese language only site all three forms of comma should be able to separate items in a list.

See http://blog.northclick.de/archives/25 for an approach (and approaching PHP functions which do not treat UTF-8 characters as multiple bytes per character).

Steps to reproduce

  1. Install Drupal standard profile
  2. Enable Language and Content Translation
  3. Go to admin/config/regional/language, add Chinese, Traditional
  4. Go to admin/config/regional/content-language, Enable translation for Content, Article and Taxonomy Term, Tags
  5. Go to /node/add/article, add an article.
    1. Select Chinese as the language
    2. In the tags field, enter 测试, 测试1
    3. Save node
    4. Notice the two tags are saved
  6. Go to admin/structure/taxonomy/manage/tags/overview. See that 测试 and 测试1 are separate tags.
  7. Go to /node/add/article, add another article.
    1. Select Chinese as the language
    2. In the tags field, enter 测试，测试1，测试2
    3. Save node
    4. Notice the three tags are saved as one
  8. Go to admin/structure/taxonomy/manage/tags/overview. See that 测试2 has not been added as a tag, the entire string 测试，测试1，测试2 has been incorrectly added as a single tag

Proposed resolution

TBA

Remaining tasks

Patch
Review
Commit

User interface changes

API changes

Data model changes

Release notes snippet

CommentFileSizeAuthor
#33 Screenshot 2023-02-13 at 12.11.15.png61.74 KBpandaski
#33 Screenshot 2023-02-13 at 12.11.08.png69.63 KBpandaski
#33 Screenshot 2023-02-13 at 12.11.01.png161.26 KBpandaski
#14 582534-freetagging-fullwidth-ideographic-commas.patch739 bytessitiveni
#14 582534-freetagging-fullwidth-ideographic-commas-D6.patch753 bytessitiveni
#12 582534-freetagging-fullwidth-ideographic-commas.patch731 bytessitiveni
#12 582534-freetagging-fullwidth-ideographic-commas-D6.patch745 bytessitiveni
#3 582534-freetagging-fullwidth-comma.patch919 bytesbrianV
#3 582534-freetagging-fullwidth-comma-D6.patch863 bytesbrianV
#2 582534-freetagging-fullwidth-comma.patch918 bytesbrianV
#2 582534-freetagging-fullwidth-comma-D6.patch862 bytesbrianV

Comments

mattyoung’s picture

mattyoung commented

sub

brianV’s picture

brianV commented
Version: 6.14 » 7.x-dev
Status: Active » Needs review
Issue tags: +i18n, +Usability, +internationalization
StatusFileSize
new 582534-freetagging-fullwidth-comma-D6.patch862 bytes
new 582534-freetagging-fullwidth-comma.patch918 bytes

Bumping to D7, and adding patched for both D7 and D6.

brianV’s picture

brianV commented
StatusFileSize
new 582534-freetagging-fullwidth-comma-D6.patch863 bytes
new 582534-freetagging-fullwidth-comma.patch919 bytes

Oops, forgot to send the 'Unicode' argument.

Err, the d.org server sends the patch files as ISO-8859-1, at least for me. So the ， character appears corrupted in the patch file when viewed. See http://drupal.org/node/89349

brianV’s picture

brianV commented
Status: Needs work » Needs review
Issue tags: +i18n, +Usability, +internationalization

Since the patches can't be seen correctly when the files are viewed, the changes are pasted below:

D7:

Index: includes/common.inc
===================================================================
RCS file: /cvs/drupal/drupal/includes/common.inc,v
retrieving revision 1.1046
diff -u -p -r1.1046 common.inc
--- includes/common.inc	12 Nov 2009 20:40:27 -0000	1.1046
+++ includes/common.inc	13 Nov 2009 18:03:54 -0000
@@ -6031,7 +6031,7 @@ function watchdog_severity_levels() {
 function drupal_explode_tags($tags) {
   // This regexp allows the following types of user input:
   // this, "somecompany, llc", "and ""this"" w,o.rks", foo bar
-  $regexp = '%(?:^|,\ *)("(?>[^"]*)(?>""[^"]* )*"|(?: [^",]*))%x';
+  $regexp = '%(?:^|,|，\ *)("(?>[^"]*)(?>""[^"]* )*"|(?: [^",|，]*))%xu';
   preg_match_all($regexp, $tags, $matches);
   $typed_tags = array_unique($matches[1]);

D6:

Index: includes/common.inc
===================================================================
RCS file: /cvs/drupal/drupal/includes/common.inc,v
retrieving revision 1.756.2.73
diff -u -p -r1.756.2.73 common.inc
--- includes/common.inc	6 Nov 2009 08:34:58 -0000	1.756.2.73
+++ includes/common.inc	13 Nov 2009 18:02:42 -0000
@@ -3634,7 +3634,7 @@ function watchdog_severity_levels() {
 function drupal_explode_tags($tags) {
   // This regexp allows the following types of user input:
   // this, "somecompany, llc", "and ""this"" w,o.rks", foo bar
-  $regexp = '%(?:^|,\ *)("(?>[^"]*)(?>""[^"]* )*"|(?: [^",]*))%x';
+  $regexp = '%(?:^|,|，\ *)("(?>[^"]*)(?>""[^"]* )*"|(?: [^",|，]*))%xu';
   preg_match_all($regexp, $tags, $matches);
   $typed_tags = array_unique($matches[1]);

Status: Needs review » Needs work
Issue tags: -i18n, -Usability, -internationalization

The last submitted patch failed testing.

Issue tags: -i18n, -Usability, -internationalization

Re-test of 582534-freetagging-fullwidth-comma.patch from comment #3 was requested by brianV.

Status: Needs review » Needs work

The last submitted patch, 582534-freetagging-fullwidth-comma.patch, failed testing.

Status: Needs work » Needs review

Re-test of 582534-freetagging-fullwidth-comma.patch from comment #3 was requested by brianV.

Status: Needs review » Needs work
Issue tags: +i18n, +Usability, +internationalization

The last submitted patch, 582534-freetagging-fullwidth-comma.patch, failed testing.

sitiveni’s picture

sitiveni commented

We were stumbling upon this issue (free tagging taxonomy terms with CJK commas) on our site.
Is there a reason why you didn't include the ideographic comma (U+3001) in the regular expression?

I temporarily made the following change to my core's common function drupal_explode_tags, which seems to work:

$regexp = '%(?:^|,|、|，\ *)("(?>[^"]*)(?>""[^"]* )*"|(?: [^",|、|，]*))%xu';

What is the reason that this issue is slumbering on and the solution isn't incorporated yet (apart from the obvious)?

brianV’s picture

brianV commented

sitiveni - please feel free to roll a revised patch to handle this.

sitiveni’s picture

sitiveni commented
Status: Needs work » Needs review
StatusFileSize
new 582534-freetagging-fullwidth-ideographic-commas-D6.patch745 bytes
new 582534-freetagging-fullwidth-ideographic-commas.patch731 bytes

Here we go then with the patches (D7 + D6):

Status: Needs review » Needs work

The last submitted patch, 582534-freetagging-fullwidth-ideographic-commas.patch, failed testing.

sitiveni’s picture

sitiveni commented
Status: Needs work » Needs review
StatusFileSize
new 582534-freetagging-fullwidth-ideographic-commas-D6.patch753 bytes
new 582534-freetagging-fullwidth-ideographic-commas.patch739 bytes

Oops, haven't encoded the files in UTF-8.

Status: Needs review » Needs work

The last submitted patch, 582534-freetagging-fullwidth-ideographic-commas.patch, failed testing.

wwwwoicn’s picture

wwwwoicn commented

Subscribe...

xjm’s picture

xjm
she/her
Primary language English
commented
Version: 7.x-dev » 8.x-dev

Marked #742404: Deal with Chinese commas properly as duplicate of this issue.

rogical’s picture

rogical commented

can we back port this to D7?

m.bdwey’s picture

m.bdwey commented
Issue summary: View changes

also same issuse with arabic
need to use ، not ,

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.6 was released on August 1, 2018 and is the final bugfix release for the Drupal 8.5.x series. Drupal 8.5.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.6.0 on September 5, 2018. (Drupal 8.6.0-rc1 is available for testing.)

Bug reports should be targeted against the 8.6.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.8.x-dev

Drupal 8.6.x will not receive any further development aside from security fixes. Bug reports should be targeted against the 8.8.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.9.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.7 was released on June 3, 2020 and is the final full bugfix release for the Drupal 8.8.x series. Drupal 8.8.x will not receive any further development aside from security fixes. Sites should prepare to update to Drupal 8.9.0 or Drupal 9.0.0 for ongoing support.

Bug reports should be targeted against the 8.9.x-dev branch from now on, and new development or disruptive changes should be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.2.x-dev

Drupal 8 is end-of-life as of November 17, 2021. There will not be further changes made to Drupal 8. Bugfixes are now made to the 9.3.x and higher branches only. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.15 was released on June 1st, 2022 and is the final full bugfix release for the Drupal 9.3.x series. Drupal 9.3.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.4.x-dev branch from now on, and new development or disruptive changes should be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

quietone’s picture

quietone commented
Status: Needs work » Postponed (maintainer needs more info)
Issue tags: +Bug Smash Initiative

Is this issue still a problem?

There has been no activity here for 7 years suggesting this may be resolved.

If you are experiencing this problem on a supported version of Drupal reopen the issue, by setting the status to 'Active', and provide complete steps to reproduce the issue (starting from "Install Drupal core").

Since we need more information to move forward with this issue, I am setting at Postponed (maintainer needs more info). If we don't receive additional information to help with the issue, it may be closed after three months.

Thanks!

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.9 was released on December 7, 2022 and is the final full bugfix release for the Drupal 9.4.x series. Drupal 9.4.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.5.x-dev branch from now on, and new development or disruptive changes should be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

pandaski’s picture

pandaski commented
StatusFileSize
new Screenshot 2023-02-13 at 12.11.01.png161.26 KB
new Screenshot 2023-02-13 at 12.11.08.png69.63 KB
new Screenshot 2023-02-13 at 12.11.15.png61.74 KB

Unfortunately, this issue is still available with Drupal 9.4.* and above.

Steps to reproduce:

1. Install a vanilla Drupal 9.4.* or above.
2. Create a content type and a taxonomy vocabulary - "tags" for example
3. Add a new field "free tags" for example - "Entity reference field" to Step 2 taxonomy vocabulary and Allow the number of values "unlimited"
4. Check "Create referenced entities if they don't already exist"
5. Choose "Autocomplete (Tags style)" in "Manage form display"
6. Type/Copy 测试, 测试1 in the "tag" field then save the node - the tag is separated as expected
7. Type/Copy 测试，测试1，测试2 in the "tag" field then save the node - the tag is not separated

pandaski’s picture

pandaski commented
Status: Postponed (maintainer needs more info) » Active
quietone’s picture

quietone commented
Issue summary: View changes

@pandaski, thanks for the steps to reproduce.

I have updated the Issue Summary to add the steps to reproduce, starting from a standard install. I used the info provided in #33.

pandaski’s picture

pandaski commented

There is a change record "drupal_implode_tags() and drupal_explode_tags() replaced with Drupal\Component\Utility\Tags class"

https://www.drupal.org/node/2212099

With function here for Drupal 9 and 10

https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Component%21Util...

and

https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Component%21Util...

$regexp = '%(?:^|,|、|，\ *)("(?>[^"]*)(?>""[^"]* )*"|(?: [^",|、|，]*))%xu'; // From comment 10

The above regex looks still good at testing.

penyaskito’s picture

penyaskito
he/him
Primary language Spanish
Location 🧑🏽‍🌾 Seville 💃 Andalusia, UTC+2 🇪🇺
commented

Looking at core\misc\autocomplete.es6.js, if you go to autocompleteSplitValues you'll see a hardcoded ",", so I guess this is still an issue.

quietone’s picture

quietone commented
Related issues: +#3076171: Provide a new library to replace jQuery UI autocomplete

@penyaskito, thanks for commenting. This was discussed in a bugsmash group triage meeting with Pandaski. catch and myself. catch noted that the replacement for JQuery UI autocomplete should be checked to see if it has the same problem. #3076171: Provide a new library to replace jQuery UI autocomplete. I have updated the other issue.

Version: 9.5.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.