1) I had a problem with poorly formatted unicode characters in my csv file. This caused the whole import to abort at that point. I don't think it should have stopped but...

Anyway I fixed those characters and carried on with the import.... and .....

2)
SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'name' at row 1
SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'title' at row 1
SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'name' at row 1
SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'title' at row 1
SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'title' at row 1
SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'name' at row 1
SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'name' at row 1
SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'name' at row 1
SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'name' at row 1
SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'title' at row 1

Status message

Created 56705 nodes.
Failed importing 10 nodes.

So now I have 10 out of approximately 56715 lines which didn't import - and I have no idea which ones they are.... (OK - a hint to look at the Feeds Log would have helped.)

Right, So what's in my Feed log? the problem seems to be that my data has quotes in them.... escaped with a \
eg

575173,"Parodies Tossed: Alfred Bester\"s \"The Demolished Man\"","1956-01-00",\N,"Randall Garrett","http://en.wikipedia.org/wiki/Randall_Garrett",\N

The CSV parser seems to get confused by this and treats several lines as all part of the same field eg

SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'title' at row 1
Original item

array (
'title_id' => '1294965',
'title_title' => 'Afterword to \\The Country of the Blind,1988-00-00,\\N,Jack Williamson,http://en.wikipedia.org/wiki/Jack_Williamson,\\N
1295722,The Great Illusion (Part 2 of 5),1936-09-00,\\N,Jack Williamson,http://en.wikipedia.org/wiki/Jack_Williamson,\\N
1161985,The Great Illusion,1936-09-00,\\N,Jack Williamson,http://en.wikipedia.org/wiki/Jack_Williamson,\\N
1298015,L. Sprague de Camp: An Appreciation,2000-12-00,\\N,Jack Williamson,http://en.wikipedia.org/wiki/Jack_Williamson,\\N
1300044,Poul Anderson,2001-09-00,\\N,Jack Williamson,http://en.wikipedia.org/wiki/Jack_Williamson,\\N
1309896,Terraforming Earth,2001-10-00,\\N,Jack Williamson,http://en.wikipedia.org/wiki/Jack_Williamson,\\N
1309937,Letter (Locus #490),2001-11-00,\\N,Jack Williamson,http://en.wikipedia.org/wiki/Jack_Williamson,\\N
1313083,Gateway to Paradise,2011-09-00,\\N,Jack Williamson,http://en.wikipedia.org/wiki/Jack_Williamson,\\N
1313084,With Folded Hands . . . and Searching Mind,2011-09-00,\\N,Jack Williamson,http://en.wikipedia.org/wiki/Jack_Williamson,\\N
1313361,Damon Knight,2002-05-00,\\N,Jack Williamson,http://en.wikipedia.org/wiki/Jack_Williamson,\\N
1318320,Letter (Locus #500),2002-09-00,\\N,Jack Williamson,http://en.wikipedia.org/wiki/Jack_Williamson,\\N

So actually, rather than 10 records lost, I've not imported something like 10 times that amount because the subsequent records failed to be parsed.

Comments

gaele’s picture

gaele commented

Work around: http://drupal.org/node/1369874#comment-5755222

lmeurs’s picture

lmeurs commented

According to https://drupal.org/node/273939#comment-1413864 you can escape a double quote in a CSV file by adding a... double quote!

I am generating a simple CSV file in PHP, before outputting ie. a title I run it through a regular expression like this:

$title = preg_replace('/"/', '""', $title);

At http://board.phpbuilder.com/showthread.php?10369726-Escaping-quotes-when... I found another regular expression to fix an existing CSV file with.

$csv = preg_replace('/(?<!,)"(?!,)/', '""', $csv);

Hope this helps anyone!

gisle’s picture

gisle
he/him
Primary language Norwegian Bokmål
Location Norway
commented

As noted in comment #2 above you don't escape double quotes in a csv-file with a backslash, you escape it by adding another double quote. However, Feeds will strip both unless they're inside a quoted field.

The following line will preserve the double quotes around "To be or not to be." in the second field.

first field,"Hamlet: ""To be or not to be."" (second field)",third field

twistor’s picture

twistor commented
Status: Active » Postponed (maintainer needs more info)

Is there a bug here? Should Feeds be treating double quotes as an escape sequence?

gisle’s picture

gisle
he/him
Primary language Norwegian Bokmål
Location Norway
commented

I am not the OP (the issue originates in March 2012 and OP may not even be around anymore).

Is there a bug here?

Here's my two cents worth: This is not a bug.

It difficult to understand the post that started this issue. However, I think the OP is confused on two counts. 1) He believe the \ character should escape the double quotes (but csv are not regexps, so that is nonsense); 2) He believes that a newline can't be part of a field (it can, if it is inside a single double quote).

The cvs-format isn't documented, but most people treat the behaviour of MS Excel as the "official" cvs-standard. IMHO Feeds do a decent job of parsing the output produced by MS Excel when you Save as ... cvs

Should Feeds be treating double quotes as an escape sequence?

It already does, and as far as I am able to tell, it does it right. However, it may be a good idea to expand the documentation with examples of how to prepare csv-files that contain double quotes and newlines inside fields.

I think it it fair do give the OP some extra time to come forward with information about why he thinks he's found a bug. But if there is no response after some reasonable time, if I were the maintainer, I would close this issue with status "works as designed".

twistor’s picture

twistor commented
Status: Postponed (maintainer needs more info) » Closed (works as designed)

Sounds like a plan.