MS Excel for Mac exports the CSV with funny line endings (if you do a less or vim on the file it shows '^M' symbols instead of new lines). I attached the file in question as an example.
Currently PImport doesn't take this case into consideration and thus sees the CSV as being just one line.
The attached patch normalizes the line endings.

Comments

andreiashu’s picture

Status: Active » Needs review
pobster’s picture

Good catch but two things;

function _pimport_normalize_line_endings($s)

$s is not a variable name - it's a letter ;o) Use $string, $input, etc - it makes the code easier for other people to read. Meaningful variable names really do help with debugging, module names however - yeah they can be as random as you like ;o)

$s = str_replace("\r\n", "\n", $s);
$s = str_replace("\r", "\n", $s);

This doesn't need duplication, we could just as easily use;

$s = str_replace(array("\r\n", "\r"), "\n", $s);

It still reads fine.

Thanks,

Pobster

andreiashu’s picture

StatusFileSize
new1.34 KB

Cool, thanks for the code review.
The attached patch should address your remarks. ;-)

Let me know if there are any other issues.

Thanks,
Andrei

pobster’s picture

Status: Needs review » Reviewed & tested by the community

Looks good to me, will test it tonight and commit it (unless Marcus can test and commit it sooner?)

* Setting to reviewed and tested because I can see that it'll work but I'll still actually physically test it later! ;o)

Thanks,

Pobster

pobster’s picture

Status: Reviewed & tested by the community » Fixed

Apologies, I completely forgot to commit this, even though I did actually remember to test it!

I see that you've already committed it (and rightly so!) so marking this as fixed.

Thanks,

Pobster

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

marcus_clements’s picture

Status: Closed (fixed) » Needs review

I've just realised that this patch means that blank lines are removed before the file is processed which means the row counting doesn't work properly for reporting. I think it's more important to give accurate row numbers for problems than to save processing on the import.

I've taken out the code that strips the blank lines and comitted.

pobster’s picture

Just remove this one line;

+  // Don't allow out-of-control blank lines
+  $string = preg_replace("/\n{2,}/", "\n\n", $string);

Pobster
edit: Oh umm... READ Pobster, READ... You said that already ;o)

andreiashu’s picture

@rayvaughn: agreed.
Initially I thought that removing that line of code you'll still have the same problem (wrong row counting) for files that have funny line endings. But it should actually work ok. I would suggest testing it with the file at the beginning of this issue just to make sure.

marcus_clements’s picture

Status: Needs review » Closed (fixed)