Separating words and stop-word problem

When run on my system i get errors from preg_split
"Compilation failed: missing terminating ] for character class at offset 27 "

I fiddled around some using your example-code at the bottom of your module and found out that some of the umlauts in that expression are causing this issue.

Workaround #1
de_stemmer.module
@@ -28 +28 @@
-return preg_split('/([^a-zA-ZÄÖÜßäëïöüáéíóúè]+)/u', $text, -1, PREG_SPLIT_NO_EMPTY);
+return preg_split('/([^a-zA-ZÄÖÜßäöüáóú]+)/u', $text, -1, PREG_SPLIT_NO_EMPTY);

But that is bad in case you get words with these umlauts. So i thought i turn it around into:

Workaround #2
de_stemmer.module
@@ -28 +28 @@
-return preg_split('/([^a-zA-ZÄÖÜßäëïöüáéíóúè]+)/u', $text, -1, PREG_SPLIT_NO_EMPTY);
+return preg_split('/([,\.; -\/\[\]><:\'\"]+)/u', $text, -1, PREG_SPLIT_NO_EMPTY);

So i set up a pile of possible in-text non alphanumeric characters to search and split at.

btw. i have to stick to a rather old PHP 4.3.3 so that might be the cause of my problems. But with this solution i can work with your fine extension. I also added some more stopwords and exeptions for the technical documents i have to search through.