Intermittent test failures in field.test

Looking at this, I'm wondering if it's a problem with duplications in the $value values.

That is to say, the test line that's failing is basically testing that a regexp like
input value="val1" .* input value="val2" .* input val="val3" ...
exists on the page, wehre val1, val2, etc. are random integers. But what if val6 and val20 are the same, for instance? Maybe the regexp when it gets to val6 will skip all the way to val20 and then the rest of the regexp will fail? I'm not all up on regexp greediness, so I'm not sure if that would happen or not (or if I'm making sense).

Suggested solution (if that's the problem) would be to do something with values like what is being done with the weights [i.e. make sure they are all different].

yched: this looks like it would work, but perhaps it would be better to have the "unique weights" and "unique values" loops separate instead of put into one loop together?

Also, in the first hunk you forgot to remove the $value call that was there, so it will be reset after you carefully made sure it was unique:

      do {
         $weight = mt_rand(-$delta_range, $delta_range);
-      } while (in_array($weight, $weights));
+        $value = mt_rand(1, 127);
+      } while (in_array($weight, $weights) || in_array($value, $values));
       $value = mt_rand(1, 127);

That looks more reasonable. :)

Let's make sure the test bot passes it (at least once), and then it's probably good for RTBC.

The values could probably just be generated with a $i++ type thing to make them unique, instead of random.

I am not sure I agree with you carlos8f. Randomness like this causes intermittent test failures that are not reproducible and cause people to pull their hair out. If these failures are not detected when the test is first committed (like this one), they occur rancomly, breaking the test bot, later on.

I just got bitten by this failure, just now, on a totally unrelated patch (a doc-only patch), though, so I hope we can get this fixed one way or another.

Or to put it another way, maybe randomness is a good thing, but if you use it as a big feature of your test, maybe you should have a loop in your test that runs the random section several times. This would verify that it's really passing based on several runs at the randomness, rather than relying on the fact that it worked when you first submitted the patch.

PHP and other programming languages don't really have "random" numbers -- they just have pseudo-random numbers, which will come out the same every time, given the same random seed. This test was not failing much until recently (at least, so I believe). Maybe if it had had a loop in it, the failure would have been detected and fixed before the patch was initially committed?