Weblog entry #83 for dkg

unreproducible buildd test suite failures
Posted by dkg on Tue 21 Jun 2011 at 15:05
I've been getting strange failures on some architectures for xdotool. xdotool is a library and a command-line tool to allow you to inject events into an existing X11 session. I'm trying to understand (or even to reproduce) these errors so i can fix them.

The upstream project ships an extensive test suite; this test suite is failing on three architectures: ia64, armel, and mipsel; it passes fine on the other architectures (the hurd-i386 failure is unrelated, and i know how to fix it). The suite is failing on some "typing" tests -- some symbols "typed" are getting dropped on the failing architectures -- but it is not failing in a repeatable fashion. You can see two attempted armel builds failing with different outputs:

The first failure shows [ and occasionally < failing under a us,se keymap (that is, after the test-suite's invocation of setxkbmap -option grp:switch,grp:shifts_toggle us,se):

Running test_typing.rb
Setting up keymap on new server as us
Loaded suite test_typing
Started
...........F..
Finished in 19.554214 seconds.

  1) Failure:
test_us_se_symbol_typing(XdotoolTypingTests)
    [test_typing.rb:58:in `_test_typing'
     test_typing.rb:78:in `test_us_se_symbol_typing']:
<"`12345678990-=~ !@\#$%^&*()_+[]{}|;:\",./<>?:\",./<>?"> expected but was
<"`12345678990-=~ !@\#$%^&*()_+]{}|;:\",./>?:\",./<>?">.

14 tests, 14 assertions, 1 failures, 0 errors
The second failure, on the same buildd, a day later, shows no failures under us,se, but several failures under other keymaps:
Running test_typing.rb
Setting up keymap on new server as us
Loaded suite test_typing
Started
..F.F.F.......
Finished in 16.784192 seconds.

  1) Failure:
test_de_symbol_typing(XdotoolTypingTests)
    [test_typing.rb:58:in `_test_typing'
     test_typing.rb:118:in `test_de_symbol_typing']:
<"`12345678990-=~ !@\#$%^&*()_+[]{}|;:\",./<>?:\",./<>?"> expected but was
<"`12345678990-=~ !@\#$%^&*()_+]{}|;:\",./<>?:\",./<>?">.

  2) Failure:
test_se_symbol_typing(XdotoolTypingTests)
    [test_typing.rb:58:in `_test_typing'
     test_typing.rb:108:in `test_se_symbol_typing']:
<"`12345678990-=~ !@\#$%^&*()_+[]{}|;:\",./<>?:\",./<>?"> expected but was
<"`12345678990-=~ !@\#$%^&*()_+[]{|;:\",./<>?:\",./<>?">.

  3) Failure:
test_se_us_symbol_typing(XdotoolTypingTests)
    [test_typing.rb:58:in `_test_typing'
     test_typing.rb:88:in `test_se_us_symbol_typing']:
<"`12345678990-=~ !@\#$%^&*()_+[]{}|;:\",./<>?:\",./<>?"> expected but was
<"`12345678990-=~ !@\#$%^&*()_+{}|;:\",./>?:\",./<>?">.

14 tests, 14 assertions, 3 failures, 0 errors
I've tried to reproduce on a cowbuilder instance on my own armel machine; I could not reproduce the problem -- the test suites pass for me.

I've asked for help on the various buildd lists, and from upstream; no one resolutions have been proposed yet. I'd be grateful for any suggestions or hints of things i might want to look for. It would be a win if i could just reproduce the errors.

Of course, one approach would be to disable the test suite as part of the build process, but it has already helped catch a number of other issues with the upstream source. It would be a shame to lose those benefits.

Any thoughts?