Bigger items

============

* Internationali[sz]ation.

* Use a throw-away slave interpreter for running each test case.

* Transfer timeouts should be dependent on file size and link speed.

* Add more support for target boards and RTOSes.

* Use the new expect terminal support for an "escape codes" API.

* Use expectk and write a GUI testing API, complete with record/playback.

Date: Thu, 29 Aug 2013 19:42:07 +0200

From: Jan Kratochvil <jan.kratochvil@redhat.com>

To: dejagnu@gnu.org

Subject: dejagnu-2.0 feature wishlist (from Cauldron 2013)

Hi,

I haven't found any discussion here about the features in hypothetical

dejagnu-2.0, as presented by Rob Savoye at Cauldron 2013.

I wrote some scripts on top of DejaGNU but I think at least some of the

functionality could be integrated into DejaGNU itself.  It depends whether

dejagnu-2.0 scope will remain the same or whether DejaGNU should be used

together with tools like buildbot or whether dejagnu-2.0 will integrate some

of the buildbot-like functionality (multi-note continuous runs).

Maybe there exists something similar already?  Originally I wrote it only for

myself but I see nowadays such tool may be useful for more people.

Former announcement of my scripts:

	https://sourceware.org/ml/archer/2010-q3/msg00194.html

URLs are no longer valid, the files can be found now at:

	git clone git://git.jankratochvil.net/nethome

	(that is my whole $HOME, not just the testsuite scripts)

The primary script 'hammock' is at:

	http://git.jankratochvil.net/?p=nethome.git;a=blob;f=bin/hammock

Essential fixup of current DejaGNU:

------------------------------------------------------------------------------

--orphanripper: It is used by default, normal DejaGNU scripts do not track

their spawned children which share fds 0/1/2 (stdio).  This means some such

children are due to *.exp code bugs occasionally leftover running forever.  As

they have their fds still open the testsuite with output redirected somewhere

will lock up at the end.  Some runaway processes also hog CPU for 100%.  The

following utility identifies runaway processes by using custom pty for them

and kills them at the end of testsuite run:

	http://pkgs.fedoraproject.org/cgit/gdb.git/plain/gdb-orphanripper.c

It sure should be better integrated in DejaGNU somehow.

Features outside of the current scope of DejaGNU:

------------------------------------------------------------------------------

--distro: Testing in various OSes.  My script implement it based on chroot

(Fedora/RHEL has tool 'mock' for it), it has some performance/management

advantages but it has to (1) run all OSes with the same kernel, (2) mock

supports only Fedora/RHEL OSes, (3) it can run only x86_64/i386 arch this way.

The real solution should be multi-node (so that it can also support non-x86*

testing), for x86* it would be commonly using VMs.  But it still could support

even mock/chroot as it runs without the hassle of disk images.

--component: Pre-set remote repositories for download of gdb/binutils/gcc etc.

I want to run my patches on top of clean tree, not in some existing directory

which may have leftover files forgotten to be checked into repository etc.

Understandably it also supports local repository caches.

--srcrpm is similar, it builds tree from a prepared archive - I should be able

to provide also .src.rpm (or .tar.gz) to run the test for.

--branch asks for example for branch "gdb_7_6-branch" from the repository.

--file: Provide custom patches for the newly built tree.

--target: Provide a list of custom configure --target options.  This could be

more general such as to provide any custom configure options.

--parallel: Parallelization of multiple build+testsuite runs, not just

parallelization of the testsuite run part.

If I ask to build 40x binutils with 40 different targets I may want to do

run it in parallel (like with make -j8).

Convenience:

------------------------------------------------------------------------------

--gdbserver, --valgrind, --bfd32, --gdbindex, --dwz, --dwarf=X, --stabs:

Various pre-set options.  One can configure it by hand but it is too difficult

for daily use, for example for --dwz it means for GDB

	runtest CC_FOR_TARGET=/bin/sh\ $PWD/../contrib/cc-with-tweaks.sh\ -m\ gcc CXX_FOR_TARGET=/bin/sh\ $PWD/../contrib/cc-with-tweaks.sh\ -m\ g++ ...

	(plus also GNATMAKE_FOR_TARGET, GO_FOR_TARGET and GO_LD_FOR_TARGET)

For --valgrind it means other cryptic options like:

	RUNTESTFLAGS=--target_board=valgrind

Incomplete racy reads ("read1"):

------------------------------------------------------------------------------

https://sourceware.org/bugzilla/show_bug.cgi?id=12649

GDB testsuite contains (yes, it still contains them) various racy cases:

  gdb_test_multiple "set dprintf-style agent" $msg {

      -re "warning: Target cannot run dprintf commands.*" {

It commonly works as when expect does the read() syscall all the GDB output is

ready.  But occasionally the next testcase FAILs.  This is because

occasionally only part of the output gets read by the read() syscall, regex

gets matched but the final $gdb_prompt is not discarded - and the leftover

$gdb_prompt corrupts the next testcase below.  Sure the fix is:

  gdb_test_multiple "set dprintf-style agent" $msg {

    -re "warning: Target cannot run dprintf commands.*\r\n$gdb_prompt $" {

There is LD_PRELOAD *.so file in the Bug above to reproduce these cases

reliably.  There is also a reproducer of different kind of bugs ("writew")

although those do not happen so often AFAIK.  This functionality could be

better integrated into DejaGNU.

(Sure the primary problem is that the testsuite should not use regex matching

and it should use generic GDB MI output parser.  But that is a problem that

only a few GDB features have implemented the GDB MI interface.)

Diffing of results:

------------------------------------------------------------------------------

http://git.jankratochvil.net/?p=nethome.git;a=blob;f=bin/diffgdb

 * GDB has various known FAILs.  They should be but they are not KFAILed or

   XFAILed.  (On recent Fedora I see there are only 23 of them but on CentOS-5

   there is 1063 of them.)

 * One is only interested in introduced regressions so one needs to diff two

   *.sum files.  Looking again and again at the same known FAILing cases is

   not productive.

 * During diff one is not interested for example in newly PASSing testcases.

   One also is not interested in FAIL->PASS cases.  One is definitely

   interested in PASS->FAIL regressions.  New FAILing testcases are also

   interesting.

Therefore the script above does a filtering of the diff results.  It parses

DejaGNU *.sum output although DejaGNU did already knew them internally.

The script also filters out unstable/racy results.  This may be outside of the

scope but in fact an unimplemented feature would be to provide statistics on

unstable results (so one can fix those) if I run the same build+testsuite many

times.

Not yet implemented: Finding a regression common reason:

------------------------------------------------------------------------------

I run 73 testsuite runs daily - primarily GDB in different OSes, for each OS

its x86_64 and i686 variant, for x86_64 OS also in -m32 mode.

If there happens a general regression I get 73 times PASS->FAIL result.  That

is not too convenient to filter out other changes out of the 73 regressions.

Moreover sometimes the regression affects for example only 32-bit OSes

- therefore there will be only about 24 PASS->FAILs and I have to figure out

in which testsuite combinations they happen.

In other cases the regression happens for example only on (older) RHELs and

not on Fedoras but that again means about 6 PASS->FAIL cases.

There would be nice some summary that this PASS->FAIL occured in testsuite

runs in directories rhel* and it did not occur in directories fedora* etc.