Commit f6a01213 authored by Luis Chamberlain's avatar Luis Chamberlain Committed by Shuah Khan
Browse files

selftests: allow runners to override the timeout 



The default timeout for selftests tests is 45 seconds. Although
we already have 13 settings for tests of about 96 sefltests which
use a timeout greater than this, we want to try to avoid encouraging
more tests to forcing a higher test timeout as selftests strives to
run all tests quickly. Selftests also uses the timeout as a non-fatal
error. Only tests runners which have control over a system would know
if to treat a timeout as fatal or not.

To help with all this:

  o Enhance documentation to avoid future increases of insane timeouts
  o Add the option to allow overriding the default timeout with test
    runners with a command line option

Suggested-by: default avatarShuah Khan <skhan@linuxfoundation.org>
Signed-off-by: default avatarLuis Chamberlain <mcgrof@kernel.org>
Reviewed-by: default avatarMuhammad Usama Anjum <usama.anjum@collabora.com>
Tested-by: default avatarMuhammad Usama Anjum <usama.anjum@collabora.com>
Signed-off-by: default avatarShuah Khan <skhan@linuxfoundation.org>
parent 1977ecea

Documentation/dev-tools/kselftest.rst

+22 −0
Original line number Diff line number Diff line
@@ -168,6 +168,28 @@ the `-t` option for specific single tests. Either can be used multiple times:: 


For other features see the script usage output, seen with the `-h` option.



Timeout for selftests

=====================



Selftests are designed to be quick and so a default timeout is used of 45

seconds for each test. Tests can override the default timeout by adding

a settings file in their directory and set a timeout variable there to the

configured a desired upper timeout for the test. Only a few tests override

the timeout with a value higher than 45 seconds, selftests strives to keep

it that way. Timeouts in selftests are not considered fatal because the

system under which a test runs may change and this can also modify the

expected time it takes to run a test. If you have control over the systems

which will run the tests you can configure a test runner on those systems to

use a greater or lower timeout on the command line as with the `-o` or

the `--override-timeout` argument. For example to use 165 seconds instead

one would use:



   $ ./run_kselftest.sh --override-timeout 165



You can look at the TAP output to see if you ran into the timeout. Test

runners which know a test must run under a specific time can then optionally

treat these timeouts then as fatal.



Packaging selftests

===================

tools/testing/selftests/kselftest/runner.sh

+10 −1
Original line number Diff line number Diff line
@@ -8,7 +8,8 @@ export logfile=/dev/stdout 
export per_test_logging=



# Defaults for "settings" file fields:

# "timeout" how many seconds to let each test run before failing.

# "timeout" how many seconds to let each test run before running

# over our soft timeout limit.

export kselftest_default_timeout=45



# There isn't a shell-agnostic way to find the path of a sourced file,
@@ -90,6 +91,14 @@ run_one() 
		done < "$settings"

	fi



	# Command line timeout overrides the settings file

	if [ -n "$kselftest_override_timeout" ]; then

		kselftest_timeout="$kselftest_override_timeout"

		echo "# overriding timeout to $kselftest_timeout" >> "$logfile"

	else

		echo "# timeout set to $kselftest_timeout" >> "$logfile"

	fi



	TEST_HDR_MSG="selftests: $DIR: $BASENAME_TEST"

	echo "# $TEST_HDR_MSG"

	if [ ! -e "$TEST" ]; then

tools/testing/selftests/run_kselftest.sh

+5 −0
Original line number Diff line number Diff line
@@ -26,6 +26,7 @@ Usage: $0 [OPTIONS] 
  -l | --list			List the available collection:test entries

  -d | --dry-run		Don't actually run any tests

  -h | --help			Show this usage info

  -o | --override-timeout	Number of seconds after which we timeout

EOF

	exit $1

}
@@ -33,6 +34,7 @@ EOF 
COLLECTIONS=""

TESTS=""

dryrun=""

kselftest_override_timeout=""

while true; do

	case "$1" in

		-s | --summary)
@@ -51,6 +53,9 @@ while true; do 
		-d | --dry-run)

			dryrun="echo"

			shift ;;

		-o | --override-timeout)

			kselftest_override_timeout="$2"

			shift 2 ;;

		-h | --help)

			usage 0 ;;

		"")