#!/usr/bin/env bash
#
# A script to automate testing programs on the command line.
# author: Chris Kauffman <kauffman@umn.edu>
# license: GPLv3-or-later
# RELEASE: Wed Jan 26 09:39:43 PM CST 2022
read -r -d '' usage <<EOF
usage: testy <testfile.org> [test# test# ...]
testy --help
examples: testy test_prob1.org # runs all tests in file
testy test_prob1.org 3 5 7 # runs tests 3,5,7 in file
testy test_prob1.org 5 # runs only test 5 and shows failures to stdout
SHOW=1 testy test_prob1.org # runs tests and prints all failures to stdout
DEBUG=1 testy test_prob1.org # runs printing LOTS of debug messages
Run tests for a shell program specified in an org-like file and report
the results.
----------------------------------------
--- RUNNING TESTS ---
----------------------------------------
Running a test is done from the command line and will default to
running all tests in a provided test file. Output shows each test with
a pass/fail and failures have results files indicating what went
wrong. Below is an example from the examples/ directory:
>> cd examples/
>> ../testy bash_tests.org
============================================================
== testy bash_tests.org
== Running 2 / 2 tests
1) Output Tests : ok
2) Failure Demo : FAIL -> results in file 'test-results/test-02-result.tmp'
============================================================
RESULTS: 1 / 2 tests passed
Inspecting the failure file indicated (always under freshly created
directory 'test-results/' ) shows the following output (plain text but
easier easier to read in org-mode):
----------------------------------------
>> cat test-results/test-02-result.tmp
* (TEST 2) Failure Demo
COMMENTS:
This test will fail and produce output associated to show the
side-by-side diff that primarily reports failures.
** program: bash -v
** --- Failure messages ---
- FAILURE: Output Mismatch at lines marked
** --- Side by Side Differences ---
- Expect output in: test-results/raw/test-02-expect.tmp
- Actual output in: test-results/raw/test-02-actual.tmp
- Differing lines have a character like '|' '>' or '<' in the middle
#+BEGIN_SRC sbs-diff
==== EXPECT ==== ==== ACTUAL ====
>> echo "Matching Line" >> echo "Matching Line"
Matching Line Matching Line
>> echo "Mismatching Line" >> echo "Mismatching Line"
Misma____ Li__ | Mismatching Line
>> echo "Extra line in ACTUAL" >> echo "Extra line in ACTUAL"
> Extra line in ACTUAL
>> echo "Extra line in EXPECT" >> echo "Extra line in EXPECT"
This is the extra line <
Extra line in EXPECT Extra line in EXPECT
>> printf "Matches fine\nAnd again\n" >> printf "Matches fine\nAnd again\n"
Matches fine Matches fine
And again And again
#+END_SRC
** --- Line Differences ---
EXPECT: 4) Misma____ Li__
ACTUAL: 4) Mismatching Line
ACTUAL: 6) Extra line in ACTUAL
EXPECT: 7) This is the extra line
----------------------------------------
The main section in the middle of the file is a side-by-side diff of
expected and actual output with the center column indicating what
differences were detected. Generally whitespace differences are
ignored.
----------------------------------------
--- TEST FILE FORMAT ---
----------------------------------------
Tests are specified in org-like files. Each top-level section starts
with a * with a test title, followed by comments and test sessions of
input/output. Each test can have multiple sessions. As a session is
found it is run. If the session fails, subsequent sessions for that
test are not run.
Sample input file (sample_tests.org):
----------------------------------------
#+TITLE: Sample Tests
* Test echo
Check that the 'echo' command in bash is working.
The org-mode 'sh' param is not honored in testy; it is for easy
editing/formatting in Emacs but does not reflect what program will
actually run the and can be replaced with whatever.
#+BEGIN_SRC sh
>> echo 'hello'
hello
>> echo 'Hi there!'
Hi there!
#+END_SRC
* Test printf, will fail
Tests whether printf works.
#+BEGIN_SRC sh
>> printf "Hello world\n"
Hello world
>> printf "Multi-line\noutput is expected\nhere\n"
Multi-line
output is expected
here
>> printf "%s\n" "substitute me"
substitute me
#+END_SRC
This second session below will fail (intentionally) as the output of
the printf will not match the expected output. The results of the
failure will be in a file which is listed by testy as the tests run.
#+BEGIN_SRC sh
>> echo 'hi'
hi
>> printf 'INTENTIONAL fail\n'
INTENTIONALly fails
#+END_SRC
* Test bc
This test uses a different interpreter than the standard 'bash'. The
'bc' program interprets standard mathematical expressions. Note the
use of #+TESTY expression to change the program for this test.
#+TESTY: program="bc -iq"
#+BEGIN_SRC sh
>> 1+1
2
>> 3*5+12
27
#+END_SRC
----------------------------------------
Running the command './testy sample_tests.org' will produce output like the following:
----------------------------------------
> ./testy sample_tests.org
============================================================
== sample_tests.org : Sample Tests
== Running 3 / 3 tests
1) Test echo : ok
2) Test printf, will fail : FAIL -> results in file 'test-results/test-02-result.tmp'
3) Test bc : ok
============================================================
RESULTS: 2 / 3 tests passed
----------------------------------------
The file listed will will contain information on the failure.
----------------------------------------
--- BEHAVIOR / ENVIRONMENT VARIABLES ---
----------------------------------------
The following variables can be specified in test files via lines like
#+TESTY: var="value"
or via an environment variable during a program run as in
> VAR="value" testy testfile.org
or via exporting an environment variable as in
> export VAR="value"
> testy testfile.org
They will change the behavior of how the test data is interpreted.
GLOBAL VARIABLES that are usually specified at the beginning of a test
file before any other tests.
PROGRAM="bash -v" : program to run/test; input is fed to this program
PROMPT=">>" : prompt that indicates input to the program
ECHOING="input" : {input, both} for program input echoing style,
"input" means the program echoes only input provided by testy, testy will add back in prompts
"both" echoes both prompt and input so testy won't add back anything
NOTE: testy does not support mocked interaction tests for programs that don't echo input
as this is generally hard to do
PREFIX="test" : prefix for test output files, often changed to reflect program name like 'myprog'
RESULTDIR="test-results" : directory where the results will be written
RESULTRAW="RESULTDIR/raw" : directory where actual / expect / valgrind results are stored
TIMEOUT="5s" : maximum time to complete test before it is failed due to timeout; passed to the 'timeout' utility
POST_FILTER="" : program to adjust output from test before evaluating, run as 'cat output | post_filter > actual.tmp'
USE_VALGRIND="0" : set to 1 to run programs under Valgrind which checks for memory errors; useful for C programs especially
VALGRIND_REACHABLE="1" : under valgrind, report errors if memory is still reachable at the end of the program
VALGRIND_OPTS="" : pass additional options to valgrind such as '--suppressions=test_valgrind.supp' to use a suppression file
SKIPDIFF="0" : skip diffing results, useful if checking only valgrind with actual output varying between runs
Each of the above Global variables can be set Locally during a single
test by setting their lower-case version. For example:
* Test 5: A test of bc
#+TESTY: program="bc -i"
will send input to the program "bc -i" and check output rather than
the default PROGRAM. The lower case options are reset during each test
run but NOT in between sessions in single test.
Finally, these variables control some global behavior of the testy.
SHOW=0 : set to 1 to print test error results after completing
DEBUG=0 : set to 1 to print LOTS of debugging messages
REPORT_FRACTION=0 : report the fraction of tests passed rather than the count
----------------------------------------
--- TESTY MULTI ---
----------------------------------------
Standard tests are for a single program running at a time. If several
programs need to run concurrently and coordinated during a test, one
can use the special program line
#+TESTY: PROGRAM='TESTY_MULTI'
for all tests or
#+TESTY: program='TESTY_MULTI'
for a single test.
The test itself then takes as input a series of commands which dictate
when to start programs, feed them input, sned them signals, and wait
for them to shut down.
--- TESTY_MULTI Commands are (briefly) ---
- START <key> <program> [args]
>> START server ./banter_server gotham # runs program 'banter_server gotham' and refers to it via key 'server'
>> START bruce ./banter_client gotham bruce # runs program 'banter_client gotham bruce' and refers to it via key 'bruce'
- SIGNAL <key> <sigspec>
>> SIGNAL server -15 # sends program w/ key 'server' signal 15 (TERM)
>> SIGNAL bruce -INT # sends program w/ key 'server' a keyboard interrupt signal (15)
- INPUT <key> text text text
>> INPUT bruce Robin? Barbara? # sends text input to program w/ key 'bruce'
>> INPUT clark <EOF> # sends End of Input to program w/ key 'clark'
- WAIT <key>
>> WAIT server # causes testy to wait for program w/ key 'server' to complete
- WAIT_ALL
>> WAIT_ALL # waits for all programs to complete
- OUTPUT <key> <filter>
>> OUTPUT server cat # testy prints the output for program w/ key 'server' passing to through filter 'cat'
>> OUTPUT bruce ./test_filter_client_output # ditto but passes through the specified filter program
- OUTPUT_ALL
>> OUTPUT_ALL cat # testy prints output for all programs for comparison in the test results; filtered through 'cat'
>> OUTPUT_ALL ./test_filter_client_output # ditto but passes through the specified filter program
- CHECK_FAILURES <key> <filter>
>> CHECK_FAILURES server cat # for 'server', prints any failures like timeout, non-zero return, valgrind problems, etc.
# prints nothing if no failures detected
- CHECK_ALL <filter>
>> CHECK_ALL cat # checks failures in all programs that are part of test passing through 'cat' as a filter
- SHELL cmd cmd cmd
>> SHELL rm some-file.txt # runs a shell command in the middle of the test in this case removing a file
----------------------------------------
An example of a TESTY_MULTI testing file is in
testy/examples/banter_tests.org
which tests a tiny chat server/client written in bash. A server is
started and several clients 'join' the server and exchange messages.
TESTY_MULTI has a few more control global variables to dictate
behaviors specific to it.
TICKTIME="0.1" # amount of time to wait in between test commands during a TESTY_MULTI session
VALGRIND_START_TICKS="8" # number of ticks to wait during TESTY_MULTI when starting a program under valgrind
# valgrind slows things down so it takes more time for programs to start up
Depending on system speed, one may wish to lengthen these parameters
through setting them globally at the top of the testy file as in:
#+TESTY: TICKTIME=0.1
#+TESTY: VALGRIND_START_TICKS=8
----------------------------------------
--- CAVEATS ---
----------------------------------------
testy is in ALPHA stage and actively being developed. For that reason
no guarantees are made about its reliability. Especially TESTY_MULTI
sessions have some known failings not to mention the fact that relying
on a tick time to coordinate programs is doomed to fail at some point.
All the same, enjoy!
- Chris
EOF
################################################################################
# BASIC GLOBALS
# some global options/programs
STDBUF="stdbuf -i 0 -o 0 -e 0" # disables standard IO buffering for a program
SDIFF="diff -ytbB" # side by side diff, no tabs, ignore space changes, ignore blank lines
DIFF="diff -bB \
--unchanged-line-format='' \
--old-line-format='EXPECT:%4dn) %L' \
--new-line-format='ACTUAL:%4dn) %L'" # diff which will show prepend EXPECT/ACTUAL to differing lines
TIMEOUTCMD="timeout --signal KILL" # kills user programs after a certain duration of 'timeout'
VALG_ERROR="13" # error code that valgrind should return on detecting errors
VALGRIND_PROG="valgrind"
VALGRIND_PROG+=" --error-exitcode=13"
VALGRIND_PROG+=" --leak-check=full"
VALGRIND_PROG+=" --show-leak-kinds=all"
VALGRIND_PROG+=" --track-origins=yes"
# VALGRIND_PROG+=" --keep-debuginfo=yes" # old valgrind on gradescope does not support this option
# Uses gsed if available for BSD-based operating systems and sed
# otherwise
#
# NOTE: it would be better to honor this as an environment variable
# that defaults to "normal" sed. That way systems with a different sed
# could set the env variable to tailor the sed cmd. Do this in a
# future release.
if command -v gsed >/dev/null 2>&1; then
SEDCMD=$(command -v gsed)
else
SEDCMD=$(command -v sed)
fi
TMPFIFOS="" # blank if local directory supports FIFOs, '/tmp' if FIFOS supported there instead
# default values for the program to be tested
PROGRAM=${PROGRAM:-"bash -v"} # program to run if no option is specified
PROMPT=${PROMPT:-">>"} # prompt to honor if none is specified
ECHOING=${ECHOING:-"input"} # {input, both} for program input echoing style
PREFIX=${PREFIX:-"test"} # prefix for the files that are produced by testy
RESULTDIR=${RESULTDIR:-"test-results"} # directory where the results will be written
RESULTRAW=${RESULTRAW:-"$RESULTDIR/raw"} # directory where actual / expect / valgrind results are stored
TIMEOUT=${TIMEOUT:-"5s"} # time after which to kill a test, passed to 'timeout' command and program_wait();
POST_FILTER=${POST_FILTER:-""} # run this program on output to adjust it if needed
USE_VALGRIND=${USE_VALGRIND:-"0"} # use valgrind while testing
VALGRIND_REACHABLE=${VALGRIND_REACHABLE:-"1"} # report valgrind errors if memory is still reachable
SKIPDIFF=${SKIPDIFF:-"0"} # skip diffing results, useful if checking valgrind but actual output can vary
CHECKRETURN=${CHECKRETURN:-"0"} # program exiting with a nonzero value triggers a failure
VALGRIND_OPTS=${VALGRIND_OPTS:-""} # additional options to valgrind,
TICKTIME=${TICKTIME:-"0.1"} # multi: amount of time to wait in between test commands
VALGRIND_START_TICKS=${VALGRIND_START_TICKS:-"8"} # multi: number of ticks to wait when starting valgrind programs which take a while
# VALGRIND_OPTS="--suppressions=test_valg_suppress_leak.conf"
# LONGTICKS=${LONGTICKS:-"8"} # number of ticks to wait starting a program w/ valgrind during TESTY_MULTI
# INPUT_STYLE="normal"
RETCODE_TIMEOUT=137 # code usually returned by timeout when it kills programs
RETCODE_SEGFAULT=139 # code usually returned when OS kills a program
PASS_STATUS="ok" # status message associated with passing
FAIL_STATUS="FAIL" # default status message associated with failing a test, anything not $PASS_STATUS is a failure though
TEST_TITLE_WIDTH=20 # initial width for test test_titles, set to widest during initial parsing
function reset_options() { # reset options to defaults, run before each test session
program=$PROGRAM
prompt=$PROMPT
echoing=$ECHOING
prefix=$PREFIX
resultdir=$RESULTDIR
resultraw=$RESULTRAW
timeout=$TIMEOUT
post_filter=$POST_FILTER
use_valgrind=$USE_VALGRIND
valgrind_reachable=$VALGRIND_REACHABLE
skipdiff=$SKIPDIFF
checkreturn=$CHECKRETURN
valgrind_opts=$VALGRIND_OPTS
ticktime=$TICKTIME
valgrind_start_ticks=$VALGRIND_START_TICKS
# input_style=$NORMAL
# longticks=$LONGTICKS
}
# fail if a tool is not found, used to check for utilities like timeout and valgrind
function checkdep_fail() {
dep="$1"
if ! command -v "$dep" >&/dev/null; then
echo "ERROR: testy requires the program '$dep', which does not appear to be installed"
echo "Consult your OS docs and install '$dep' before proceeding"
echo "If '$dep' is installed, adjust your PATH variable so '$dep' can be found using 'which $dep'"
which "$dep" # Intentionally using 'which' as it shows the program name
exit 1 # and path on stderr to help diagnose missing programs
fi
}
function debug() { # print a debug message
if [[ -n "$DEBUG" || -n "$TESTY_DEBUG" ]]; then
echo "==DBG== $1" >/dev/stderr
fi
}
function updateline() { # processes $line to set some other global variables
line="$REPLY" # copy from REPLY built-in variable to avoid losing whitespace
((linenum++)) # update the current line number
first="${line%% *}" # extracts the first word on the line
rest="${line#* }" # extracts remainder of line
}
################################################################################
# Multi test functions
function tick() { # pause execution during multi test sessions
sleep "$ticktime"
}
# Calls wait on program with given key, captures return value of
# program from wait, marks it as no longer running. If program is
# unresponsive for TIMEOUT seconds, kills it and marks it as timed
# out.
function program_wait() {
debug "program_wait '$1'"
key="$1"
wait "${program_pid[$key]}" &>/dev/null # wait on the child to finish, safe as its done
retcode=$?
debug "wait on '$key' pid ${program_pid[$key]} gave retcode: $retcode"
program_retcode[$key]=$retcode
program_state[$key]="Done" # state will be changed when checking for failures
case "$retcode" in # inspect return and print appropriate inline messages
0) # no action for normal return code
;;
"$RETCODE_TIMEOUT")
program_state[$key]="Timeout"
printf "Return Code %s: TIMEOUT, program killed, not complete within %s sec limit\n" "$retcode" "$timeout"
;;
"$RETCODE_SEGFAULT")
program_state[$key]="SegFault"
printf "Return Code %s: SIGSEGV (segmentation fault) from OS\n" "$retcode"
;;
"$VALG_ERROR")
program_state[$key]="ValgErr"
printf "Return Code %s: Valgrind Detected Errors\n" "$retcode"
;;
*)
printf "Non-zero return code %s\n" "$retcode"
;;
esac
if [[ "${program_input__fifo_fd[$key]}" != "CLOSED" ]]; then
to="${program_input__fifo_fd[$key]}" # input_fifo still open in testy, close
debug "closing $to"
exec {to}>&- # close the input_fifo, may already have been done
rm -f "${program_input__fifo[$key]}" # remove input_fifo from disk
program_input__fifo_fd[$key]="CLOSED" # mark as closed
else
debug "file descriptor $fd already closed"
fi
return 0
}
# Check if the program is running or dead. Update the program_state[]
# array for the given key setting the entry to 0 if the program is no
# longer alive. Uses the 'kill -0 pid' trick which doesn't actually
# deliver a signal but gives a 0 return code if a signal could be
# delivered and a 1 error code if not. A return value from this of 0
# indicates success (program is still alive) and nonzero indicates the
# program is dead. Use in conditional constructs like:
#
# if ! program_alive "server"; then
# printf "It's dead, Jim"
# fi
function program_alive() {
key="$1"
if [[ "${program_state[$key]}" != "Running" ]]; then
printf "Program '$key' has already died\n"
return 0
fi
pid=${program_pid[$key]}
output=$(kill -0 "$pid" 2>&1)
ret=$? # capture return val for kill: 0 for alive, 1 for dead
if [[ "$ret" != "0" ]]; then
printf "Program '%s' is not alive: %s\n" "$key" "$output"
program_wait "$key" # wait on program and mark as dead
fi
return $ret
}
# TODO: add checking for if the program fails immediately to exec;
# e.g. not found, not compiled; adjust an error message for this;
# found this is not detected in some cases and makes debugging
# difficult
# TODO Design Note: Originally thought that I Cannot use the 'timeout'
# program when starting programs as this would not allow one deliver
# signals. However, on further experimentation, a program sequence
# like
#
# > timeout 1000s valgrind gwc < in.fifo &
#
# can be passed signals which are forwarded on to the program
# (gwc). This opens up use of 'timeout' instead of manual 'waiting'
# for programs which is likely to be somewhat more robust though the
# timeout utility is not present everywhere. Likely SWITCH TO TIMEOUT
# UTILITY at some point in the future for simplicity.
SHELL_SYMBOLS='.*(>|<|\||&|&&|\|\|).*' # regular expression used to detect commands that contain shell symbols which are barred
# Start a program and populate various arrays with the programs
# information such as PID, output / input sources. Checks that the
# program is actually alive after starting it and prints an error
# message if not. Calls tick() before returning. If Valgrind is
# used, then starts the program under Valgrind and waits a longer time
# (VALGRIND_START_TICKS) before checking on it as Valgrind usually
# takes much longer to start up programs.
function program_start() {
debug "program_start '$1' '$2'"
key="$1"
progcmd="$2"
if [[ "$progcmd" =~ $SHELL_SYMBOLS ]]; then
{
printf "ERROR with '%s'\n" "$progcmd"
printf "TESTY_MULTI does not support program commands with shell redirects, pipes, booleans, or backgrounds\n"
printf "The following symbols in program commands will trigger this error: > < | & || && \n"
printf "Please rework the test file to avoid this\n"
} >/dev/stderr
exit 1
fi
program_keys+=("$key")
debug "Adding program w/ key '$key' command '$progcmd'"
program_command[$key]="$progcmd"
program_name[$key]="${progcmd%% *}"
program_output_file[$key]=$(printf "%s/%s-%02d-%s_output_file.tmp" "$resultraw" "$prefix" "$testnum" "$key")
program_input__fifo[$key]=$(printf "%s/%s-%02d-%s_input__fifo.tmp" "$resultraw" "$prefix" "$testnum" "$key")
if [[ -n "$TMPFIFOS" ]]; then # use /tmp instead, likely due to Windows/WSL
program_input__fifo[$key]=$(printf "%s/%s-%02d-%s_input__fifo.tmp" "${TMPFIFOS}" "$prefix" "$testnum" "$key")
fi
if [[ "$use_valgrind" == 1 ]]; then
program_valgfile[$key]=$(printf "%s/%s-%02d-%s_valgrd.tmp" "$resultraw" "$prefix" "$testnum" "$key")
VALGRIND="${VALGRIND_PROG} ${VALGRIND_OPTS} --log-file=${program_valgfile[$key]}"
# program_valgfile[$key]=$(mktemp $resultraw/testy_valg.XXXXXX)
# program_output_file[$key]=$(printf "%s/%s-%02d-%s_output_file.tmp" "$resultraw" "$prefix" "$testnum" "$key")
else
program_valgfile[$key]="NONE"
VALGRIND=""
fi
rm -f "${program_input__fifo[$key]}" "${program_output_file[$key]}" # remove just in case
mkfifo "${program_input__fifo[$key]}" # create the fifo going to the program
# Below block starts a subshell to close extraneous file
# descriptors then exec's the actual program. MUST close the other
# program file descriptors otherwise when testy closes an input
# fifo via <EOF>, other children will still have it open via FD
# inheritance; do this in a subshell so as not to mess with testy
# then exec to replace the process image with the child process.
cmd="$TIMEOUTCMD $timeout $STDBUF $VALGRIND $progcmd <${program_input__fifo[$key]} &> ${program_output_file[$key]}"
debug "running: '$cmd'"
(
for sig in ${suppress_signals[@]} ${exit_signals[@]}; do
trap - $sig # reset trap/signal handling
done # in child procresses to default
for tofd in "${program_input__fifo_fd[@]}"; do # close fds for input to other programs so only
if [[ "$tofd" != "CLOSED" ]]; then # testy owns the input
exec {tofd}>&-
fi
done
eval exec $cmd # exec replaces current image with child process in a
) & # subshell, started in background, no quoting to all redirect
program_pid[$key]=$!
debug "PID is '${program_pid[$key]}'"
program_state[$key]="Running"
program_retcode[$key]="?"
exec {to}>"${program_input__fifo[$key]}" # open connection to fifo for writing
program_input__fifo_fd[$key]=$to
debug "to: $to program_input__fifo_fd: ${program_input__fifo_fd[$key]}"
if [[ "$use_valgrind" == "1" ]]; then
debug "use_valgrind=1, long ticks while starting program"
for i in $(seq "${valgrind_start_ticks}"); do
tick
done
else
tick
fi
# NOTE: Below code checks for the subprocess starting up BUT this
# is a race condition as if the program finishes before the check,
# then it will spuriously report that the code did not start.
#
# if ! program_alive "$prog_key"; then
# printf "Failed to start program: %s\n" "${program_command[$key]}"
# return 1
# fi
}
# Sends an input line to a program on standard input using the
# pre-established FIFO for that program. The special message '<EOF>'
# will close the FIFO used for input which should give the program end
# of input. Checks that the program is alive before sending and if not
# prints and error message. Calls tick() before returning.
function program_send_input() {
key="$1"
msg="$2"
debug "program_send_input '$key' '$msg'"
if ! program_alive "$key"; then
printf "Can't send INPUT to dead program '%s' (%s)\n" "$key" "${program_command[$key]}"
return 1
fi
tofd=${program_input__fifo_fd[$key]} # extract the file descriptor for sending data to the child program
case "$msg" in
"<EOF>") # end of input
debug "EOF: closing fd $tofd"
exec {tofd}>&- # close fifo to child program
program_input__fifo_fd[$key]="CLOSED"
# debug "Closed"
# printf "Test message\n" >&$tofd
;;
*)
printf "%s\n" "$msg" >&$tofd # print to open file descriptor, possibly replace with direct reference to fifo name
;;
esac
tick
}
# Send a program a signal. Checks that the program is still alive
# before sending the signal. Calls tick() before returning.
function program_signal() {
debug "program_signal '$1' '$2'"
key="$1"
sig="$2"
if ! program_alive "$key"; then
printf "Can't send SIGNAL to dead program '%s' (%s)\n" "$prog_key" "${program_command[$prog_key]}"
return 1
fi
cmd="kill $prog_rest ${program_pid[$prog_key]}"
eval "$cmd"
tick
}
# Show the output for program with given key. Makes use of the output
# file found in the program_output_file[] array. Second argument is a
# filter to use, most often 'cat' to just show the output though other
# commands/scripts can be passed to adjust the output as desired.
function program_get_output() {
debug "program_get_output '$1' '$2'"
key="$1"
filter="$2"
outfile=${program_output_file[$key]}
debug "output for '$key' is file '$outfile' with filter '$filter'"
$filter "$outfile" # output the program by passing through given filter, usually 'cat'
return $?
}
# TODO: checks several output codes which do not have to do with
# valgrind, may want to convert this to 'program_failure_checks'
# instead.
# TODO: need to make compatible with the failures for the overall test
# though could rely on the output.
# Check the return code and valgrind output for the program for
# errors. Print any that appear. Second argument is a filter to use
# when displaying the valgrind output, most often 'cat' to just show
# the output though other commands/scripts can be passed to adjust the
# output as desired. This function is affected by several options
# that dictate Valgrind applications most notably 'use_valgrind'.
function program_check_failures() {
debug "program_check_failures '$1' '$2'"
# if [[ "$use_valgrind" == "0" ]]; then # check for use of valgrind first
# printf "Valgrind Disabled\n"
# return 0;
# fi
key="$1"
filter="$2"
progcmd="${program_command[$key]}"
retcode="${program_retcode[$key]}"
case "$retcode" in # inspect return code for errors
"$VALG_ERROR")
status="$FAIL_STATUS"
msg=""
msg+="Valgrind found errors for program ${key} ($progcmd : return code ${retcode[VALG_ERROR]})\n"
msg+="Valgrind output from '${program_valgfile[$key]}'\n"
msg+="#+BEGIN_SRC text\n" # org mode source block for vlagrind output
msg+=$(cat ${program_valgfile[$key]})
msg+="#+END_SRC\n"
fail_messages+=("$msg")
;;
"$RETCODE_TIMEOUT")
status="$FAIL_STATUS"
msg=""
msg+="${key} returned $retcode (TIMEOUT):\n"
msg+="Program '${program_command[$key]}' still running after $timeout seconds\n"
fail_messages+=("$msg")
;;
"$RETCODE_SEGFAULT")
status="$FAIL_STATUS"
msg=""
msg+="${key} returned $retcode (SIGSEGV):\n"
msg+="Program '${program_command[$key]}' signalled with segmentation fault by OS\n"
msg+="#+BEGIN_SRC text\n" # org mode source block for vlagrind output
msg+="Valgrind output from '${program_valgfile[$key]}'\n"
msg+=$(cat ${program_valgfile[$key]})
msg+="#+END_SRC\n"
fail_messages+=("$msg")
;;
*)
debug "checking return value '$retcode'"
if [[ "$checkreturn" == "1" && "$retcode" != "0" ]]; then
status="$FAIL_STATUS"
msg=""
msg+="${key} returned $retcode (non-zero) triggering failure"
fail_messages+=("$msg")
fi
;;
esac
if [[ "$use_valgrind" == "1" ]]; then # if valgrind is enabled, check its output
valgfile="${program_valgfile[$key]}"
debug "use_valgrind: $use_valgrind, valgfile: $valgfile"
$filter $valgfile > ${valgfile/.tmp/.filtered.tmp} # create a filtered version of the valgrind file to
mv $valgfile ${valgfile/.tmp/.unfiltered.tmp}
mv ${valgfile/.tmp/.filtered.tmp} $valgfile # remove spurious errors and use that output instead
debug "Checking Valgrind '$key' ($progcmd) filter '$filter' valgfile '$valgfile'"
if [[ "$valgrind_reachable" == "1" ]] && # and checking for reachable memory
! awk '/still reachable:/{if($4 != 0){exit 1;}}' ${valgfile};
then # valgrind log does not contain 'reachable: 0 bytes'
status="$FAIL_STATUS"
program_state[$key]="ReachErr"
msg=""
msg+="${key} MEMORY REACHABLE: Valgrind reports '${program_command[$key]}' has\n"
msg+="reachable memory, may need to add free() or fclose() before exiting\n"
msg+="\n"
msg+="Valgrind output from file '${program_valgfile[$key]}'\n"
msg+="--------------------\n"
msg+="$(cat ${program_valgfile[$key]})"
fail_messages+=("$msg")
# printf "FAILURE $msg\n"
fi
fi
return 0
}
# Handles a TESTY_MULTI command; run in a context where
# printing/echoing will not go to the screen but is instead redirected
# into a file which will the "actual" results for the test session to
# be compared to the "expected" results from the test specification
# file.
function handle_multi_command() {
debug "handle_multi_command: '$1'"
multi_line="$1"
multi_cmd="${multi_line%% *}" # extracts the first word on the line
multi_rest="${multi_line#* }" # extracts remainder of line
prog_key="${multi_rest%% *}" # key to identify program, only applicable to some lines
prog_rest="${multi_rest#* }" # remainder of program line, only applicable to some lines
debug "multi_cmd: '$multi_cmd' multi_rest: '$multi_rest'"
debug "prog_key: '$prog_key' prog_rest: '$prog_rest'"
case "$multi_cmd" in
"START")
program_start "$prog_key" "$prog_rest"
;;
"INPUT")
program_send_input "$prog_key" "$prog_rest"
;;
"SIGNAL") # 'SIGNAL server -15' == 'kill -15 ${program_pid["server"]}'
program_signal "$prog_key" "$prog_rest"
;;
"OUTPUT")
# cat "${program_output_file[$prog_key]}"
program_get_output "$prog_key" "$prog_rest"
;;
"OUTPUT_ALL")
for pk in "${program_keys[@]}"; do
printf "\n<testy> OUTPUT for %s\n" "$pk"
program_get_output "$pk" "$prog_key" # second arg is a filter to run output through
done
;;
"CHECK_FAILURES")
program_check_failures "$prog_key" "$prog_rest"
;;
"CHECK_ALL")
for pk in "${program_keys[@]}"; do
printf "<testy> CHECK_FAILURES for %s\n" "$pk"
program_check_failures "$pk" "$prog_key" # second arg is a filter to run all valgrind checks through
done
;;
"WAIT")
program_wait "$prog_key"
;;
"WAIT_ALL")
for pk in "${program_keys[@]}"; do
printf "<testy> WAIT for %s\n" "$pk"
program_wait "$pk"
done
;;
"SHELL") # run a shell command in testy, could remove files, sleep testy, etc.
eval "$multi_rest"
;;
*)
printf "TESTY FAILURE in handle_multi_command():\n" >/dev/stderr
printf "Unknown command '%s' in line '%s'\n" "$multi_cmd" "$linenum" >/dev/stderr
printf "Aborting testy\n" >/dev/stderr
exit 1
;;
esac
return $?
}
function diff_expect_actual() {
debug "diff_expect_actual '$1' '$2'"
expect_file="$1"
actual_file="$2"
actual_width=$(awk 'BEGIN{max=16}{w=length; max=w>max?w:max}END{print max}' "$actual_file")
expect_width=$(awk 'BEGIN{max=16}{w=length; max=w>max?w:max}END{print max}' "$expect_file")
col_width=$((actual_width > expect_width ? actual_width : expect_width))
# total_width=$((col_width*2 + 3)) # width to pass to diff as -W
total_width=$((actual_width + expect_width + 5)) # width to pass to diff as -W, tighter than previous
debug "actual_width $actual_width expect_width $expect_width"
debug "col_width $col_width total_width $total_width"
diff_file=$(printf "%s/%s-%02d-diff.tmp" "$resultraw" "$prefix" "$testnum")
diffcmd="$DIFF ${expect_file} ${actual_file}" # run standard diff to check for differences
diffresult=$(eval "$diffcmd")
diffreturn="$?" # capture return value for later tests
debug "diffreturn: $diffreturn"
{ # create diff file
printf "TEST OUTPUT MISMATCH: Side by Side Differences shown below \n"
printf "%s\n" "- Expect output in: $expect_file"
printf "%s\n" "- Actual output in: $actual_file"
printf "%s\n" "- Differing lines have a character like '|' and '<' in the middle"
printf "\n"
printf "%s\n" "#+BEGIN_SRC sbs-diff"
printf "%-${col_width}s %-${col_width}s\n" "==== EXPECT ====" "==== ACTUAL ===="
$SDIFF -W $total_width "$expect_file" "$actual_file"
printf "%s\n" "#+END_SRC"
printf "\n"
printf "%s\n" "--- Line Differences ---"
printf "%s" "$diffresult"
} >"$diff_file"
if [[ "$skipdiff" == "1" ]]; then # skipping diff
debug "Skipping diff (skipdiff=$skipdiff)"
diffreturn=0
elif [[ "$diffreturn" != "0" ]]; then
status="$FAIL_STATUS" # differences found, trigger failure
msg="$(cat "$diff_file")"
fail_messages+=("$msg")
fi
return $diffreturn
}
# Used when program is TESTY_MULTI. Run a test session where several
# programs must be started and coordinated at once. The session
# comprises a set of commands on when to start programs and what input
# should be given them at what time. The function is run in a context
# where 'read' will extract lines from the test session.
function run_test_multi_session() {
if [[ "$use_valgrind" == 1 ]]; then
checkdep_fail "valgrind" # only check for valgrind if the test requires it
fi
# Set up the global arrays used in multi testing to track
# input/output/state for all programs involved in the test.
indexed_arrays=(
program_keys # each program has a unique key like 'server'
)
assoc_arrays=(
program_pid # pid of the multiple programs used during the test
program_state # 1 for program still running, 0 for program complete/killed
program_name # name of programs, 1st word in command, useful for pkill
program_command # full command for each program
program_input__fifo # file names for fifos for writing to the program
program_input__fifo_fd # fds for the fifos for writing to the clients
program_output_file # names of files for data coming from the files
program_retcode # return codes for programs
program_valgfile # valgrind output files for programs
)
for a in "${indexed_arrays[@]}"; do
unset "$a" # remove any existing binding
declare -g -a "$a" # declare as -g global, -a indexed array
done
for a in "${assoc_arrays[@]}"; do
unset "$a" # remove any existing binding
declare -g -A "$a" # declare as -g global, -A Associative array
done
# Set up the testing environment
mkdir -p "$resultdir" # set up test results directory
mkdir -p "$resultraw" # set up raw directory for temporary files/raw output
result_file=$(printf "%s/%s-%02d-result.tmp" "$resultdir" "$prefix" "$testnum")
actual_file=$(printf "%s/%s-%02d-actual.tmp" "$resultraw" "$prefix" "$testnum")
expect_file=$(printf "%s/%s-%02d-expect.tmp" "$resultraw" "$prefix" "$testnum")
status="$PASS_STATUS" # initial status, change to 'FAIL' if things go wrong
fail_messages=() # array accumulating failure messages
session_beg_line=$((linenum + 1)) # mark start of session to allow it to be extracted
# main input loop to read lines and handle them
while read -r; do # read a line from the test session
updateline
debug "$linenum: $line"
case "$first" in
"#+END_SRC") # end of test, break out
debug "^^ end of testing session"
break
;;
"$prompt")
debug "^^ handle_multi_command"
printf "%s\n" "$line" # print test line so it appears in 'actual' output
handle_multi_command "$rest"
;;
"#" | "")
debug "^^ comment"
printf "%s\n" "$line" # print comment so it appears in 'actual' output
;;
*) # other lines are test output which should be generated by the programs
debug "^^ expected output"
;;
esac
done >"${actual_file}" # redirect output of printf/echo into actual output
session_end_line=$((linenum - 1)) # note the line session ends on to enable #+TESTY_RERUN:
for key in "${program_keys[@]}"; do # clean up files and other artifacts for each program
if [[ "${program_state[$key]}" == "Running" ]]; then
program_wait "$key"
fi
done >>"${actual_file}" # capture failures of any unresponsive_programs
# extract expected output from test file, filter #+TESTY_ , store result in expect_file
$SEDCMD -n "${session_beg_line},${session_end_line}p" <"$specfile" |
grep -v '^#+TESTY_' \
>"${expect_file}"
diff_expect_actual "$expect_file" "$actual_file" # diff expect/actual output, sets 'status'
# Calculate the Width of various table fields tight
cmd_str="COMMAND" # start with width of headings for several columns
max_cmd_width=${#cmd_str}
out_file="OUTPUT FILE"
max_out_width=${#out_file} # valgrind files now in same column as output files
for key in "${program_keys[@]}"; do
cmd_str="${program_command[$key]}"
cmd_width=${#cmd_str}
if ((cmd_width > max_cmd_width)); then
max_cmd_width=$cmd_width
fi
out_file="${program_output_file[$key]#${resultdir}/}" # include raw/ path to make it easier to recognize file location
out_width=${#out_file}
debug "File '$outfile' with width $outwidth"
if ((out_width > max_out_width)); then
max_out_width=$out_width
debug "File '$out_file' has new max width $out_width"
fi
out_file="${program_valgfile[$key]#${resultdir}/}" # print output file/valg files in the same column
out_width=${#out_file} # so assign the same max width as out files
if (( $out_width > $max_out_width )); then
max_out_width=$out_width
fi
done
cw=$max_cmd_width
ow=$max_out_width
vw=$max_valg_width
{
printf '(TEST %d) %s\n' "$testnum" "$test_title"
printf 'COMMENTS:\n'
printf "%b\n" "${comments}"
printf 'program: %s\n' "$program"
printf '\n'
# printf '%s\n' '- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -'
printf '%s\n' '-----------------------------------------------------------------------------------------------'
printf "* Summary Program Information\n"
printf "\n"
printf "| %-8s " "KEY"
printf "| %-${cw}s " "COMMAND"
printf "| %3s " "RET"
printf "| %-10s " "STATE"
printf "| %-${ow}s " "OUTPUT/VALGRIND FILES"
printf "| \n"
for key in "${program_keys[@]}"; do
printf "| %-8s " "$key"
printf "| %-${cw}s " "${program_command[$key]}"
printf "| %3s " "${program_retcode[$key]}"
printf "| %-10s " "${program_state[$key]}"
printf "| %-${ow}s " "${program_output_file[$key]#${resultdir}/}" # local path for out file under raw/...
printf "| \n"
printf "| %-8s " "" # print a blank row ending with the valgrind file
printf "| %-${cw}s " ""
printf "| %3s " ""
printf "| %-10s " ""
printf "| %-${ow}s " "${program_valgfile[$key]#${resultdir}/}" # local path for valg file under raw/...
printf "| \n"
done
printf "\n"
if [[ "$status" == "$PASS_STATUS" ]]; then # test passed
printf "ALL OK\n"
else # test failed
debug "Fail with status '$status'"
nfails="${#fail_messages[@]}"
printf "%d FAILURES FOUND\n\n" "$nfails"
fidx=1
for msg in "${fail_messages[@]}"; do
# printf '%s\n' '- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -'
printf '%s\n' '-----------------------------------------------------------------------------------------------'
printf "* FAILURE %d\n" "$fidx"
printf "%b\n" "${msg}"
printf "\n"
((fidx++))
done
printf "\n"
fail_files+=("${result_file}") # test failed, add to files to show later
status="FAIL -> results in file '$result_file'" # show results file on command line
fi
} >"${result_file}"
# TODO: may need to clean up files that were created during
# testing at this point and are not needed for results
return 0
}
# Handle a #+BEGIN_QUoTE region which will create a file with the
# specified name in it and place text in the region in file. Useful
# for creating small-ish input files on the fly during tess. This
# output is also echoed into the comments for the test results.
function catfile_session() { # for #+BEGIN_QUOTE some-file-name.txt
debug "catfile_session: $line"
outfilename="${line##* }" # extract last token from the line as file name
debug "outfilename: $outfilename"
comments="${comments}${line}\n"
while read -r; do # read a line from the test session
updateline
comments="${comments}${line}\n"
if [[ "$first" == '#+END_QUOTE' ]]; then
break
fi
printf "%s\n" "$line"
done > "$outfilename"
}
################################################################################
# Run a Test Session for a Single Program
#
# Sets up a test session which is denoted by the #+BEGIN_SRC/#+END_SRC
# tags in the input file. Will set the 'status' variable before
# exiting to indicate whether the test passes or fails. Influenced by
# many of the run variables including
# - program
# - tag
# - prompt
#
# The function is run in a context where 'read' will extract lines
# from the test session.
function run_test_session() {
mkdir -p "$resultdir" # set up test results directory
mkdir -p "$resultraw"
result_file=$(printf "%s/%s-%02d-result.tmp" "$resultdir" "$prefix" "$testnum")
actual_file=$(printf "%s/%s-%02d-actual.tmp" "$resultraw" "$prefix" "$testnum")
expect_file=$(printf "%s/%s-%02d-expect.tmp" "$resultraw" "$prefix" "$testnum")
valgrd_file=$(printf "%s/%s-%02d-valgrd.tmp" "$resultraw" "$prefix" "$testnum")
rm -f "${actual_file}" "${expect_file}" "${result_file}" "${valgrd_file}"
if [[ "$use_valgrind" == 1 ]]; then
checkdep_fail "valgrind" # only check for valgrind if the test requires it
VALGRIND="${VALGRIND_PROG} ${VALGRIND_OPTS} --log-file=${valgrd_file}"
else
VALGRIND=""
fi
fromprog_file=$(printf "%s/%s-%02d-fromfile.tmp" "$resultraw" "$prefix" "$testnum") # may wish to alter this to honor TMPDIR
toprog_fifo=$(printf "%s/%s-%02d-tofifo.tmp" "$resultraw" "$prefix" "$testnum") # set up communication with the program being tested
if [[ -n "$TMPFIFOS" ]]; then
toprog_fifo=$(printf "%s/%s-%02d-tofifo.tmp" "$TMPFIFOS" "$prefix" "$testnum") # use /tmp instead, likely due to Windows/WSL
fi
debug "toprog_fifo: $toprog_fifo"
debug "fromprog_file: $fromprog_file"
rm -f "${toprog_fifo}" "${fromprog_file}" # remove just in case
mkfifo "$toprog_fifo" # create the fifos
# RUN THE PROGRAM
#
# - timeout will kill the program after a certain duration
# - stdbuf disables buffering and prevents stalled output problems
# - valgrind may be turned on to check for memory errors
# - input is read from a fifo from the specfile
# - output is directed to a file
cmd="$TIMEOUTCMD $timeout $STDBUF $VALGRIND $program <${toprog_fifo} &> ${fromprog_file} &"
debug "running: '$cmd'"
eval "$cmd" # eval is required due to the complex redirections with < and >
pid=$!
debug "child pid: $pid"
# open to after running the program or testy will stall
exec {to}>"${toprog_fifo}" # open connection to fifo for writing
debug "to fd: $to"
status="$PASS_STATUS" # initial status, change to 'FAIL' if things go wrong
fail_messages=() # array accumulating failure messages
all_input=() # array accumulating input fed to program
session_beg_line=$((linenum + 1))
eof_set=0
# LOOP to feed lines of input to the program, output is fed to a
# file, modified later if needed to re-add the prompt
while read -r; do # read a line from the test session
updateline
debug "$linenum: $line"
case "$first" in
"#+END_SRC") # end of test, break out
debug "^^ end of testing session"
break
;;
"#+TESTY_EOF:") # end of input, remaining session is output
eof_set=1
debug "^^ eof_set=1"
;;
"$prompt") # test input, feed to program
if [[ "$eof_set" == "0" ]]; then
input="$rest"
all_input+=("$input") # append to all_input array for later processing
debug "^^ sending input"
printf "%s\n" "$input" >&$to # send input after prompt to program, printf in this way preserves embedded newlines
else
debug "^^ ignoring prompt after EOF"
fi
;;
*) # other lines are test output
debug "^^ expected output"
;;
esac # DONE with test input, either pass or fail
done
session_end_line=$((linenum - 1)) # note the line session ends on to enable #+TESTY_RERUN:
debug "session lines: beg $session_beg_line end $session_end_line"
debug "closing to fifo fd ${to}"
exec {to}>&- # closes to fifo
debug "waiting on finished child"
wait $pid &>/dev/null # wait on the child to finish
retcode="$?" # capture return code from program run
debug "wait returned: $retcode"
debug "removing to fifo"
rm -f "${toprog_fifo}" # ${toprog_fifo}
case "$retcode" in # inspect return code for errors
"$VALG_ERROR")
status="$FAIL_STATUS"
fail_messages+=("FAILURE($retcode): Valgrind detected errors")
;;
"$RETCODE_TIMEOUT")
status="$FAIL_STATUS"
fail_messages+=("FAILURE($retcode) due to TIMEOUT: Runtime exceeded maximum of '$timeout'")
;;
"$RETCODE_SEGFAULT")
status="$FAIL_STATUS"
fail_messages+=("FAILURE($retcode) due to SIGSEGV (segmentation fault) from OS")
;;
*)
debug "checking return value '$retcode'"
if [[ "$checkreturn" == "1" && "$retcode" != "0" ]]; then
status="$FAIL_STATUS"
msg="program returned $retcode (non-zero) triggering failure"
fail_messages+=("$msg")
skipdiff=0 # hack to show diffed output on non-zero return, possibly from valgrind subprocess
fi
esac
if [[ "$use_valgrind" == "1" ]] && # if valgrind is on
[[ "$valgrind_reachable" == "1" ]] && # and checking for reachable memory
! awk '/still reachable:/{if($4 != 0){exit 1;}}' "${valgrd_file}";
then # valgrind log does not contain 'reachable: 0 bytes'
status="$FAIL_STATUS"
fail_messages+=("FAILURE: Valgrind reports reachable memory, may need to add free() or fclose()")
fi
# ADDING IN PROMPTS TO ECHOED INPUT
#
# NOTE: The code below handles adding prompts to input lines that
# are echoed without it. Originally was trying to do this with
# sed or awk but the quoting becomes a huge mess: any input lines
# with special characters like $ or " need to be escaped leading
# to huge headaches. The shell string equality = operator is
# actually cleaner here. The below uses the shell
# directly. Output is redirected to an open FD to prevent needing
# constantly re-open the file for appending (could alternatively
# do this with { } construct). This approach can be fooled: if an
# output line matches an input line, the prompt may be added at
# the wrong spot.
if [[ "$echoing" == "input" ]]; then # program may only echo input necessitating adding prompts to output
idx=0 # index for input line
exec {mod}>"${fromprog_file}.mod"
while read -r || [[ "$REPLY" != "" ]]; do # read from output file into default REPLY var, second condition catches last line which may not end with a newline char
if ((idx < ${#all_input[@]})) && # still in bounds for input lines
[[ "${all_input[idx]}" == "$REPLY" ]]; # input line matches the program output
then
REPLY="$prompt $REPLY" # add the prompt to this line
((idx++)) # move to the next input to look for
debug "added prompt to input $idx: $REPLY"
fi
printf '%s\n' "$REPLY" >&$mod # output the (un)modified line into the modified file
done <"${fromprog_file}" # reading from the original output file
exec {mod}>&- # close the modified file
mv "${fromprog_file}.mod" "${fromprog_file}" # copy modified file back to original
fi
if [[ "$post_filter" != "" ]]; then # use a filter to post-process the output
debug "running post filter '$post_filter'" # use of 'eval' for "grep -v "stuff here"' filters w/ spaces in command
cat "${fromprog_file}" | eval ${post_filter} >"${fromprog_file}.tmp"
mv "${fromprog_file}.tmp" "${fromprog_file}"
fi
# To avoid confusion, replace message from timeout program with
# easier to interpret Segfault Message
$SEDCMD -i'' 's/timeout: the monitored command dumped core/Segmentation Fault/' "${fromprog_file}"
mv "${fromprog_file}" "${actual_file}" # copy temp file to final destination
# extract expected output from test file, filter #+TESTY_ , store result in expect_file
range="${session_beg_line},${session_end_line}p" # range of lines for sed to print
if ((session_beg_line > session_end_line)); then # in sed, if start > end then the
range="" # start line still gets printed so
fi # set range to nothing for empty output
$SEDCMD -n "$range" <"$specfile" |
grep -v '^#+TESTY_' \
>"${expect_file}"
# Try to compute the width of expected/actual outputs to make the
# side-by-side diff as narrow as possible. 'diff -y -W' is a bit
# funky as it tries to split the side-by-side comparison into evey
# column widths. The below computation finds the maximum width of
# the two compared files and doubles it adding 3 for the middle
# diff characters. This may result in an grossly wide display if
# the left EXPECT column is narrow while the right ACTUAL column
# wide. Later code filters to remove extraneous whitespace from
# the left column.
actual_width=$(awk 'BEGIN{max=16}{w=length; max=w>max?w:max}END{print max}' "$actual_file")
expect_width=$(awk 'BEGIN{max=16}{w=length; max=w>max?w:max}END{print max}' "$expect_file")
col_width=$((actual_width > expect_width ? actual_width : expect_width))
total_width=$((col_width * 2 + 3)) # width to pass to diff as -W
debug "actual_width $actual_width"
debug "expect_width $expect_width"
debug "col_width $col_width"
debug "total_width $total_width"
diffcmd="$DIFF ${expect_file} ${actual_file}" # run standard diff to check for differences
diffresult=$(eval "$diffcmd")
diffreturn="$?" # capture return value for later tests
debug "diffresult: $diffresult"
debug "diffreturn: $diffreturn"
if [[ "$skipdiff" == "1" ]]; then # skipping diff
debug "Skipping diff (skipdiff=$skipdiff)"
diffreturn=0
elif [[ "$diffreturn" != "0" ]]; then
status="$FAIL_STATUS"
fail_messages+=("FAILURE: Output Mismatch at lines marked")
fi
if [[ "$status" == "$PASS_STATUS" ]]; then # test passed
debug "NORMAL cleanup" # normal finish
debug "Checking child status with kill -0"
kill -0 $pid >&/dev/null # check that the child is dead, return value 1
debug "kill returned: $?"
else
debug "FAILURE cleanup" # test failed for some reason
{ # begin capturing output for results file
printf '* (TEST %d) %s\n' "$testnum" "$test_title"
printf 'COMMENTS:\n'
printf "%b" "${comments}"
printf '** program: %s\n' "$program"
printf "\n"
printf '** --- Failure messages --- \n'
for msg in "${fail_messages[@]}"; do # iterate through failure messages
printf "%s\n" "- $msg"
done
printf "\n"
if [[ "$diffreturn" != "0" ]]; then # show differences between expect and actual
printf "%s\n" "** --- Side by Side Differences ---"
printf "%s\n" "- Expect output in: $expect_file"
printf "%s\n" "- Actual output in: $actual_file"
printf "%s\n" "- Differing lines have a character like '|' '>' or '<' in the middle"
printf "%s\n" "#+BEGIN_SRC sbs-diff"
printf "%-${col_width}s %-${col_width}s\n" "==== EXPECT ====" "==== ACTUAL ===="
$SDIFF -W $total_width "$expect_file" "$actual_file"
printf "%s\n" "#+END_SRC"
printf "\n"
printf "%s\n" "** --- Line Differences ---"
printf "%s\n" "$diffresult"
printf "\n"
fi
if [[ "$use_valgrind" == "1" ]]; then # show valgrind log if enabled and test failed
printf "%s\n" "--- Valgrind Log from: $valgrd_file ---"
cat "$valgrd_file"
printf "\n"
fi
} &>"${result_file}" # end of results file output
# The below eliminate extra spaces in diff results mostly for
# the left EXPECT column that would make the output very wide.
# This is a bit risky as it may eliminate some real expected
# output so take care if the output is very wide. This is
# mitigated by NOT changing anything in the first
# $actual_width columns of the output file.
if ((actual_width - expect_width > 10)); then
extra_space_width=$((actual_width - expect_width))
extra_space_width=$((extra_space_width - 5))
debug "Eliminating $extra_space_width spaces from result file"
$SEDCMD -i -E "s/(.{$expect_width})[ ]{$extra_space_width}/\1 /" "$result_file"
fi
fail_files+=("${result_file}") # test failed, add to files to show later
status="FAIL -> results in file '$result_file'"
fi
comments="" # clear comments for next session
return 0
}
function signal_exit (){ # function to run on exit signals
printf '\ntesty was signaled: Exiting\n' > /dev/stderr
exit 1
}
################################################################################
# BEGIN main processing
unset BASH_ENV # ensure subshells don't spit out extra gunk
suppress_signals=(SIGSEGV SIGFPE SIGILL SIGBUS) # trap these signals to suppress child processes
for sig in ${suppress_signals[@]}; do # from generating bash error messages when they
trap "" $sig # receive these signals; testy should not generate
done # these signals but child processes might
exit_signals=(SIGINT SIGTERM) # signals that will cause testy to exit
for sig in ${exit_signals[@]}; do
trap "signal_exit" $sig
done
funcs=$(declare -x -F | awk '{print $3}') # eliminate any exported functions in bash
for f in $funcs; do # as these are output with the -v option
unset -f "$f"
done
if [[ "$#" -lt 1 ]]; then # check for presence of at least 1 argument
printf "usage: testy <testspec> [testnum]\n"
printf " testy --help\n"
exit 1
fi
# Command line argument processing
specfile=$1 # gather test file
shift # shift test file off the command line
alltests="$*" # remaining args are tests to run
debug "Testing $specfile"
debug "alltests='$alltests"
if [[ "$specfile" == "--help" ]]; then # check for --help option
printf "%s\n" "$usage" # print usage and exit
exit 0
fi
if [[ ! -r "$specfile" ]]; then # check specfile exists / readable
printf "ERROR: could not open '%s' for reading\n" "$specfile" >/dev/stderr
exit 1
fi
deps="timeout stdbuf awk $SEDCMD grep diff" # check for baseline necessary tools
for dep in $deps; do
checkdep_fail "$dep"
done
rm -f ./test-fifo.fifo /tmp/test-fifo.fifo # Test FIFO creation, often fails for Windows file systems on WSL, can
if ! mkfifo ./test-fifo.fifo &> /dev/null ; then # FIFOs in current directory?
debug "Can't create fifos in $PWD"
if ! mkfifo /tmp/test-fifo.fifo &>/dev/null; then # FIFOs in /tmp?
printf "ERROR: Can't create FIFOs in %s or /tmp; Bailing out\n" "$PWD"
rm -f ./test-fifo.fifo /tmp/test-fifo.fifo
exit 1
else # use FIFOS in /tmp
TMPFIFOS="/tmp"
debug "Local dir $PWD can't handle FIFOs, Creating FIFOs in /tmp"
fi
fi
rm -f ./test-fifo.fifo /tmp/test-fifo.fifo
##################################################
# first processing loop: read whole file into testdata array which will
# contain the text of each test. Record ONLY the start/end lines of
# each test to be used later. Other side effects: evaluate any global
# #+TESTY: expressions, calculate the widest test title width for nice
# display later.
eval_testy_expr=1 # set to 0 after getting into the first test
test_beg_line=(-1)
test_end_line=(-1)
testnum=0 # current test number
linenum=0
while read -r; do # read from test file, -r to prevent \-escaped chars
updateline
debug "$linenum: $line\n"
case "$first" in
"*")
debug "^^ Test Start"
eval_test_expr=0 # in a test, wait to evaluate #+TESTY: expr until during test
if ((testnum > 0)); then # if not the first test
endline=$((linenum - 1))
test_end_line+=("$endline")
beg=${test_beg_line[testnum]}
end=${test_end_line[testnum]}
debug "Test $testnum beg $beg end $end"
fi
((testnum++)) # reset and start collecting text for the new test
test_beg_line+=("$linenum")
if ((${#rest} > TEST_TITLE_WIDTH)); # calculate maximum width of any title
then
TEST_TITLE_WIDTH=${#rest}
fi
;;
"#+TESTY:") # evaluate global expressions
if [[ "$eval_testy_expr" == "1" ]]; then
debug "Evaluating '$rest'"
eval "$rest"
if [[ $? != 0 ]]; then # check syntax error, print error message
printf "%s: line %d: Syntax error in test file, bailing out\n" "$specfile" "$linenum"
exit 1 # this format used to be amenable to emacs compilation/jump to error
fi
fi
testtext="$testtext\n$line" # append line to current test text as it may be a local test option
;;
"#+TITLE:" | "#+title:")
global_title="$rest"
debug "^^ setting global_title"
;;
*)
debug "^^ Ignoring line in first pass"
;;
esac
done <"$specfile"
endline=$((linenum)) # append the last test end
test_end_line+=("$endline")
beg=${test_beg_line[testnum]}
end=${test_end_line[testnum]}
debug "Test $testnum beg $beg end $end"
totaltests=$testnum # set the total number of tests read from the file
# Debug output
for i in $(seq "$testnum"); do
debug "-----TEST $i: beg ${test_beg_line[i]} end: ${test_end_line[i]} -----"
while read -r; do # iterate over all lines of test
debug ":TEST $i: $REPLY"
done <<<"$($SEDCMD -n "${test_beg_line[i]},${test_end_line[i]}p" "$specfile")"
done
##################################################
# Second loop: run tests
if [[ -z "$alltests" ]]; then # no individual tests specified on the command line
alltests=$(seq "$totaltests") # so run all tests
fi
ntests=$(wc -w <<<"$alltests") # count how many tests will be run
checktests="$alltests" # filter out tests not in range
alltests=""
for t in $checktests; do
if ((t <= totaltests)); then
alltests+="$t "
else
printf "WARNING: skipping out of range test %s\n" "$t"
fi
done
ntests=$(wc -w <<<"$alltests") # update cout of total tests
if [[ "$ntests" == "1" && "$SHOW" == "" ]]; then
debug "Running single test, setting SHOW=1 to display single test results"
SHOW=1
fi
testcount=0
failcount=0
# Print header info
printf "============================================================\n"
if [[ "$global_title" == "" ]]; then
printf "== testy %s\n" "$specfile"
else
printf "== $specfile : %s\n" "$global_title"
fi
printf "== Running %d / %d tests\n" "$ntests" "$totaltests"
for testnum in $alltests; do # Iterate over all tests to be run
((testcount++)) # increment # of tests attempted
reset_options
comments="" # initialize comments
linenum=$((test_beg_line[testnum] - 1))
debug ":TEST $testnum: START at line $linenum"
while read -r; do # iterate over all lines of test
updateline
debug "$linenum: $line"
case "$first" in
"*") # usually first line with title of the test
test_title="$rest"
debug "test_title: $test_title"
;;
"#+TESTY:") # eval some code to set options
debug "evaluating '$rest'"
eval "$rest"
;;
"#+BEGIN_SRC") # test session starting
debug ":TEST $testnum: Begin testing session"
if [[ "$program" == "TESTY_MULTI" ]]; then
run_test_multi_session
else
run_test_session
fi
if [[ "$status" != "$PASS_STATUS" ]]; then
((failcount++)) # test failed, bail out of this test
break
fi
;;
"#+BEGIN_QUOTE") # create a file with the quoted material in it
catfile_session
;;
"#+TESTY_RERUN:") # eval some code to set options
old_linenum=$linenum
beg=$((session_beg_line)) # #+BEGIN_SRC line
end=$((session_end_line + 1)) # #+END_SRC line
linenum=$((beg - 1))
debug "^^ Re-running session on lines $beg to $end"
if ((beg == 0)); then
{
printf "ERROR in test %s with directive '#+TESTY_RERUN'\n" "$testnum"
printf "Alas, testy does not support rerunning a test that hasn't already been run\n"
printf "Try running all tests instead\n"
} >/dev/stderr
exit 1
fi
if [[ "$program" == "TESTY_MULTI" ]]; then
run_test_multi_session <<<"$($SEDCMD -n "${beg},${end}p" "$specfile")"
else
run_test_session <<<"$($SEDCMD -n "${beg},${end}p" "$specfile")"
fi
debug "Done re-running session on lines $beg to $end"
linenum=$old_linenum
if [[ "$status" != "$PASS_STATUS" ]]; then # this block should be here, right?
((failcount++)) # test failed, bail out of this test
break
fi
;;
*) # any other lines are comments associated with a session
if [[ "$comments" != "" ]] || [[ "$line" != "" ]]; then
debug "^^ comment" # ignore leading blank lines in comments
comments="${comments}${line}\n"
fi
;;
esac
done <<<"$($SEDCMD -n "${test_beg_line[testnum]},${test_end_line[testnum]}p" "$specfile")"
# report the final status of this test
printf "%-3s %-${TEST_TITLE_WIDTH}s : %s\n" "${testnum})" "$test_title" "$status"
done
########################################
# Final Output
passcount=$((testcount - failcount)) # calculate number of tests passed
if [[ "$REPORT_FRACTION" == "1" ]]; then # reporting fraction of tests passed
passcount=$(awk "BEGIN{printf(\"%0.2f\n\",$passcount / $testcount)}")
testcount="1.00"
fi
printf "============================================================\n"
printf "RESULTS: %s / %s tests passed\n" "$passcount" "$testcount"
debug "SHOW: $SHOW"
if [[ "$SHOW" == "1" && "${#fail_files[@]}" -gt 0 ]]; then # show failure results if requested
printf "\n\n"
printf "============================================================\n"
printf "== FAILURE RESULTS\n"
# printf "%s\n" "----------------------------------------"
for f in "${fail_files[@]}"; do # iterate over all failure files outputting them
printf "============================================================\n"
cat "$f"
done
fi