TPS Testing

For running all the tests, enter this directory and type:

./run test-all.sh

Invoking the script test-all.sh in each directory (with the run command) runs all the tests in that directory and its subdirectories.

A large portion of tests are generated by capturing the behavior of the code (when it seems to work correctly) with the purpose of regression tests. These are called "captured" tests. In order to regenerate those tests, run:

./run capture-all.sh

Running the script capture-all.sh in a directory, (re)generates all the captured tests in that directory and its subdirectories.

You can use run.bat instead of ./run in windows (with msys). Though, the tool runs slower in windows.

By default, some details about the process of running/capturing the tests is printed. Adding the option -q or --quiet after run prevents writing these information. Examples:

./run -q test-all.sh
./run --quiet capture-all.sh

Important testing commands and concepts

The sandbox directory

Path: sandbox

All the temporary files created throughout the test executions are stored in the sandbox directory. This directory is ignored by git.

Note: Due to the current design and implementation of the testing infrastructure, you cannot run two test_all.sh (or capture_all.sh) scripts concurrently, as they all use the same sandbox directory.

The stage directory

Path: sandbox/stage

The commands being tested are (usually) run inside the stage directory. The stage directory is needed to be initialized before executing the command under test.

A path is considered relative to the stage directory only if it does not start with "/".

Command stage_dir

stage_dir <a-given-dir>

Initializes the stage directory by (deleting its current files and then) copying directory <a-given-dir>.

Command stage_an_empty_dir

stage_an_empty_dir

Initializes the stage directory by (deleting its current files and then) creating an empty directory.

Command run_in_stage

run_in_stage <command> [command arguments...]

Executes the given command line in the stage directory.

Command expect_exec

A major tool in the tests is the command expect_exec, which runs a command and verifies its expected behavior. The command is (usually) executed in the stage directory.

expect_exec <options...> <command> [command arguments...]

Options:

• -d <a-dir>

  Sets the current working directory of the command (relative to the stage directory). If not specified, it checks the environment variable EXEC_WORKING_DIRECTORY (which can be set/unset respectively with set_exec_cwd and unset_exec_cwd). If the environment variable is not specified either, then the command is run in the root of the stage directory.

• -abs

  Makes the command path absolute. Default: true for and only for commands ending with .sh and .py. Files ending with .sh are automatically run by bash. Files ending with .py are automatically run by ${PYTHON}, python3, or python.

• -r <an-integer>

  -rnz

  -rignore

  These flags specify the expected exit code of the command execution. Flag -r specifies the exact value of the expected exit code. Flag -rnz specifies that the exit code is expected to be a nonzero value, but its exact value is not important. Flag -rignore specifies that the exit code is not important at all and should be totally ignored. If none of these flags are specified, a zero exit code is expected.

• -oignore

  -oempty

  -o <a-file-path>

  -oh <a-text-parameter>

  -oH <a-text-parameter>

  -oh2 <line-1> <line-2>

  -oH2 <line-1> <line-2>

  -oh<an-integer-k> <line-1> <line-2> ... <line-k>

  -oH<an-integer-k> <line-1> <line-2> ... <line-k>

  These flags specify the expected standard output of the command execution. It is mandatory to use exactly one of them.

  Flag -oignore specifies that the output is not important at all and should be ignored.

  Flag -oempty specifies that the output is expected to be empty.

  Flag -o specifies that the output is expected to be exactly the same as the contents of the given file <a-file-path>.

  Flags -oh and -oH specify that the output is expected to be the same as the given parameter <a-text-parameter>. If -oH is used, the output must be exactly the same as the given parameter. On the other hand, if -oh is used, the output must have an extra new-line character after <a-text-parameter>.

  Flags -oh2 and -oH2 work respectively similar to -oh and-oH, with the difference that the output is expected to have the two specified lines. The last line of output is expected to end with the new-line character if -oh is used.

  Flags -oh<an-integer-k> and -oH<an-integer-k> are respectively the generalizations of -oh2 and -oH2, expecting the output to have the $k$ specified lines.

  So, -oempty is equivalent to -oH "", and -oh "xyz" is equivalent to -oH2 "xyz" "".

• -eignore

  -eempty

  -e <a-file-path>

  -eh <a-text-parameter>

  -eH <a-text-parameter>

  -eh<an-integer-k> <line-1> <line-2> ... <line-k>

  -eH<an-integer-k> <line-1> <line-2> ... <line-k>

  These flags specify the expected standard error of the command execution. It is mandatory to use exactly one of them. Their definitions are quite similar to the -o... flags.

• -iempty

  -i <a-file-path>

  -ih <a-text-parameter>

  -iH <a-text-parameter>

  -ih<an-integer-k> <line-1> <line-2> ... <line-k>

  -iH<an-integer-k> <line-1> <line-2> ... <line-k>

  These flags specify the standard input provided for the command execution.

  If none of these flags is given or flag -iempty is used, the standard input will be redirected from /dev/null.

  Flag -i redirects the standard input from the given file <a-file-path>.

  Flag -iH sets the given string value <a-text-parameter> as the exact contents of the input. On the other hand, with flag -ih, the contents of the input is set to the given parameter <a-text-parameter> plus an extra new-line character.

  Flags -ih... and -iH... are generalizations of -ih and -iH, similar to the corresponding -o... flags.

• -fu <probed-file-path>

  -fempty <probed-file-path>

  -fempty-dir <probed-file-path>

  -f <probed-file-path> <path-to-a-given-file-or-directory>

  -fh <probed-file-path> <a-text-parameter>

  -fH <probed-file-path> <a-text-parameter>

  -fh<an-integer-k> <probed-file-path> <line-1> <line-2> ... <line-k>

  -fH<an-integer-k> <probed-file-path> <line-1> <line-2> ... <line-k>

  These flags specify the expected state of the file or directory <probed-file-path> after the execution of the command. The path <probed-file-path> is considered relative to the stage directory. These flags can be used more than once (for probing different files or directories).

  Flag -fu specifies that no file or directory is expected to be in <probed-file-path>.

  Flag -fempty specifies that an empty file is expected to be in <probed-file-path>.

  Flag -fempty-dir specifies that an empty directory is expected to be in <probed-file-path>.

  Flag -f specifies that the file or directory <probed-file-path> is expected to have exactly the same structure as <path-to-a-given-file-or-directory> (compared with diff -rq).

  Flags -fh... and -fH... specify the expected state of file <probed-file-path> similar to the corresponding -o... flags.

• -vu <probed-var-name>

  -vs <probed-var-name> <a-string>

  -va <probed-var-name> <an-integer-k> <item-1> <item-2> ... <item-k>

  -vf <probed-var-name> <a-given-var-dump-file>

  These flags specify the expected state of the variable <probed-var-name> after the execution of the command. These flags can be used more than once (for probing different variables).

  Flag -vu specifies that the variable <probed-var-name> is expected to be undefined.

  Flag -vs specifies that the variable <probed-var-name> is expected to be defined as a string variable with value <a-string>.

  Flag -va specifies that the variable <probed-var-name> is expected to be defined as an array of length <an-integer-k> with elements <item-1>, <item-2>, ..., <item-k>.

  Flag -vf specifies that the variable <probed-var-name> is expected to have exactly the same type and value as the variable "variable_value" defined (dumped) in file <a-given-var-dump-file>.

Capturing commands

Command capture_exec

The major tool in capturing the tests is the command capture_exec, which runs a command, captures its behavior, and generates an expect_exec command. The command is (usually) executed in the stage directory.

capture_exec <captured-test-key> [options...] <command> [command arguments...]

The parameter <captured-test-key> is a unique string specifying the captured test. The options are similar to expect_exec. The expectation options of the generated expect_exec command are specified by capture_exec unless explicitly specified as an option for capture_exec. Additionally, the following options are available for capture_exec:

• -fc <probed-file-path>

  Captures the state of the file or directory <probed-file-path> after the execution of the captured command and adds the corresponding -f... options in the generated expect_exec command line.

• -vc <probed-var-name>

  Captures the state of the variable <probed-var-name> after the execution of the captured command and adds the corresponding -v... options in the generated expect_exec command line.

Command capture_run

A useful tool in capturing scripts is the command capture_run, which runs a command, and also generates the same command as a command to be run in the captured tests.

capture_run <command> [command arguments...]

Command capture_run_in_stage

capture_run_in_stage <command> [command arguments...]

Works similar to capture_run, but executes the given command line in the stage directory.

Implementation notes

The prefix _TT_ in names of variables and functions stands for TPS TESTING. This prefix is added to prevent name clashes between the test codes and main codes.