[RFE] Reduce amount of boilerplate code needed by the beakerlib library

[i386x]

Story

As a test writer using beakerlib I want just to focus on tests. I want a tool that will generate beakerlib's boilerplate code for me.

Detail

Actually tests in beakerlib have the following structure (generated by tmt or beakerlib-wizard):

# (1) Include beakerlib
. /usr/share/beakerlib/beakerlib.sh || exit 1

# (2) Define what should be checked by `rlAssertRpm`
PACKAGES=...
REQUIRES=...

# (3) User-defined auxiliary functions and variables
T_FOO=...

# (4) The test code
rlJournalStart
  rlPhaseStartSetup
    rlAssertRpm --all
    rlImport "some/libraryA"
    rlImport "another/libraryB"

    rlRun "TmpDir=\$(mktemp -d)" 0 "Creating tmp directory"
    rlRun "pushd ${TmpDir}"
    # (4.1) User's setup code
  rlPhaseEnd

  rlPhaseStartTest "Optional test description"
    # (4.2) Test #1
  rlPhaseEnd

  rlPhaseStartTest "Optional test description"
    # (4.3) Test #2
  rlPhaseEnd

  # (4.4) Cleanup
  rlPhaseStartCleanup
    rlRun "popd"
    rlRun "rm -rfv $TmpDir" 0 "Removing tmp directory"
  rlPhaseEnd
# (4.5) Print the summary report
rlJournalPrintText
rlJournalEnd

When implemented, this proposal (described after the following snippet) allows user to reduce the test above to:

T_FOO=...

function Setup() {
  rlMakeTmpDir || return $?
  rlPushTmpDir || return $?
  # User's setup code
}

#: Optional description
function test_something_1() {
  # Test #1
}

#: Optional description
function test_something_1() {
  # Test #2
}

Now lets describe what will be needed to done on the beakerlib and eventually on the tmt side to make this happen. Suppose users have their tests prepared in runtest.sh file alongside with main.fmf¹ file containing the necessary metadata.

• Introduce a cleanup stack. Extend beakerlib about a stack-like storage where cleanup actions can be registered and functions that manipulate with it. Additionally implement
  • rlMakeTmpDir which makes a temporary directory and pushes code that removes it on the cleanup stack;
  • rlPushTmpDir which pushd to the temporary directory and pushes popd on the cleanup stack.
• Introduce a test runner script. In beakerlib, add a new executable script (call it rlrun for now), that roughly performs following steps (details can be discussed further):
  1. Look if the current working directory contains runtest.sh.
  2. Source /usr/share/beakerlib/beakerlib.sh
  3. Gather PACKAGES, REQUIRES and possibly rlImports from main.fmf.
  4. Gather tests and their optional descriptions. It is assumed that
     • a test has a form of shell function which name is prefixed with test_
     • an optional test description is a one-line comment of the form #: text which precedes the test it describes
  5. Source ./runtest.sh.
  6. Run the setup phase:
     • run rlAssertRpm --all
     • for every gathered beakerlib library to be rlImported, run rlImport
     • if Setup function is defined, run it
  7. If setup phase succeeded, run every test that had been gathered.
  8. Run all cleanup actions stored on cleanup stack.

Background

As a beakerlib user, I was quite unsatisfied with current beakerlib behavior. In a scenario when some part of setup phase fails, like a certificate is not generated, dependent tests also automatically fail, making unnecessary noise in the test log.

A discussion with @AloisMahdal and his idea to wrap tests written in beakerlib to functions to allow exit from tests whenever there is no reason to continue in execution inspired me to create a simple shell library I am using in my still-in-progress burning stack test suite. The library is using if setup the run_tests; cleanup pattern used by many test execution frameworks. It reduces a beakerlib's boilerplate a bit but there's still a room for improvement. This proposal is the next evolution step of the library.

In case of interest or if someone will find this useful I will implement it and open a PR.

Footnotes

1. In this case, the content of main.fmf may looks like:

   component:
     - foo
   require:
     - pkgA
     - pkgB
     - library(some/libraryA)
     - url: url/to/another/libraryB
       name: /another/libraryB
   
   test: rlrun
   framework: beakerlib
   

   ↩

[sopos]

First, beakerlib is just a library.
Second, there are other extension libraries which introduce kind-of framework functionality, e.g. mcase.

I'm not saying no but the original ambition was not the framework. We would need wider consensus on this. What do you say @hegerj ?

In case of more difficult tests with more setup and cleanup phases or phases generate in a loop it would be much more complicated and I can imagine the users of this approach would be asking for that sooner or later.

[hegerj]

This seems like a quite a tall order and I am not quite sure what is the primary use case for this. Is your biggest problem that if test fails in Setup phase it is pointless to run Test phase @i386x?

[i386x]

First, beakerlib is just a library.

I see. I can make this a standalone script then and request a repo for it either in beakerlib namespace so QEs can contribute to it or in pagure.io.

Second, there are other extension libraries which introduce kind-of framework functionality, e.g. mcase.

mcase combined with Cleanup is closest to what I am aiming for. The test runner can be build on top of them.

I'm not saying no but the original ambition was not the framework. We would need wider consensus on this. What do you say @hegerj ?

In case of more difficult tests with more setup and cleanup phases or phases generate in a loop it would be much more complicated and I can imagine the users of this approach would be asking for that sooner or later.

@hegerj My primary use case is to have some level of comfort both for devel when writing tests and for QE when reading logs. Let me demonstrate on volume_key. When volume_key fails to create a packet file, or a certificate is not created in the setup phase due to crypto policies change, tests that uses this packet file or certificate also fail, creating unnecessary confusion on my QE contact side, since only the failed to create a packet/certificate error is important and the rest is just a noise. When I looked at the code of volume_key tests I start dreaming about something like pytest, but on top of beakerlib, where you can exit from test phase prematurely, skip some test if a certain condition is not met, or skip the entire test suite if setup phase failed.

Again, my aim is not to change beakerlib code to satisfy my use case, but to create a configurable test runner script build on the top of beakerlib. beakerlib should stay atomic. I opened my RFE here since my initial idea was to distribute such a test runner alongside with beakerlib.

(Adding @psss and @pcahyna in case they are interested.)