How should we structure our bash testing in CTSM?

[ekluzek]

We are starting to add more testing of bash scripts into CTSM. As such I think we should formalize some of how we plan to do it, and to make sure there are utilities that make the testing simpler to do. And standard patterns of how to test that can be followed, and that are similar to our python unit testing (as well as the PF unit testing).

Here are two locations where bash testing is handled:
https://github.com/ESCOMP/CTSM/tree/master/cime_config/testdefs/testmods_dirs/clm/FatesSetupParamBuild
https://github.com/ESCOMP/CTSM/blob/master/doc

This came up as a result of review in #3231 with this comment:

And I think if we are going to have more bash testing, we should probably develop some philosophy around how to do it. I'll start a discussion on that. I know there are some frameworks for it, and looked into it a little bit with the testing I did in cime_config/testdefs/testmods_dirs/clm/FatesSetupParamBuild.

I think looking there would be useful also. Some of what we have there should likely be used here. So we probably want a directory for bash unit testing utilities, and make useful things for testing put in that place.

The functions in shell_commands there are candidates for this:

fatal_error() {
log_msg_if_debug () {
check_error () {
check_if_runable () {

Similarly these functions in run_shell_commands_tests:

conda_for_host() {
handle_error() {
expect_fail() {
expect_nofail() {
test_run_shell_commands() {

The test structure in the last one brings us closer to how the python unit testing is done, which is helpful in getting people up to speed on how to do this type of thing for bash.

https://github.com/ESCOMP/CTSM/pull/3231/files#r2162530553

Questions to figure out:

• What shared utilities should we have for bash testing?
• Where should they go?
• What pattern should we follow for bash testing?
• Should we use "set -e" to error out when a command returns an error code? And if so -- when?
• Should separate tests be in separate files or functions?
• Should we move to a BASH unit test framework like bats-core?

[samsrabin]

I think most of the shared utilities you list are really useful. Would it make sense to consolidate them into a top-level bash/ directory like we have for python/?

I like the idea of short tests being grouped together in thematic files like we have for Python unit/system testing.

[ekluzek]

Good question, that would provide logical symmetry to the python testing. Which would give us a pattern to follow. And that would make the pattern for bash code and testing to be really similar to the python testing. And that's a really good goal to work toward -- to make it easier for all developers.

I do think we want to support less bash scripts and tests than python. So I hope moving this direction, doesn't encourage us to make too many scripts in bash. But, for example, shell_commands pretty much have to be in bash, and smaller simpler things are convenient to do in bash.

So I think that's probably a yes here.

[ekluzek]

I looked into unit-test frameworks for bash, and thought the best one would be bats-core. Other ones are out there, but they have a GPL license and we shouldn't use them. bats-core could be added a submodule with git-fleximod, and has a MIT license.

https://github.com/bats-core/bats-core

The ending tests look similar to PFunit.