Extensive integration tests

[iboB]

I think we need more robust and extensive integration testing so we can test new features and especially regressions.

So my idea is to basically have test/integration which includes several projects with multiple dependencies with multiple configurations (with and without package-lock, external cache), some projects with errors. These dependencies could be existing GitHub repos, but I think that having multiple (perhaps 10-15) repos in cpm-cmake named test-package-<something> will give us more control over what exactly is tested. These test repos can provide versions, archives, themselves have interdependencies and more.

The thing is I don't think that driving the integration tests with CMake is a good idea. CMake is just not a very nice language for such things, and while possible, I think using Python, or Ruby, or another, more general purpose language, will make things easier and much more pleasant, and crucially help us minimize bugs in the tests themselves.

The integration test projects may also serve as examples of advanced usage of CPM

[Andres6936]

I recently managed to run the test suite for my project within Github Actions using Doctest, I used CPM for downloading and configuring the dependency, it also works using Catch2 (although I don't have the evidence).

[TheLartians]

I currently use a rather large project with ~40 total dependencies (many of them are nested) myself, which I often use as a "works on my machine" integration test before releases. Unfortunately, due to the closed-source nature it can't be used as a public integration test in CI, but some test repos with some interdependent CPM.cmake dependencies and using the full feature scope would be great to have!

I agree that CMake isn't the best test runner for integration tests. It's still great for unit tests, as it allows to call functions and inspect local variables directly, but integration tests are quite clumsy there. Note that the examples (which already serve as small integration tests) are also using a python script for building.

[iboB]

@TheLartians how do you feel about the ruby programming language? 😄

I'm thinking of starting the initial work on the integration tests and to me using ruby as the driver feels most natural. I have extensive experience with it and would definitely work fastest writing in ruby.

Basically the integration test driver's job would be to:

• Generate CMakeLists.txt for root projects
• Configure with various command-line options and env vars (say CPM_SOURCE_CACHE)
• Parse the CMake output for specific expected failures (not just test that if fails, but test that it fails in with a certain error)
• Read the generated CMakeCache.txt for certain values
• Read package lock when applicable and check for expected output
• Run builds
• Check build output for the existence of expected targets (for example that the examples or tests of a depenency are not there if we expect them not to be there)
• Other things as they come up

PS Perhaps this discussion should be turned into an issue, what do you think?

[TheLartians]

TBH I'm more fluent with Python and JavaScript than Ruby, but if you feel that Ruby is the most natural fit for such things, I'll trust you and try to catch up! 😄

Sure I think tracking progress in an issue makes sense at this point, can we transform the discussion back to issues though?
Otherwise feel free to open one.

[iboB]

Oops... you're right. It seems this is not possible. Ok. So I'll add some stuff in a branch in the next couple of days and figure out what do to about issues later