Update testing intro doc #4735

mwichmann · 2025-06-10T21:10:48Z

This does not affect SCons at all - only updates the document (which is eventually duplicated on the Wiki) about making use of the SCons test framework.

It's not going to be easy to review in the classical way, maybe just scan through the full doc and see if it makes sense this way?

https://scons.org/guidelines.html

Contributor Checklist:

I have created a new test or updated the unit tests to cover the new/changed functionality.
I have updated CHANGES.txt and RELEASE.txt (and read the README.rst).
I have updated the appropriate documentation

Signed-off-by: Mats Wichmann <[email protected]>

bdbaddog · 2025-06-11T01:30:38Z

testing/framework/test-framework.rst

-   the unit tests for the ``Builder.py`` module are in the
-   ``BuilderTests.py`` script.  Unit tests tend to be based on assertions.
+   ``SCons/`` subdirectory (that is alongside the code they are testing)
+   and use the Python ``unittest`` module to perform assertion-based


Should include that files have to be named *Tests.py to be found.

I thought that's what that section was saying?

I don't see it in this section.. it's much later in the doc..

I'm not seeing the reason for the fuss here. This is an "about" section, not a "how to" section. I can add these but it seems like extra noise.

You removed it from the original, which I think is useful information to have up front.

bdbaddog · 2025-06-11T01:31:42Z

testing/framework/test-framework.rst

+   run by calling ``scons`` (or in a few cases, ``sconsign`` or ``scons-time``)
+   to execute them, and thus capture the results of
+   running all the way through an invocation - end to end.
+   They are implemented as Python scripts which are responsible for setup,


Must have the .py suffix.

It says that clearly somewhere... line 112 in the current copy after I hunt for it.

better to duplicate than to mention this bit here and that bit only later.

bdbaddog · 2025-06-11T01:32:11Z

testing/framework/test-framework.rst

-a reproducer are by their nature more like an e2e test - this is something
-a user has tried in their SConscripts that didn't have the expected result.
+End-to-end tests exist to verify hardened parts of the public interface:
+featurers documented in the reference manual that are available


typo "featurers" -> features.

bdbaddog · 2025-06-11T01:33:18Z

testing/framework/test-framework.rst

+("I put this in my SConscripts, and something went wrong"),
+so it's often useful to code an e2e test to match a bug report,
+and develop a solution that turns that to a pass.
+So pick the testing strategy that makes sense (sometimes both do).


And if it's possible to reduce an end-to-end test to a unit test, please do.

I'll add that but I'm not sure that's always the preference, I think it's context-dependent.

bdbaddog · 2025-06-11T01:34:56Z

testing/framework/test-framework.rst



 Testing Architecture
 ====================

 The test framework provides a lot of useful functions for use within a
 test program. This includes test setup, parameterization, running tests,
-looking at results and reporting outcomes. You can run a particular test
+examining results and reporting outcomes. You can run a particular test


Does this work for end-to-end tests? I thought it was only unit tests..

You're right, it doesn't - I thought it did. It doesn't find the SCons instance properly. Possible that could be fiddled too, but I'll get rid of the implication.

bdbaddog · 2025-06-11T01:35:39Z

testing/framework/test-framework.rst

   tells ``runtest.py`` to skip the contents of the directory completely.
-#. Create a file ``.exclude_tests`` in each directory in question, and in
-   it list line-by-line the files to exclude from testing.
+   * Create a file ``.exclude_tests`` in yur subdirectory, and in


yur => your

or "ur"? :-)

bdbaddog · 2025-06-11T01:35:59Z

testing/framework/test-framework.rst

-   Sometimes other Python modules are necessary or helpful, and get
-   imported before this line.
+   If you need Python standard library modules in your code,
+   the conventio it to import those before the framework.


conventio -> convention

Signed-off-by: Mats Wichmann <[email protected]>

Update testubg intro doc [skip ci]

83b7fcb

Signed-off-by: Mats Wichmann <[email protected]>

mwichmann added the testsuite Things that only affect the SCons testing. Do not use just because a PR has tests. label Jun 10, 2025

bdbaddog reviewed Jun 11, 2025

View reviewed changes

mwichmann added 2 commits June 11, 2025 07:42

test framework doc: fix review comments [ci skip]

cce3ef8

Signed-off-by: Mats Wichmann <[email protected]>

Reword e2e test intro [ci skip]

16a554b

Signed-off-by: Mats Wichmann <[email protected]>

Uh oh!

Update testing intro doc #4735

Are you sure you want to change the base?

Update testing intro doc #4735

Uh oh!

Conversation

mwichmann commented Jun 10, 2025

Contributor Checklist:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!