RFC: organizing release testing efforts

[miri64]

In the past we used tick-lists in the OP like this to organize the testing efforts of a release:

• test 1
• test 2
• ...

With the 2019.04 release @danpetry tried out a spreadsheet solution.

Another alternative could be a Kanban board (aka Github Projects).

Another alternative could be to automatically generate and push results to an online dashboard, for the tests that are automated. For example, creating JUnit XML test results documents and pushing to https://www.cdash.org/.

All of these solutions have drawbacks and benefits and none of them are perfect. Here a benefits and drawbacks I found.

Solution	Benefit	Drawback
Tick-list	Immediate overview	easy to accidentally override changes of others
	GitHub integration
	easy to auto-generate (markdown template)
Google spreadsheet	Synchronizable	External to GitHub
		No immediate overview
		not as easy to auto-generate (Google access required)
Github projects	Synchronizable	No immediate overview
	GitHub integration	not as easy to auto-generate but doable (GitHub access required)
		Can easily end in chaos (?)
Automation and dashboard	Instant	Lots of upfront work
		Could require time to come to consensus on solution

So here is my question: Anyone got any other idea? Do we want to change the organization of the testing efforts at all? Are there more drawbacks/benefits to the ones I listed?

[danpetry]

I added automated dashboard, which @aabadie suggested.

[danpetry]

FYI the reason why I came up with a spreadsheet in the first place wasn't primarily to solve the overwrite issue above, it was to make it easier to be on top of the state of testing (incl passes, fails, bugfix status...). An inline conversation really means you have to keep all this in your head, re-read if necessary to find information, etc, which is kind of a drain energy- and time-wise from an organizing perspective. I initially was just going to use it for myself but then got a comment offline that I could upload it so everyone could use it. Just for some background.

[miri64]

I added automated dashboard, which @aabadie suggested.

Mh. I don't know what this should be? Some custom software or is there a link were you can point to @aabadie?

[danpetry]

Link in comments above table. https://www.cdash.org

[aabadie]

Another alternative could be to automatically generate and push results to an online dashboard, for the tests that are automated. For example, creating JUnit XML test results documents and pushing to https://www.cdash.org/.

I did some tests of this for 2019.01 (see https://my.cdash.org/index.php?project=RIOT&date=2019-02-01, not sure you can access it). But I found bugs with the integration JUnit XML in CDash. It might be fixable in CDash though, but I haven't taken the time to do it.

I added automated dashboard, which @aabadie suggested.

Do you have a link or something ? Not sure what "automated dashboard" means.

[aabadie]

Link in comments above table. https://www.cdash.org

Ok but that doesn't tell where you put the dashboard. Are you self-hosting an instance of CDash or use an application in my.cdash.org hosting offer ?

[miri64]

I did some tests of this for 2019.01 (see https://my.cdash.org/index.php?project=RIOT&date=2019-02-01, not sure you can access it). But I found bugs with the integration JUnit XML in CDash. It might be fixable in CDash though, but I haven't taken the time to do it.

Another thing I noticed also is that I have to create an account there, just to see what you did there ;-).