-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Labels
A-meta-test-infrastructureArea: Meta: Test infrastructureArea: Meta: Test infrastructure
Description
Split from #140.
- comprehensibly document UI testing in the currently empty
README.adoc - introduce a format option
-f/--formatwhich acceptsprettyorterse(taken from rust'slibtest) where the terse format prints single tests single characters without space and line breaks (.for passing tests,ifor ignored ones,gfor gilded ones,Ifor invalid ones,afor auxiliary ones (not shown by default)) and maybe even make the terse the default - syntactic or preferably semantic highlighting for test configurations by the VS Code extension
- improve the looks of the generated diffs by diff'ing diff'ed lines by character (only those lines where it makes sense, I think that's single
+lines preceded by single-lines) and highlight them (changing their background color) - check how much the current architecture of auxiliary files impacts the (parallel) performance of the test runner. I suspect since we are skipping so many files, every thread wastes a lot of time and performance is not as good. basically the division of work is suboptimal. also for packages, we heavily rely on auxiliary files and the more package tests we add the more the performance will probably degrade. could the
auxiliary/folder design from rustc's test suites help us out? - revision / matrix system
- support putting the test configuration into a separate file named
<testname>.testconfigwhich allows us to test files where we don't want to have any comments e.g. to test everythings ok with truly empty files or lexing and spans are fine without comments when lexing (…), you get what I mean - smh. (somewhere) record the modification date of passing tests and skip them (printing
ignoredor ideally smth. more descriptive) when they are tested and if they were indirectly referenced (e.g. through a folder) - use proper lexers to obtain the comments (to get rid of false-postive test config params) (via Lexer-based UI tester #162)
- ensure that the unit tests of the testing framework are executed in CI (haven't checked yet)
- check if the framework actually works on Windows (we currently use the POSIX?
timeoutcommand for which I am not sure if it is available) - add a flag to test all git-unstaged or all git-staged test files
- skip / ignore files that have already been tested & passed: record modification time / hash of test files & the compiler; if any of them differ from the current run, don't skip; this might not be very useful right now; add a flag to force-run all tests
Metadata
Metadata
Assignees
Labels
A-meta-test-infrastructureArea: Meta: Test infrastructureArea: Meta: Test infrastructure