Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixture phases should be processed in order #6

Open
baileyparker opened this issue Feb 14, 2018 · 2 comments
Open

Fixture phases should be processed in order #6

baileyparker opened this issue Feb 14, 2018 · 2 comments
Assignees

Comments

@baileyparker
Copy link
Owner

Not yet relevant, but the ordering of the phases depends on how unittest discover does things. Ideally, we want the phase tests to be run in the order of dependency within the compiler (ex. scanner -> parser -> semantic analysis -> etc.). In this way, if the compiler under test has a bug in an earlier phase we don't test a later phase first (and mislead the user that the error occurred in the later phase).

@baileyparker
Copy link
Owner Author

Temporary fix was to enumerate the tests in order in bin/run_harness. This is actually a decent solution, if it wasn't for the error prone enumeration. A better solution is having bin/run_harness delegate to simple_test.main, which can do introspection on its module to find tests (and perhaps provide some args for nice programmatic test settings--like disabling passing CST tests a la #14).

@baileyparker
Copy link
Owner Author

0963158 has a better solution. run_harness now delegates to simple_test.main, which can have a lot more smarts. Currently, by default the order is explicitly enumerated in simple_test.main (automatic enumeration as mentioned above would still be nice). The user also has the freedom to specify which suites to run (and can run them in any order). For example:

# Runs cst, ast, symbol table, and scanner (in that order)
$ bin/run_harness cst ast st scanner

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant