Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Conformance tests: Align automated and manual scoring #1692

Open
24 of 25 tasks
JelleZijlstra opened this issue Apr 6, 2024 · 1 comment
Open
24 of 25 tasks

Conformance tests: Align automated and manual scoring #1692

JelleZijlstra opened this issue Apr 6, 2024 · 1 comment
Labels
topic: other Other topics not covered

Comments

@JelleZijlstra
Copy link
Member

JelleZijlstra commented Apr 6, 2024 •
edited by erictraut

In #1678 I added an automated tool for scoring the conformance test suite, and #1682 and #1683 added some features to support edge cases.

Now I am working on going through the remaining files where the automated conformance judgment on a file differs from the manual one (open PRs: #1684, #1685, #1687, #1688, #1689, #1690, #1691).

I'll use this issue to track the remaining discrepancies (from conformance/src/unexpected_fails.py):

Any help is appreciated.
@JelleZijlstra JelleZijlstra added the topic: other Other topics not covered label Apr 6, 2024
@JelleZijlstra JelleZijlstra mentioned this issue Apr 7, 2024
Merged
JelleZijlstra added a commit to JelleZijlstra/typing that referenced this issue Apr 9, 2024
@JelleZijlstra
Conformance tests: Fix automated scoring for pyright dataclasses tests
21b9f46 
Part of python#1692
@JelleZijlstra JelleZijlstra mentioned this issue Apr 9, 2024
Merged
JelleZijlstra added a commit to JelleZijlstra/typing that referenced this issue Apr 9, 2024
@JelleZijlstra
Conformance tests: Fix generics_defaults scoring for pyright
af63a7c 
Pyright produces an error unrelated to the purpose of the test, and I
don't see a good way to write the test to avoid the error.

Part of python#1692
@JelleZijlstra JelleZijlstra mentioned this issue Apr 9, 2024
Merged
JelleZijlstra added a commit to JelleZijlstra/typing that referenced this issue Apr 9, 2024
@JelleZijlstra
Conformance tests: Ignore overlapping overload errors in LiteralString
bd79ac2 
Part of python#1692
@JelleZijlstra JelleZijlstra mentioned this issue Apr 9, 2024
Merged
JelleZijlstra added a commit that referenced this issue Apr 9, 2024
@JelleZijlstra
Conformance tests: Fix automated scoring for pyright dataclasses tests (
284dc36 
#1699)

Part of #1692
JelleZijlstra added a commit that referenced this issue Apr 9, 2024
@JelleZijlstra
Conformance tests: Fix generics_defaults scoring for pyright (#1700)
1bb065a 
Pyright produces an error unrelated to the purpose of the test, and I
don't see a good way to write the test to avoid the error.

Part of #1692
JelleZijlstra added a commit to JelleZijlstra/typing that referenced this issue Apr 9, 2024
@JelleZijlstra
Conformance tests: Fix pyright scoring for protocols_variance
29014df 
Pyright produces warnings instead of errors for many issues in this test case.

Part of python#1692
@JelleZijlstra JelleZijlstra mentioned this issue Apr 9, 2024
Merged
JelleZijlstra added a commit to JelleZijlstra/typing that referenced this issue Apr 9, 2024
@JelleZijlstra
Conformance tests: Fix tuples_unpacked.py
4b97bbd 
I can't find the quoted phrase in the spec and I don't see a reason
why this should error.

Part of python#1692
@JelleZijlstra JelleZijlstra mentioned this issue Apr 9, 2024
Merged
JelleZijlstra added a commit that referenced this issue Apr 9, 2024
@JelleZijlstra
Conformance tests: Fix pyright scoring for protocols_variance (#1702)
3998da0 
Pyright produces warnings instead of errors for many issues in this test case.

Part of #1692
JelleZijlstra added a commit that referenced this issue Apr 9, 2024
@JelleZijlstra
Conformance tests: Fix tuples_unpacked.py (#1703)
d61944d 
I can't find the quoted phrase in the spec and I don't see a reason
why this should error.

Part of #1692
JelleZijlstra added a commit to JelleZijlstra/typing that referenced this issue Apr 9, 2024
@JelleZijlstra
Conformance tests: mark pytype as failing generics_scope
7ed7028 
It fails to produce errors on a few lines expected by the test.

Also fix an unrelated pytype error and ignore an unrelated pyre error.

Part of python#1692
@JelleZijlstra JelleZijlstra mentioned this issue Apr 9, 2024
Merged
JelleZijlstra added a commit to JelleZijlstra/typing that referenced this issue Apr 9, 2024
@JelleZijlstra
conformance tests: fix scoring for typeddicts_inheritance
6a18d07 
Pyre and pytype produced an expected error on the wrong line.

Part of python#1692
@JelleZijlstra JelleZijlstra mentioned this issue Apr 9, 2024
Merged
JelleZijlstra added a commit to JelleZijlstra/typing that referenced this issue Apr 9, 2024
@JelleZijlstra
Conformance tests: mark pytype as failing assert_type test
5d5bd1f 
It does not produce an error for `x: Literal[4]; assert_type(x, int)`.

Part of python#1692
@JelleZijlstra JelleZijlstra mentioned this issue Apr 9, 2024
Merged
JelleZijlstra added a commit that referenced this issue Apr 9, 2024
@JelleZijlstra
Conformance tests: mark pytype as failing assert_type test (#1706)
4025826 
It does not produce an error for `x: Literal[4]; assert_type(x, int)`.

Part of #1692
JelleZijlstra added a commit that referenced this issue Apr 9, 2024
@JelleZijlstra
conformance tests: fix scoring for typeddicts_inheritance (#1705)
b9124c0 
Pyre and pytype produced an expected error on the wrong line.

Part of #1692
JelleZijlstra added a commit that referenced this issue Apr 9, 2024
@JelleZijlstra
Conformance tests: mark pytype as failing generics_scope (#1704)
aed95f8 
It fails to produce errors on a few lines expected by the test.

Also fix an unrelated pytype error and ignore an unrelated pyre error.

Part of #1692
JelleZijlstra added a commit that referenced this issue Apr 10, 2024
@JelleZijlstra
Conformance tests: Ignore overlapping overload errors in LiteralString (
8ff42d7 
#1701)


Part of #1692
JelleZijlstra added a commit to JelleZijlstra/typing that referenced this issue Apr 11, 2024
@JelleZijlstra
Conformance tests: Fix automated scoring for pyright dataclasses tests (
ad9fe6e 
python#1699)

Part of python#1692
JelleZijlstra added a commit to JelleZijlstra/typing that referenced this issue Apr 11, 2024
@JelleZijlstra
Conformance tests: Fix generics_defaults scoring for pyright (python#…
af96dff 
…1700)

Pyright produces an error unrelated to the purpose of the test, and I
don't see a good way to write the test to avoid the error.

Part of python#1692
JelleZijlstra added a commit to JelleZijlstra/typing that referenced this issue Apr 11, 2024
@JelleZijlstra
Conformance tests: Fix pyright scoring for protocols_variance (python…
ddb8697 
…#1702)

Pyright produces warnings instead of errors for many issues in this test case.

Part of python#1692
JelleZijlstra added a commit to JelleZijlstra/typing that referenced this issue Apr 11, 2024
@JelleZijlstra
Conformance tests: Fix tuples_unpacked.py (python#1703)
3398751 
I can't find the quoted phrase in the spec and I don't see a reason
why this should error.

Part of python#1692
JelleZijlstra added a commit to JelleZijlstra/typing that referenced this issue Apr 11, 2024
@JelleZijlstra
Conformance tests: mark pytype as failing assert_type test (python#1706)
8537aac 
It does not produce an error for `x: Literal[4]; assert_type(x, int)`.

Part of python#1692
JelleZijlstra added a commit to JelleZijlstra/typing that referenced this issue Apr 11, 2024
@JelleZijlstra
conformance tests: fix scoring for typeddicts_inheritance (python#1705)
e253bf6 
Pyre and pytype produced an expected error on the wrong line.

Part of python#1692
JelleZijlstra added a commit to JelleZijlstra/typing that referenced this issue Apr 11, 2024
@JelleZijlstra
Conformance tests: mark pytype as failing generics_scope (python#1704)
9e9d253 
It fails to produce errors on a few lines expected by the test.

Also fix an unrelated pytype error and ignore an unrelated pyre error.

Part of python#1692
JelleZijlstra added a commit to JelleZijlstra/typing that referenced this issue Apr 11, 2024
@JelleZijlstra
Conformance tests: Ignore overlapping overload errors in LiteralString (
48e7891 
python#1701)


Part of python#1692
JelleZijlstra added a commit to JelleZijlstra/typing that referenced this issue Jun 7, 2024
@JelleZijlstra
Conformance tests: Fix manual scoring discrepancy for pyre
7d9d627 
Pyre infers "Literal[1]" instead of "int", which seems legal. Change
the test to get an int from a function parameter instead, so that all
type checkers are aligned.

Part of python#1692.
@JelleZijlstra JelleZijlstra mentioned this issue Jun 7, 2024
Merged
JelleZijlstra added a commit to JelleZijlstra/typing that referenced this issue Jun 7, 2024
@JelleZijlstra
Conformance tests: Mark pytype as passing annotations_coroutines
40cdaa1 
It appears to produce the correct output.

Part of python#1692
@JelleZijlstra JelleZijlstra mentioned this issue Jun 7, 2024
Merged
JelleZijlstra added a commit that referenced this issue Jun 7, 2024
@JelleZijlstra
Conformance tests: Fix manual scoring discrepancy for pyre (#1770)
0105dc2 
Pyre infers "Literal[1]" instead of "int", which seems legal. Change
the test to get an int from a function parameter instead, so that all
type checkers are aligned.

Part of #1692.
JelleZijlstra added a commit that referenced this issue Jun 7, 2024
@JelleZijlstra
Conformance tests: Mark pytype as passing annotations_coroutines (#1771)
20efe7b 
It appears to produce the correct output.

Part of #1692
@JelleZijlstra
Copy link
Member Author

The only remaining issue now is mypy/namedtuples_define_functional.toml. There the next mypy release should have full support (python/mypy#17247), which will remove the discrepancy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
topic: other Other topics not covered
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@JelleZijlstra