Skip to content

Commit 02c016f

Browse files
gaogaotiantianueshin
authored andcommitted
[SPARK-54762][PYTHON] Fix _create_converter and covert overload signature
### What changes were proposed in this pull request? Fix the overload signature of `_create_converter`. The default value should be `False`. Also fixed the overload signature of `convert`. The return value should be List[Row | tuple] if `return_as_tuples` is passed - because an explicit `True` can be passed. ### Why are the changes needed? I would guess that the original author tried to express that "if `none_on_identity` is `True`, then it could return `None`", but this is the wrong expression. You can't express such a concept in the current Python typing system. (You can use `Literal` but that requires the users to pass the literal `True` or `False`, not a variable). The current signature means the default value of `none_on_identity` is `True` which is incorrect. The default value of a overload signature is not used by the type checker, but it will be shown in IDE and this is misleading. Same thing for `convert`. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? No code is changed. ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#53467 from gaogaotiantian/conversion-overload. Authored-by: Tian Gao <[email protected]> Signed-off-by: Takuya Ueshin <[email protected]>
1 parent a26c209 commit 02c016f

File tree

1 file changed

+4
-2
lines changed

1 file changed

+4
-2
lines changed

python/pyspark/sql/conversion.py

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -113,7 +113,7 @@ def _create_converter(
113113
dataType: DataType,
114114
nullable: bool = True,
115115
*,
116-
none_on_identity: bool = True,
116+
none_on_identity: bool = False,
117117
int_to_decimal_coercion_enabled: bool = False,
118118
) -> Optional[Callable]:
119119
pass
@@ -811,7 +811,9 @@ def convert(table: "pa.Table", schema: StructType, *, binary_as_bytes: bool) ->
811811

812812
@overload
813813
@staticmethod
814-
def convert(table: "pa.Table", schema: StructType, *, return_as_tuples: bool) -> List[tuple]:
814+
def convert(
815+
table: "pa.Table", schema: StructType, *, return_as_tuples: bool
816+
) -> List[Row | tuple]:
815817
pass
816818

817819
@staticmethod # type: ignore[misc]

0 commit comments

Comments
 (0)