[Epic] Python: Improved Typing

[justinvp]

For Python codegen we generate classes for inputs of object type to enable strong typing for these inputs. However, this leads to somewhat verbose syntax, requiring to instantiate the "args classes". See #11500 for a motivating example.

With PEP 589 – TypedDict and PEP 655 – Marking individual TypedDict items as required or potentially-missing there is now a more concise way to achieve the same result.

For each input object type we can generate a Args class as before, as well as a ArgsDict TypedDict based type. To support backwards compatibility, inputs of object type will accept the union of the two types.

class MyTypeArgsDict(TypedDict):
    my_prop: pulumi.Input[str]
    my_other_prop: NotRequired[pulumi.Input[float]]

Related issues: #11732

Example with Args

from pulumi_kubernetes.apps.v1 import Deployment
from pulumi_kubernetes.apps.v1 import Deployment, DeploymentSpecArgs
from pulumi_kubernetes.meta.v1 import LabelSelectorArgs, ObjectMetaArgs
from pulumi_kubernetes.core.v1 import ContainerArgs, PodSpecArgs, PodTemplateSpecArgs

app_labels = {"app": "nginx"}
deployment = Deployment(
    "nginx",
    spec=DeploymentSpecArgs(
        selector=LabelSelectorArgs(match_labels=app_labels),
        replicas=1,
        template=PodTemplateSpecArgs(
            metadata=ObjectMetaArgs(labels=app_labels),
            spec=PodSpecArgs(containers=[ContainerArgs(name="nginx", image="nginx")]),
        ),
    ),
)

Example with ArgsDict

from pulumi_kubernetes.apps.v1 import Deployment

app_labels = {"app": "nginx"}

deployment = Deployment(
    "nginx",
    spec={
        "selector": {"match_labels": app_labels},
        "replicas": 1,
        "template": {
            "metadata": {"labels": app_labels},
            "spec": {"containers": [{"name": "nginx", "image": "nginx"}]},
        },
    },
)

Design (M103)

• Design doc https://docs.google.com/document/d/1qNBzw-W1e80xhVpwQHNbwZ3hlfhMSBX-6TYF6UNcx0c/edit#heading=h.ojtxje7btj58
• Public copy https://docs.google.com/document/d/1fr1qAUE41mQq_YOmrP_aMncfFQXxU85jMpuOTeN9bYo/edit#heading=h.ojtxje7btj58

Implementation Plan

M104

• [sdk-gen/python] Generate non-plain object types being used as inputs for output-version invokes #14871
• Generated TypedDicts for python inputs #15957
• Verify mypy/pyright/pycharm performance for TypedDict
• Verify that generated docs use the new TypedDict types
• Review typing error messages for TypedDict

Rollout & Announce (M105)

• Rollout to tier 1 providers
• Update hand-written documentation examples to use the new TypedDict types
• Update templates to use new TypedDict types
• Blog post
• 🍦

[jf]

hi Justin, for folks interested in the progress of this, where would we go to observe the progress and/or help with things?

[justinvp]

@jf, keeping an eye on this issue and #11732 is the best place to observe progress. When we have a fleshed out design proposal, we'll post it here.

[julienp]

We have a working implementation in #15957, however we ran into some performance issues when using MyPy ¹.

To work around this, we currently generate the TypedDict based args to behave differently depending on the used type checker. MyPy supports a special variable MYPY that is always considered true, and we can use this to hide the TypedDict types from it, and use Mapping[str, Any] instead. The PyDantic library currently uses a similar workaround, although for a different performance issue ².

Here's an example from the AWS provider:

if not MYPY:
    class InstanceCpuOptionsArgsDict(TypedDict):
        amd_sev_snp: NotRequired[pulumi.Input[str]]
        core_count: NotRequired[pulumi.Input[int]]
        threads_per_core: NotRequired[pulumi.Input[int]]
elif False:
    InstanceCpuOptionsArgsDict: TypeAlias = Mapping[str, Any]

When using Pyright, you get the full type checking experience within dictionary arguments, but not with MyPy.

The MyPy behaviour matches the previous implementation where these input types could either take an Args class or a Mapping[str, Any]. This means that we technically don't have a regression here, and when using Pyright you get a better experience.

However the question then becomes what style of arguments should we encourage? Currently we encourage the Args style classes over passing in dictionaries. It is the default for generated code and in documentation. This is not the most pythonic approach and fairly verbose ³, but does give strong typing and better IDE discoverability.

I'll also note that Pyright does not produce very readable error messages

Example

For the following code, Pyright correctly notes an error, but MyPy checking passes.

from pulumi_kubernetes.apps.v1 import Deployment

deployment = Deployment("nginx",
    spec={
        "selector":{
            "match_labels": { "app": "nginx" }
        },
        "replicas": "one", # This should be a Input[int]
        "template": { "spec": { "containers": [{ "name": "nginx", "image": "nginx" }] }
        }
    }
)

MyPy reports no issues

python3 -m mypy __main__.py
Success: no issues found in 1 source fil

Pyright reports an error, but the error message is not very friendly, and it is not obvious what the actual error is.

__main__.py:30:10 - error: Argument of type "dict[str, dict[str, dict[str, str]] | str | dict[str, dict[str, list[dict[str, str]]]]]" cannot be assigned to parameter "spec" of type "Input[DeploymentSpecArgs | DeploymentSpecArgsDict] | None" in function "__init__"
    Type "dict[str, dict[str, dict[str, str]] | str | dict[str, dict[str, list[dict[str, str]]]]]" is incompatible with type "Input[DeploymentSpecArgs | DeploymentSpecArgsDict] | None"
      "dict[str, dict[str, dict[str, str]] | str | dict[str, dict[str, list[dict[str, str]]]]]" is incompatible with "DeploymentSpecArgs"
      "dict[str, dict[str, dict[str, str]] | str | dict[str, dict[str, list[dict[str, str]]]]]" is incompatible with "DeploymentSpecArgsDict"
      "dict[str, dict[str, dict[str, str]] | str | dict[str, dict[str, list[dict[str, str]]]]]" is incompatible with protocol "Awaitable[DeploymentSpecArgs | DeploymentSpecArgsDict]"
        "__await__" is not present
      "dict[str, dict[str, dict[str, str]] | str | dict[str, dict[str, list[dict[str, str]]]]]" is incompatible with "Output[DeploymentSpecArgs | DeploymentSpecArgsDict]"
      "dict[str, dict[str, dict[str, str]] | str | dict[str, dict[str, list[dict[str, str]]]]]" is incompatible with "None" (reportArgumentType)
1 error, 0 warnings, 0 informations

Footnotes

1. https://github.com/python/mypy/issues/17231 ↩

2. https://github.com/pydantic/pydantic-core/pull/528 ↩

3. https://github.com/pulumi/pulumi/discussions/11500 ↩