Dagster 1.5.5+ & Pydantic compatibility issues

[benpankow]

Hello all - recently, we released Dagster 1.5.5, which included long-awaited support for Pydantic 2. Dagster continues to support Pydantic 1.x, as we realize other packages that users depend on may not have updated their pins.

This discussion will collect reported issues with Pydantic 1.x and 2.x from Dagster 1.5.5+, and include troubleshooting steps.

Known issues

Dagster does not properly handle using pydantic<2.0.0 and pydantic-core

The pydantic-core package is new with Pydantic 2 and is installed automatically when downloading pydantic>=2.0.0. However, when downgrading Pydantic to 1.x and not removing pydantic-core, Dagster 1.5.5 will raise often-confusing errors based on this mismatch. A fix is shipping with Dagster 1.5.6. In the meantime, ensure you either are not installing pydantic-core or are using pydantic>=2.0.0.

Dagster libraries will error if version mismatched with the core dagster package

Because of the compatibility layer introduced in dagster 1.5.5, older versions of libraries may not work properly. Ensure that your dagster libraries are matching, either ==1.5.5 or ==0.21.5, depending on package versioning scheme.