Fix regression when creating nested ConfigurableResource with pydantic >= v.2.5.0

[HynekBlaha]

Summary & Motivation

Fixes: #18272

We are using pydantic BaseSettings to define dagster code deployment configurations. Our config framework dumps the config content into json, which we save to configmap for better visibility of changes in our workloads. We then create config instance from the json dict to maintain 1:1 correctness.

In pydantic v2.5.0, there was a change in validation of Unions: pydantic/pydantic-core#867.
As PartialResource is very lax, it passes the validation, but immediately fails on TypeError.
I tried several alternatives, but this one is least invasive and works best.

Since the issue appeared, we tried to:

• reduce the number of nested ConfigurableResources in our codebase
• pin the pydantic == v2.4.2 (very old, released on 2023-09-27 )

But eventually, we needed new pydantic features in other projects and got to the point we needed to transitively upgrade pydantic in our dagster projects.

How I Tested These Changes

First, let's verify that this piece of code used to work with pydantic v2.4.2 ✅

git checkout ec9048878b9f8d81e3a0bb6284df0f0de493da68
uv pip install pydantic==2.4.2
python -m pytest python_modules/dagster/dagster_tests/core_tests/resource_tests/pythonic_resources

================================================================================================ 121 passed, 4 deselected in 7.66s ================================================================================================

but doesn't work with latest pydantic version ❌

uv pip install pydantic --upgrade
python -m pytest python_modules/dagster/dagster_tests/core_tests/resource_tests/pythonic_resources

===================================================================================================== short test summary info =====================================================================================================
FAILED python_modules/dagster/dagster_tests/core_tests/resource_tests/pythonic_resources/test_nesting.py::test_nested_resources_direct_config_fully_populated - TypeError: PartialResource.__init__() got an unexpected keyword argument 'b'
=========================================================================================== 1 failed, 120 passed, 4 deselected in 6.95s ===========================================================================================
 

Then checkout the second commit where I fix the PartialResource.init() and rerun the tests again. ✅

Changelog

Fix regression where nested ResourceDefinition could not be validated from fully populated dict.

[HynekBlaha]

I changed the order, because it starts working again if pydantic.ValidationError is raised instead of TypeError.
So I run the validation ASAP.

[HynekBlaha]

While this isn't the prettiest, the resource_cls and data are in fact not provided to init in all cases.
Alternatively, we could make it just accept kwargs and retrieve the resource_cls and data in the method body.

[HynekBlaha]

Hello @benpankow, I made this fix for an old regression that was bothering us for a long time.
Git blame told me you were the author of this part of code, could you please take a look at it?
I tried to make it as least invasive as possible. All previous tests are passing.
Thanks ❤️

[HynekBlaha]

Hello @benpankow would you have a few minutes to check this small PR? 🤞 I would be very happy if we could get it into following release.

[benpankow]

Thanks for the PR @HynekBlaha! I made a slight tweak in #28949 which enforces the pre-Pydantic 2.5 union rule. I think this should also fix the issue and doesn't require us to adjust the signature of PartialResource.

[HynekBlaha]

Looks great, @benpankow! Closing this issue in favour of your version. Thank you for finding time to look at it! 👍