Inheriting overrides

[AdrianSosic]

• cattrs version: 23.2.3
• Python version: 3.9
• Operating System: macOS

Description

I am wondering if cattrs provides a clever built-in mechanism reuse/inherit overrides from another class. It sounds like an easy problem at first, but I really can't figure out a smart way to achieve it.

What I Did

Consider the following code example and let's assume I have an entire class hierarchy below FruitBasket. In this example, I want that apples are always unstructured as bananas no matter where we are in the class hierarchy. In addition, the subclasses themselves might want to add their own custom overrides:

import cattrs
from attrs import define
from cattrs.gen import make_dict_unstructure_fn


@define
class FruitBasket:
    apples: int


@define
class MixedBasket(FruitBasket):
    oranges: int


converter = cattrs.Converter()
converter.register_unstructure_hook(
    FruitBasket,
    make_dict_unstructure_fn(
        FruitBasket, converter, apples=cattrs.override(rename="bananas")
    ),
)
converter.register_unstructure_hook(
    MixedBasket,
    make_dict_unstructure_fn(
        MixedBasket, converter, oranges=cattrs.override(rename="peaches")
    ),
)

print(converter.unstructure(MixedBasket(apples=5, oranges=3)))

So the goal is that the latter gives a {'bananas': 5, 'peaches': 3}, which I can of course achieve by adding the override again to the second hook. However, my question is if there is an automatic way to do so, which doesn't require me to keep track of the overrides while going to down the hierarchy and register them again and again.

[Tinche]

So I gave this a little bit of thought. I don't think cattrs has anything that can particularly help you here, but I don't think we really need much from cattrs anyway.

Here's an approach I played around with. First, we can register a hook factory for all subclasses of FruitBasket.

converter.register_unstructure_hook_factory(
    lambda t: issubclass(t, FruitBasket),
    lambda t: make_dict_unstructure_fn(
        t, converter, apples=cattrs.override(rename="bananas")
    ),
)

Now, FruitBasket and all subclasses (including MixedBasket) will get hooks that rename apples to bananas.

As for also customizing MixedBasket. I refactored a little to this:

BASE_OVERRIDES = {"apples": cattrs.override(rename="bananas")}

converter = cattrs.Converter()
converter.register_unstructure_hook_factory(
    lambda t: issubclass(t, FruitBasket),
    lambda t: make_dict_unstructure_fn(t, converter, **BASE_OVERRIDES),
)
converter.register_unstructure_hook(
    MixedBasket,
    make_dict_unstructure_fn(
        MixedBasket,
        converter,
        oranges=cattrs.override(rename="peaches"),
        **BASE_OVERRIDES,
    ),
)

Now every subclass of FruitBasket gets bananas, and MixedBaskets in particular also get peaches. We just lean on how dictionaries compose in Python. There's a way to do this using function composition too but it seems more complex for little gain.

Let me know what you think.

[AdrianSosic]

Hi @Tinche, appreciate very much that you took the time to think about the problem 🥇

The solution you propose works, of course, and is pretty much exactly how I would have set it up without having access to better alternatives. However, I must admit that I don't consider it very elegant: The problem I see here is that it basically requires to mirror the entire class hierarchy to a second hierarchy of nested dictionaries, all of which needs to be manually maintained. For the very simple scenario above, we only need BASE_OVERRIDES, but now consider the my real use case, where FruitBasket not only has one but several subclasses, each of which also has subclasses etc. To consistently keep track of all overrides when going down the class hierarchy, you need an additional dict for each class you pass along the way. That doesn't seem like a neat solution to the problem... Ideally, I'd rather want to somehow make use of the overrides that have already been registered with converter when calling make_dict_unstructure_fn or somehow access them in another way. Do you see the issue?

[Tinche]

I've been giving this more thought since it's an interesting problem. I have a proposal ready, but it requires cattrs 24.1, which is still unreleased.

In cattrs 24.1, the hooks generated by cattrs.gen contain their overrides on an overrides attribute on the actual function. So when you do this:

from cattrs.gen import make_dict_unstructure_fn

hook = make_dict_unstructure_fn(..., a=override(rename=...))

the overrides can be fetched:

>>> print(hook.overrides)
{"a": AttributeOverride(...)}

We can leverage this.

First we need a base unstructure hook for FruitBasket, and only FruitBasket (no subclasses). We apply the apples -> bananas rename here.

conv = cattrs.Converter()

conv.register_unstructure_hook_func(
    lambda t: t is FruitBasket,
    make_dict_unstructure_fn(
        FruitBasket, conv, apples=cattrs.override(rename="bananas")
    ),
)

Simple so far. But subclasses (like MixedBasket) won't inherit the overrides. So now we define a hook factory for all subclasses of FruitBasket (but not FruitBasket exactly) that looks like this:

@conv.register_unstructure_hook_factory(
    lambda type: type is not FruitBasket and issubclass(type, FruitBasket)
)
def fruit_baskets_hook_factory(cl: type[FruitBasket], converter: cattrs.Converter):
    parent = cl.mro()[1]
    parent_hook = converter.get_unstructure_hook(parent)
    overrides = parent_hook.overrides
    return make_dict_unstructure_fn(cl, converter, **overrides)

What this hook does is the following: for every subclass of FruitBasket, it gets the hook for the parent from the converter, gets the overrides from that hook, and uses those overrides to create the hook for the subclass.

So now,

>>> print(conv.unstructure(MixedBasket(1, 2)))
{'bananas': 1, 'oranges': 2}

The bananas rename gets propagated.

We still need one more thing: the ability to add overrides to subclasses and have those overrides propagated. We can define our own function:

def register_unstructure_hook(
    cl: type[FruitBasket], conv: cattrs.Converter, **overrides: AttributeOverride
):
    parent = cl.mro()[1]
    parent_hook = conv.get_unstructure_hook(parent)
    overrides = parent_hook.overrides | overrides
    conv.register_unstructure_hook_func(
        lambda t: t is cl, make_dict_unstructure_fn(cl, conv, **overrides)
    )

It works similarly, combining the provided overrides with overrides from the parent hook.

Then:

>>> register_unstructure_hook(MixedBasket, conv, oranges=cattrs.override(rename="peaches"))
>>> print(conv.unstructure(MixedBasket(1, 2)))
{'bananas': 1, 'peaches': 2}

And because of the hook, subclasses will inherit the overrides.

This is pretty nifty, and I might make this a strategy in the future.