enh: internals pluggable backends #2001

camriddell · 2025-02-13T00:01:37Z

What type of PR is this? (check all applicable)

Related issues

Related issue #<issue number>
Closes #<issue number>

Checklist

Code follows style guide (ruff)
Tests added
Documented the changes

If you have comments or can explain your changes, please do so below

This design is not final and some functions need to move around, any discussion is much appreciated. It is a prototype to remove human-readable complexity out of the from_native function into a data-oriented approach.

Currently, adding in a new backend requires developers to touch many lines of code across different modules and it can be tricky to figure out where exactly to look. This is an attempt to consolidate all of this meta-data into a backends.py file.

Features that can be very readily implemented here

Hand the adapter class (e.g. PandasLikeDataFrame) the appropriate Backend object which enables easy access to information like the namespace Backend.native_namespace as well as the the version number in Backend.version(). The Backend can even validate its own versions Backen.validate_backend_versions (which now that I type it out should probably just be Backend.validate_versions)
Support for having multiple requirements for a particular Backend- with the dask.dataframe backend there is an implicit dependency on dask_expr that goes unnoticed until from_native tells one to install it. This approach consolidates that information
Flexibility- adding a new Backend can be done completely without touching narwhals internals.

import narwhals as nw
from narwhals.backends import Backend, Adaptation, register_backends
import numpy as np

class NumPySeries:
    def __init__(self, native_series, backend_version, version):
        self._native_series = native_series

    def __narwhals_series__(self):
        return self

    def mean(self):
        return self._native_series.mean()

    # numpy ndarrays do not implement std as a method
    def std(self, ddof=1):
        return np.std(self._native_series, ddof=ddof)


register_backends(
    Backend(
        requires=[
            ("numpy", "numpy.__version__", (1, 20))
        ],
        adaptations=[
            Adaptation(
                # link a Narwhals Series to a numpy.ndarray via NumPySeries
                nw.Series, "numpy.ndarray", NumPySeries, level="full"
            ),
        ],
    )
)


array = np.array([1,2,3,4])
series = nw.from_native(array, series_only=True)
print(series)
# ┌───────────────────┐
# |  Narwhals Series  |
# |-------------------|
# |array([1, 2, 3, 4])|
# └───────────────────┘


print(
    f'{array.mean()  = }',
    f'{series.mean() = }',
    f'{np.std(array, ddof=1) = }',
    f'{series.std()          = }',
    sep='\n'
)

# array.mean()  = np.float64(2.5)
# series.mean() = np.float64(2.5)
# np.std(array, ddof=1) = np.float64(1.2909944487358056)
# series.std()          = np.float64(1.2909944487358056)

FBruzzesi

Hey @camriddell thanks for the effort, it is really 🔥

I really like the idea of simplifying the user experience to extend narwhals - even if it was just for our own use, I believe it is worth it.

I could not stop myself to leave some comments unrelated from the design itself - feel free to just ignore those for now.

For me the only one thing I am not fully sold in is having some "hidden" behavior for the first tuple of requires i.e.

requires=[
    ("pandas", "pandas.__version__", (0, 25, 3)),
    ("foo", "foo.__version__", (0, 0, 1)),
],

and

requires=[
    ("foo", "foo.__version__", (0, 0, 1)),
    ("pandas", "pandas.__version__", (0, 25, 3)),
],

lead to very different outcomes.

I proposed another approach which is a bit more explicit for the requirement that corresponds to the library dataframe backend. Let me know what are your thoughts on the topic :)

FBruzzesi · 2025-02-14T21:42:51Z

narwhals/dataframe.py

+        from narwhals.translate import to_native
+
        return to_native(narwhals_object=self, pass_through=False)


This can also just be:

Suggested change

from narwhals.translate import to_native

return to_native(narwhals_object=self, pass_through=False)

return self._compliant_frame._native_frame

FBruzzesi · 2025-02-14T21:43:30Z

narwhals/backends.py

+    for b in backends:
+        BACKENDS.append(b)  # noqa: PERF402


nitpick:

Suggested change

for b in backends:

BACKENDS.append(b) # noqa: PERF402

BACKENDS.extend(backends)

🤦 not a nitpick, you can even see that I ignored ruff while working quickly to get this up.

On this topic, do you have any thoughts about BACKENDS being a mutable global entity? I am stuck on this consideration vs making it immutable and allowing users to pass extra backends through the from_native function.

On this topic, do you have any thoughts about BACKENDS being a mutable global entity?

I feel uncomfortable as well with having such global variable.
I am not very familiar with the pattern - but what about something like a BackendRegistry class in which it's possible to set/register backends but not delete them (explicitly)? I am thinking out loud here

FBruzzesi · 2025-02-14T21:47:16Z

narwhals/backends.py

+BACKENDS = []
+
+
+@dataclass


Personally it would be nice to have it kw_only - although I know that's supported from python 3.10 for dataclasses 🥲

FBruzzesi · 2025-02-14T21:48:29Z

narwhals/backends.py

+
+
+@dataclass
+class Adaptation:


TODO: some docstring 🙃

FBruzzesi · 2025-02-14T21:51:26Z

narwhals/backends.py

+            elif adapt.version is None:
+                adaptations.extend(replace(adapt, version=v) for v in Version)
+            else:
+                msg = "Adaptation.version must be {Version!r} or None, got {adapt.version!r}"


Suggested change

msg = "Adaptation.version must be {Version!r} or None, got {adapt.version!r}"

msg = f"Adaptation.version must be {Version!r} or None, got {adapt.version!r}"

FBruzzesi · 2025-02-14T21:56:00Z

narwhals/backends.py

+    adapter: str | type
+    level: str
+    kwargs: dict[str, Any] = field(default_factory=dict)
+    version: Version | None = None


Not sure if we should allow version to be None. After all, we set that based upon from where from_native is called 🤔

FBruzzesi · 2025-02-14T21:59:19Z

narwhals/backends.py

+
+@dataclass
+class Backend:
+    requires: list[tuple[str, str | Callable, tuple[int, ...]]]


Might be worth considering a list[namedtuple] to improve readability

FBruzzesi · 2025-02-14T21:59:59Z

narwhals/backends.py

+            elif isinstance(version_getter, str):
+                version_str = dynamic_import(version_getter)
+            else:
+                msg = "version_getter {version_getter!r} must be a string or callable, got {type(version_getter)}"


Suggested change

msg = "version_getter {version_getter!r} must be a string or callable, got {type(version_getter)}"

msg = f"version_getter {version_getter!r} must be a string or callable, got {type(version_getter)}"

🙈

FBruzzesi · 2025-02-14T22:10:56Z

narwhals/backends.py

+        return import_module(self.requires[0][0])
+
+    def get_native_namespace(self) -> ModuleType | None:
+        return sys.modules.get(self.requires[0][0], None)


General comment, for which I didn't really think of an alternative, is that I don't like too much the fact that self.requires is used implicitly 3 times with the first item in the list.

One idea could be to have a distinction between (let's maybe get better names):

backend_library -> this is where the adaptation.native class comes from

In the proposed implementation this is dynamically extracted from adaptation.native, but then for native_namespace the first tuple in requires is used

extra_requires -> potential list of extra requirements

FBruzzesi · 2025-02-14T22:11:34Z

narwhals/backends.py

+            for attr in attributes:
+                obj = getattr(obj, attr)
+            return obj
+    msg = "Could not import {dotted_path!r}"


Ok last one

Suggested change

msg = "Could not import {dotted_path!r}"

msg = f"Could not import {dotted_path!r}"

…ementation

- split Backend requirements into the core package requirement and extras (would be nice to refactor this out once the Implementation enum is refactored) - add Requirement class to store backend package/module requirements - add MROAdaptation class for matching against inherited types - BACKENDS represented by a deque for priority left/right appensions - refactor adaptation matching logic to exist in the Adapatation class - refactor version validation to Backend class

…ementation

camriddell · 2025-03-10T19:59:15Z

@FBruzzesi I have an update with feedback from your comments incorporated here! Still need to add tests for the backend specific code but wanted to prioritize an API we like.

I have also added in @dangotbanned since he has done some recent work on the internals as well and want to make sure this change aligns well with that work too.

dangotbanned · 2025-03-10T20:10:24Z

#2001 (comment)

Thanks for the ping @camriddell

I've only skimmed through but I don't think this conflicts with what I've been working on 👍
The biggest potentially upcoming change would be the stuff in nw._compliant.* in https://github.com/narwhals-dev/narwhals/pull/2149/files

Somehow you've managed to touch code that I haven't over there 😄

dangotbanned · 2025-03-10T20:23:53Z

I think this issue might be relevant, wrt from_native

API: io functions for v2 #2116

I'm exploring that a little bit locally at the moment - but mostly in terms of writing reusable Protocol definitions

enh pluggable backends draft

76947c0

FBruzzesi reviewed Feb 14, 2025

View reviewed changes

camriddell added 3 commits March 4, 2025 11:45

Merge branch 'main' of github.com:narwhals-dev/narwhals into enh/impl…

278825f

…ementation

Merge branch 'main' of github.com:narwhals-dev/narwhals into enh/impl…

ef189da

…ementation

camriddell requested a review from dangotbanned March 10, 2025 19:58

dangotbanned added api design internal labels Mar 10, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enh: internals pluggable backends #2001

enh: internals pluggable backends #2001

camriddell commented Feb 13, 2025

FBruzzesi left a comment

FBruzzesi Feb 14, 2025

FBruzzesi Feb 14, 2025

camriddell Feb 14, 2025

FBruzzesi Feb 15, 2025

FBruzzesi Feb 14, 2025

FBruzzesi Feb 14, 2025

FBruzzesi Feb 14, 2025

FBruzzesi Feb 14, 2025

FBruzzesi Feb 14, 2025

FBruzzesi Feb 14, 2025

FBruzzesi Feb 14, 2025

FBruzzesi Feb 14, 2025

camriddell commented Mar 10, 2025

dangotbanned commented Mar 10, 2025 •

edited

Loading

dangotbanned commented Mar 10, 2025

		from narwhals.translate import to_native

		return to_native(narwhals_object=self, pass_through=False)

	for b in backends:
	BACKENDS.append(b) # noqa: PERF402
	BACKENDS.extend(backends)

	msg = "Adaptation.version must be {Version!r} or None, got {adapt.version!r}"
	msg = f"Adaptation.version must be {Version!r} or None, got {adapt.version!r}"

	msg = "version_getter {version_getter!r} must be a string or callable, got {type(version_getter)}"
	msg = f"version_getter {version_getter!r} must be a string or callable, got {type(version_getter)}"

	msg = "Could not import {dotted_path!r}"
	msg = f"Could not import {dotted_path!r}"

		BACKENDS = []


		@dataclass

enh: internals pluggable backends #2001

Are you sure you want to change the base?

enh: internals pluggable backends #2001

Conversation

camriddell commented Feb 13, 2025

What type of PR is this? (check all applicable)

Related issues

Checklist

If you have comments or can explain your changes, please do so below

FBruzzesi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

camriddell commented Mar 10, 2025

dangotbanned commented Mar 10, 2025 • edited Loading

dangotbanned commented Mar 10, 2025

dangotbanned commented Mar 10, 2025 •

edited

Loading