Reduce the number of entries available in the top level API

[joelostblom]

Currently, the top level of the Altair API is huge, there are 573 entries in the tab completion menu of alt.<TAB>. This can make it rather hard to find what you're looking for unless you know the first few character of the name already, and exploring the API from scratch as a new user must be intimidating. Many of these seem unhelpful to have exposed at the top-level, e.g. there are 70 different entries that relate to conditions, but I only use alt.condition myself. I suggest that the top level API only includes the most relevant classes and functions that we think are helpful on a day to day basis, and the rest go under sub-modules.

The Altair API consists of four parts:

1. API functions
2. Top level objects
3. Encoding channels
4. Low level schema wrappers

I have listed my comments below on how we could reduce these, and I would love to hear what you think @mattijn, @ChristopherDavisUCI, @binste (and anyone else that wants to chime in)

Note: In all the changes I suggest below we should not break any old code, just remove the class/function from tab completion and link it to the new place in the API where this class/function now resides.

API functions

Introduced specifically by Altair as conveniences and we should keep all of them (although maybe check_fields_and_encodings is meant to be a private function based on the name and since there is no docstring?).

(just showing the first few)

Top level objects

The are only eight of these and we can probably keep all of them. Alternatively, since all except Chart have corresponding API functions and are rarely used directly we could move them to a charts sub domain: alt.charts.HConcatChart, etc, to make it more clear that the preferred usage is the API method alt.hconcat, etc.

Encoding channels

There is wasted space here by having each channel repeated three times, e.g. Angle, AngleDatum, AngleValue. Especially as the most common use of datum and value is by using the top level API functions alt.datum and alt.value. I see to it possible ways of improving the situation:

1. Move the top level classes to methods on the corresponding encoding class: instead of AngleDatum we have Angle.datum.
2. Moved the top level classes to methods on the API function: instead of AngleDatum, we would have alt.datum.Angle.

Maybe the second is lightly more natural in syntax, but it will probably make it confusing since you can use datum.<colum_name> in expression strings, so the first would be better for that reason.

(just showing the first few)

Low level schema wrappers

This is where we could make the most progress. There are around 400 low level schemer wrappers of which I've only used around 10 myself, most of which are now unnecessary since we added the method based syntax for channel encoding options.

We should identify which of these are just internal helper classes for Altair that end users rarely need to know about. To me it seems like almost everything here could be moved to a submodule, except the classes that end in Params (although they could also be moved if we introduce aliases).

I don't know what a good name of of this submodule might be: maybe alt.extras, alt.schema_wrappers, alt.misc, alt.wrappers, or similar?

(just showing the first few)

---

Again, none of these changes should break old code. For example, if we move FillDatum to alt.Fill.datum, then there should no longer be any tab completion for alt.FillDatum, but old code that use it should work as it just calls alt.Fill.datum under the hood. The tab completion should be on alt.Fill.<tab> instead, which also seems more natural to me.

[binste]

I really like the idea of cleaning up the top-level API! Thanks for bringing this up. I also find it quite overwhelming to go through the API references in the docs and to use autocompletion if you don't specifically know what you are looking for (and therefore I never use completion on alt.Chart to browse options).

In favour of cleaning up the API references in the docs which we can maybe do without any code changes except to generate_api_docs.py. I can't quite follow your example of refactoring FillDatum without breaking old code, could you expand on this a bit? Would you rename that class or just move it to a new submodule and still import it in the top-level?

Exclusion from tab completion is a bit tricky. As we noticed in #2814, hiding in Jupyter works great but we haven't yet found a way to do the same in VS Code. Of course still worth the effort to adjust __all__ if at least Jupyter users can profit from it.

[joelostblom]

Would you rename that class or just move it to a new submodule and still import it in the top-level?

My hope was to move it to a submodule and import it at top level, but hide it from the top level tab completion and only allow tab completion via the submodule. I think this would allow for old code to work, while also encouraging people to use the submodules from now on as well as reduce the top-level tab completion.

Exclusion from tab completion is a bit tricky. As we noticed in #2814, hiding in Jupyter works great but we haven't yet found a way to do the same in VS Code.

I think it would be great to support VS Code too, but if there isn't a way that pylance offers for hiding individual classes/functions, I am not sure there is much we can do until they add that. I tried asking in the discussion you linked before, let's see if there is a way.

[joelostblom]

Based on the helpful reply in microsoft/pylance-release#3709 (reply in thread) it seems like we need to get #2614 or similar merged, but maybe it is worthwhile if it resolves some of the VS Code issues we have been seeing and makes it easier to maintain both VS Code and JupyterLab support?

[binste]

Some notes from an offline discussion with @mattijn and @arvind:

• There is potential in Vega-Lite to simplify the types, i.e. reduce the number of types. This would automatically improve the situation here. This can happen in VL as a first step of a potentially bigger rewrite/refactoring. However, this first requires the appropriate funding and resources so it's not around the corner.
• Instead of doing it in VL first, we have the flexibility of doing it directly in Altair as discussed above.

What is very unlikely to break users code but would improve the situation a lot is removing classes from the top-level namespace which are never used by users anyway as they just represent a primitive datatype such as int, float. Examples: ConditionalAxisPropertyTextBaselinenull, DictSelectionInit, ... There are hundreds of these.

A first step could be to analyze which class names only appear ones in our code base, i.e. are defined but never used anywhere else, not even in a type hint. This can be a manual list and with every VL release, we could check in the PR if new entries get added to the top-level, if we really want them there.

Maybe we can remove these classes completely instead of just not making them available in the top-level namespace. This would make the Altair wheels even smaller. It might require a bit of a rewrite of from_dict.

[binste]

Due to the many objects, Altair takes up 23MB of memory (if I measure this correctly?) and this excludes it's requirements which are small:

from memory_profiler import profile


@profile
def import_packages():
    import typing_extensions
    import jinja2
    import jsonschema
    import packaging
    import narwhals

    import altair


if __name__ == "__main__":
    import_packages()

[dangotbanned]

Some notes from an offline discussion with @mattijn and @arvind:

There is potential in Vega-Lite to simplify the types, i.e. reduce the number of types. This would automatically improve the situation here. This can happen in VL as a first step of a potentially bigger rewrite/refactoring. However, this first requires the appropriate funding and resources so it's not around the corner.

Exciting to hear this being discussed on the Vega-Lite-side!

@binste is there any chance this relates to better support for generic types in more recent drafts of JSON Schema?

A while back I tried exploring what we could do in altair to reduce the number of symbols, but it started getting tricky due to https://github.com/vega/vega-lite/blob/main/scripts/rename-schema.sh

Screenshots

If there is a v6 on the table, I do think it would be worth reviewing how we can better utilise typing features - that wouldn't have been available in earlier major versions of altair