[SPARK-48008][WIP] Support UDAFs in Spark Connect

[xupefei]

What changes were proposed in this pull request?

This PR changes Spark Connect to support defining and registering Aggregator[IN, BUF, OUT] UDAFs.
The mechanism is similar to supporting Scaler UDFs. On the client side, we serialize and send the Aggregator instance to the server, where the data is deserialized into an Aggregator instance recognized by Spark Core.
With this PR we now have two Aggregator interfaces defined, one in Connect API and one in Core. They define exactly the same abstract methods and share the same SerialVersionUID, so the Java serialization engine could map one to another. It is very important to keep these two definitions always in sync.

A follow-up to this PR is to add Aggregator.toColumn API (now NotImplemented due to deps to Spark Core).

Why are the changes needed?

Spark Connect does not have UDAF support. We need to fix that.

Does this PR introduce any user-facing change?

Yes, Connect users could now define an Aggregator and register it:

val agg = new Aggregator[INT, INT, INT] { ... }
spark.udf.register("agg", udaf(agg))
val ds: Dataset[Data] = ...
val aggregated = ds.selectExpr("agg(i)")

How was this patch tested?

Added new tests.

Was this patch authored or co-authored using generative AI tooling?

Nope.

[hvanhovell]

Why is this needed?

[hvanhovell]

TypedColumn?

[xupefei]

It's for mapping the client's Aggregator class to this one. This is required because we now serialise the whole Aggregator instance on the client side.

[xupefei]

I tried multiple methods and still can't succeed without this UID. As long as we want users to define UDAF like this:

new Aggregator {
   def merge(...) = {...}
   ...
}

The serialized payload (either the whole Aggregator instance or individual methods (e.g., agg.merge _)) will carry a reference to the Aggregator instance that needs to be decoded on the server side. Without this UID, the decode will fail with an error message class UID not match.

[hvanhovell]

Can we move this common instead of having two abstract classes?

[xupefei]

Yes, that would be ideal. I was doing that before until I found that the Connect client should have another docstring @since 4.0.0. Could you suggest how could we document this on the client side if this class is moved to Common?

[hvanhovell]

How should this work on the connect side?

[hvanhovell]

I think it should take a wildcard, we have done similar things before.