Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrapper function/type for creating Scalars #787

Open
bjchambers opened this issue Oct 2, 2023 · 2 comments
Open

Wrapper function/type for creating Scalars #787

bjchambers opened this issue Oct 2, 2023 · 2 comments

Comments

@bjchambers
Copy link
Collaborator

          Could add a helper for creating the static somewhere, seen it in a few instances now

Originally posted by @jordanrfrazier in #784 (comment)

@bjchambers
Copy link
Collaborator Author

Arrow recently added Scalar which simplifies many of the APIs handling scalar values. We could potentially use this to replace our ScalarValue enum.

See https://github.com/apache/arrow-rs/pull/4793/files for some notes:

  1. We can use Int64Array::new_scalar(...) to simplify creating a scalar of a specific type.
  2. We can create a Scalar<ArrayRef> for using an array as a scalar.

This should let us create something like:

struct ScalarValue(Scalar<ArrayRef>);

// 1. Serde the underlying ArrayRef
// 2. Support use as a scalar anywhere

@bjchambers
Copy link
Collaborator Author

@jordanrfrazier FYI -- I think we should be able to completely replace our ScalarValue enum with Scalar<ArrayRef>. I think it's probably worth doing that for how we serialize scalars in the physical plan.

Thoughts on:

A) Trying to change the ScalarValue to use that representation (potentially messy since we have that protobuf in many places0.
B) Introducing a parallel Scalar??? to represent that?
C) Just using arrow_array::Scalar<arrow_array::ArrayRef> in those places?

I think with C we'd need to do the custom serialization like we did for structs with an ArrayRef, while with A or B we could encapsulate that in our serialization of the wrapper type we create.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant