feat: support aggregate function for vector. #463

my-vegetable-has-exploded · 2024-04-09T05:06:00Z

close #458

This pr mainly contains function including avg ，sum, vector_dims, vector_norm.

vector_dims(vector)

Returns the dimensions of a given vector.

Arguments：a vector.
Return type integer representing the number of dimensions of the given vector.

vector_norms(vector)

Calculates the Euclidean norm of a given vector.
Arguments:a vector.
Return double precision representing the Euclidean norm of the given vector.

AVG

Calculates the average of the processed vectors.
Arguments
vector: vectors.
Return vector representing the average of processed vectors.

SUM

Arguments： vectors.
Return type vector representing the sum of processed vectors.

The aggregate function support need to add a state transition function sfunc and an optional final calculation function ffunc, following create aggregate document. And we add COMBINEFUNC here to support parallel aggregate.

Signed-off-by: my-vegetable-has-exploded <[email protected]>

my-vegetable-has-exploded · 2024-04-09T08:56:43Z

I will update the documentation for these functions, as well as the documentation for veci8 later.

VoVAllen · 2024-04-09T12:36:24Z

src/datatype/functions_vecf32.rs

+/// accumulate intermediate state for vector average
+#[pgrx::pg_extern(immutable, strict, parallel_safe)]
+fn _vectors_vector_accum(
+    mut state: pgrx::composite_type!('static, "vectors.vector_accum_state"),


Is the schema name hardcoded now? @usamoi

src/sql/finalize.sql

src/datatype/functions_vecf32.rs

src/sql/finalize.sql

usamoi · 2024-04-10T03:43:50Z

Suggest add vector_l2_norm here. It helps users to take advantage of dot product as distance function.

my-vegetable-has-exploded · 2024-04-10T04:40:55Z

Suggest add vector_l2_norm here. It helps users to take advantage of dot product as distance function.

Is this same as vector_norm ?

usamoi · 2024-04-10T04:42:24Z

Suggest add vector_l2_norm here. It helps users to take advantage of dot product as distance function.

Is this same as vector_norm ?

#255

sorry, I refer to vector_abs_norm

my-vegetable-has-exploded · 2024-04-10T13:19:11Z

src/datatype/functions_vecf32.rs

+            .unwrap()
+            .unwrap();
+        check_matched_dims(sum.len(), value.dims());
+        // TODO: pgrx::Array<T> don't support mutable operations currently, we can reuse the state once it's supported.


I double check this solution. It seem that the memory management of AggState is well-designed. For transfn like sfunc( internal-state, next-data-values ) ---> next-internal-state , return internal-state after modification as result is just okay.
See https://github.com/postgres/postgres/blob/52b49b796cc7fd976f4da6aa49c9679ecdae8bd5/src/backend/executor/nodeAgg.c#L81-L100 and https://doxygen.postgresql.org/int8_8c.html#aff6c10f1da41c3aca8253c3e1d6f1378 for more details.
I am going to change the composite_type to a struct. And modify the state in-place.

my-vegetable-has-exploded · 2024-04-11T09:25:45Z

src/datatype/functions_vecf32.rs

+
+/// accumulate intermediate state for vector average
+#[pgrx::pg_extern(immutable, strict, parallel_safe)]
+fn _vectors_vector_accum<'a>(


I meet unused-lifetimes warning here. I don't configure out the reason here. As far as I know , this warning only occurs with #[pgrx::pg_extern].
The 235th line is the same.
cc @usamoi

The macro expands to

pub unsafe extern "C" fn _vectors_bvecf32_send_wrapper<'a>( _fcinfo: ::pgrx::pg_sys::FunctionCallInfo, ) -> ::pgrx::pg_sys::Datum

The lifetime parameter is redundant here. You could use allow to suppress this warning before pgrx fixes it.

I just discovered this problem too. After add #![allow(unused_lifetimes)] #![allow(clippy::extra_unused_lifetimes)] in file header, it works. Thanks!

Signed-off-by: my-vegetable-has-exploded <[email protected]>

my-vegetable-has-exploded · 2024-04-11T12:12:33Z

src/datatype/functions_vecf32.rs

@@ -0,0 +1,292 @@
+#![allow(unused_lifetimes)]


relate pgcentralfoundation/pgrx#1649

VoVAllen · 2024-04-18T06:50:33Z

@usamoi PTAL

src/datatype/functions_vecf32.rs

src/sql/bootstrap.sql

usamoi · 2024-04-18T07:21:25Z

src/datatype/functions_vecf32.rs

+use std::ptr::NonNull;
+
+#[repr(C, align(8))]
+pub struct AccumulateStateHeader {


It'll be better if it's more generic so that we use reuse it for more usage.

I think only AccumulateState for vecf32 and AccumulateState for vecf16 layout can be same? Just adding a generic type seems won't be so helpful?

Signed-off-by: my-vegetable-has-exploded <[email protected]>

VoVAllen · 2024-05-06T04:19:23Z

@usamoi @my-vegetable-has-exploded Is this ready to merge?

my-vegetable-has-exploded · 2024-05-06T06:14:34Z

@usamoi @my-vegetable-has-exploded Is this ready to merge?

I think it's ready to go now.

* implement vector avg. Signed-off-by: my-vegetable-has-exploded <[email protected]> * Implement sum、 vector_dims()、 vector_norm() Signed-off-by: my-vegetable-has-exploded <[email protected]> * update AccumulateState Signed-off-by: my-vegetable-has-exploded <[email protected]> * fix norm. Signed-off-by: my-vegetable-has-exploded <[email protected]> * add l2_norm for VectorBorrowed. Signed-off-by: my-vegetable-has-exploded <[email protected]> --------- Signed-off-by: my-vegetable-has-exploded <[email protected]> Signed-off-by: jinweios <[email protected]>

my-vegetable-has-exploded added 2 commits April 9, 2024 04:55

implement vector avg.

c701f78

Signed-off-by: my-vegetable-has-exploded <[email protected]>

Implement sum、 vector_dims()、 vector_norm()

b8aad70

Signed-off-by: my-vegetable-has-exploded <[email protected]>

VoVAllen reviewed Apr 9, 2024

View reviewed changes

VoVAllen requested a review from usamoi April 9, 2024 13:50

usamoi reviewed Apr 10, 2024

View reviewed changes

src/sql/finalize.sql Outdated Show resolved Hide resolved

src/sql/finalize.sql Outdated Show resolved Hide resolved

src/datatype/functions_vecf32.rs Outdated Show resolved Hide resolved

src/sql/finalize.sql Outdated Show resolved Hide resolved

usamoi mentioned this pull request Apr 10, 2024

Tracking issue for schema #255

Open

14 tasks

my-vegetable-has-exploded commented Apr 10, 2024

View reviewed changes

my-vegetable-has-exploded marked this pull request as draft April 10, 2024 15:23

my-vegetable-has-exploded force-pushed the vector-agg branch from c5ad8a2 to de0f000 Compare April 11, 2024 09:20

my-vegetable-has-exploded marked this pull request as ready for review April 11, 2024 09:21

my-vegetable-has-exploded commented Apr 11, 2024

View reviewed changes

update AccumulateState

4f8bc4f

Signed-off-by: my-vegetable-has-exploded <[email protected]>

my-vegetable-has-exploded force-pushed the vector-agg branch from de0f000 to 4f8bc4f Compare April 11, 2024 11:50

my-vegetable-has-exploded added 2 commits April 11, 2024 12:11

fix norm.

58f06db

Signed-off-by: my-vegetable-has-exploded <[email protected]>

Merge remote-tracking branch 'origin/main' into vector-agg

c28e1d1

my-vegetable-has-exploded commented Apr 11, 2024

View reviewed changes

my-vegetable-has-exploded requested review from VoVAllen and usamoi April 11, 2024 12:22

usamoi reviewed Apr 18, 2024

View reviewed changes

add l2_norm for VectorBorrowed.

c68bbe9

Signed-off-by: my-vegetable-has-exploded <[email protected]>

my-vegetable-has-exploded force-pushed the vector-agg branch from da65beb to c68bbe9 Compare April 19, 2024 04:57

usamoi approved these changes May 6, 2024

View reviewed changes

VoVAllen added this pull request to the merge queue May 6, 2024

Merged via the queue into tensorchord:main with commit d81473c May 6, 2024
13 checks passed

my-vegetable-has-exploded mentioned this pull request May 12, 2024

Implement aggregate functions for sparse vector. #476

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support aggregate function for vector. #463

feat: support aggregate function for vector. #463

my-vegetable-has-exploded commented Apr 9, 2024 •

edited

Loading

my-vegetable-has-exploded commented Apr 9, 2024

VoVAllen Apr 9, 2024

usamoi Apr 9, 2024

usamoi commented Apr 10, 2024

my-vegetable-has-exploded commented Apr 10, 2024

usamoi commented Apr 10, 2024 •

edited

Loading

my-vegetable-has-exploded Apr 10, 2024

my-vegetable-has-exploded Apr 11, 2024 •

edited

Loading

usamoi Apr 11, 2024

my-vegetable-has-exploded Apr 11, 2024

my-vegetable-has-exploded Apr 11, 2024

VoVAllen commented Apr 18, 2024

usamoi Apr 18, 2024

my-vegetable-has-exploded Apr 19, 2024

VoVAllen commented May 6, 2024

my-vegetable-has-exploded commented May 6, 2024

feat: support aggregate function for vector. #463

feat: support aggregate function for vector. #463

Conversation

my-vegetable-has-exploded commented Apr 9, 2024 • edited Loading

vector_dims(vector)

vector_norms(vector)

AVG

SUM

my-vegetable-has-exploded commented Apr 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

usamoi commented Apr 10, 2024

my-vegetable-has-exploded commented Apr 10, 2024

usamoi commented Apr 10, 2024 • edited Loading

Choose a reason for hiding this comment

my-vegetable-has-exploded Apr 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VoVAllen commented Apr 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VoVAllen commented May 6, 2024

my-vegetable-has-exploded commented May 6, 2024

my-vegetable-has-exploded commented Apr 9, 2024 •

edited

Loading

usamoi commented Apr 10, 2024 •

edited

Loading

my-vegetable-has-exploded Apr 11, 2024 •

edited

Loading