feat: replicate boundary vectors to multiple partitions #2258
Conversation
Signed-off-by: BubbleCal <[email protected]>
Signed-off-by: BubbleCal <[email protected]>
…e-boundary-vectors
| @@ -106,6 +113,80 @@ impl<T: ArrowFloatType + L2 + Dot> IvfTransformer<T> { | |||
|
|
|||
| UInt32Array::from_iter(result.iter().flatten().copied()) | |||
| } | |||
|
|
|||
| /// Compute the partition for each row in the input Matrix. | |||
There was a problem hiding this comment.
do we use this in search? or just in build time.
There was a problem hiding this comment.
only in build time
| let part_ids = self.compute_partitions(&mat).await; | ||
| let field = Field::new(PART_ID_COLUMN, part_ids.data_type().clone(), true); | ||
| Ok(batch.try_with_column(field, Arc::new(part_ids))?) | ||
| let part_ids = self.compute_multiple_partitions(&mat).await; |
There was a problem hiding this comment.
hmm, maybe we should use a different transformer for search?
There was a problem hiding this comment.
do we use transformer for search?
the code path should be IVFIndex::find_partitions -> IvfImpl::find_partitions -> kmeans::find_partitions right?
sure we can, just duplicated the code for now because |
Signed-off-by: BubbleCal <[email protected]>
No description provided.