Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add metrics dot and cos #566

Merged
merged 4 commits into from
Sep 18, 2024
Merged

feat: add metrics dot and cos #566

merged 4 commits into from
Sep 18, 2024

Conversation

cutecutecat
Copy link
Member

@cutecutecat cutecutecat commented Aug 22, 2024

Error bound Check

Check the probability that the inequality does not hold:
$est - err * 2.9 < real < est + err * 2.9$

Projection With Identity Matrix

vector + dim metric errors(attempt 1) errors(attempt 2) errors(attempt 3)
glove + 100 dot 19541 = 2.0% 29535 = 2.9% 19099 = 2.0%
glove + 100 cos 19656 = 2.0% 27026 = 2.7% 23352 = 2.3%
sift + 128 dot 100241 = 10.0% 104076 = 10.4% 97756 = 9.8%
sift + 128 cos 77400 = 7.7% 74132 = 7.4% 77400 = 7.8%
sift + 128 l2 104613 = 10.5% 88878 = 8.9% 135172 = 13.5%

Projection With Random Matrix

vector + dim metric errors(attempt 1) errors(attempt 2) errors(attempt 3)
glove + 100 dot 26702 = 2.7% 64816 = 6.5% 42559 = 4.3%
glove + 100 cos 25517 = 2.6% 11341 = 1.1% 54328 = 5.4%
sift + 128 dot 64626 = 6.5% 49139 = 4.9% 103020 = 10.3%
sift + 128 cos 88658 = 8.9% 92813 = 9.2% 89247 = 9.1%
sift + 128 l2 66545 = 6.7% 78286 = 7.8% 86253 = 8.6%

Benchmark

CPP

  • RabitQ(sift+128+L2): QPS 1816 recall 0.9964

and

  • RabitQ(gist+960+L2): QPS 414.02 recall 0.9789

Main

  • CLI(sift+128+L2): QPS 1031 recall 0.9927
  • PYO3(sift+128+L2): QPS 911 recall 0.9927

and

  • CLI(gist+960+L2): QPS 361.20 recall 0.9763
  • PYO3(sift+960+L2): QPS 323.15 recall 0.9763

With this PR

  • CLI(sift+128+L2): QPS 1059 recall 0.9925
  • CLI(sift+128+Dot): QPS 1024 recall 0.9479(L2 groundtruth)

and

  • CLI(gist+960+L2): QPS 350.68 recall 0.9766
  • CLI(gist+960+Dot): QPS 387.65 recall 0.0031(L2 groundtruth)

(Updated 9/13)With this PR

patch: no-residual

tag top QPS recall
Glove-200-l2-main 100 274 0.9002
Glove-200-l2-main-patch 100 331 0.9010
Glove-200-l2-PR 100 284 0.9019
Glove-200-l2-PR-patch 100 344 0.9010
Glove-200-cos-PR 100 285 0.9014
Glove-200-cos-PR-patch 100 342 0.9012

@cutecutecat cutecutecat force-pushed the metric branch 5 times, most recently from 9171b14 to 563cb9f Compare August 28, 2024 09:08
@cutecutecat cutecutecat marked this pull request as ready for review August 28, 2024 11:22
@cutecutecat cutecutecat requested review from usamoi and removed request for usamoi August 28, 2024 11:24
@cutecutecat cutecutecat marked this pull request as draft August 28, 2024 12:20
@cutecutecat cutecutecat requested a review from usamoi August 29, 2024 03:01
@cutecutecat cutecutecat marked this pull request as ready for review August 29, 2024 03:01
@cutecutecat cutecutecat marked this pull request as draft August 30, 2024 09:28
@cutecutecat cutecutecat removed the request for review from usamoi August 30, 2024 09:28
@cutecutecat
Copy link
Member Author

cutecutecat commented Aug 30, 2024

Implement

  • precompute of centroids_square at kmeans_lookup_many_by_dot

@cutecutecat cutecutecat force-pushed the metric branch 4 times, most recently from 99dc755 to 8774391 Compare September 2, 2024 06:48
@cutecutecat cutecutecat marked this pull request as ready for review September 2, 2024 07:00
@cutecutecat cutecutecat requested a review from usamoi September 2, 2024 07:00
@cutecutecat cutecutecat force-pushed the metric branch 4 times, most recently from e2d37e8 to a21f770 Compare September 4, 2024 09:22
@cutecutecat cutecutecat removed the request for review from usamoi September 4, 2024 09:23
@cutecutecat cutecutecat marked this pull request as draft September 4, 2024 09:24
@cutecutecat cutecutecat force-pushed the metric branch 2 times, most recently from 0552a33 to 1a90e10 Compare September 10, 2024 01:56
@cutecutecat cutecutecat marked this pull request as ready for review September 10, 2024 02:09
@cutecutecat cutecutecat requested a review from usamoi September 10, 2024 02:09
@cutecutecat cutecutecat force-pushed the metric branch 2 times, most recently from 51ebe23 to 0282f20 Compare September 12, 2024 14:36
@usamoi
Copy link
Collaborator

usamoi commented Sep 13, 2024

Please remove code that is used for dot distance + residual vector quantization.

@usamoi usamoi removed their request for review September 13, 2024 10:06
@cutecutecat cutecutecat requested a review from usamoi September 14, 2024 06:29
@usamoi
Copy link
Collaborator

usamoi commented Sep 14, 2024

Why there is centroids_squares? If it's meaningless, remove it.

@usamoi usamoi removed their request for review September 14, 2024 07:31
@cutecutecat cutecutecat force-pushed the metric branch 3 times, most recently from 4ecc9ab to 60a5acc Compare September 14, 2024 10:42
@cutecutecat cutecutecat requested a review from usamoi September 14, 2024 10:53
@usamoi
Copy link
Collaborator

usamoi commented Sep 18, 2024

Please resolve conflicts.

@usamoi usamoi added this pull request to the merge queue Sep 18, 2024
Merged via the queue into tensorchord:main with commit bb46189 Sep 18, 2024
7 checks passed
@cutecutecat cutecutecat deleted the metric branch November 21, 2024 03:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants