Unexpected performance results for jvector on 100M dataset compared to on-disk knn

[jinjunzh]

Is there any benchmark performance comparison between the current jvector plugin and the on-disk mode with knn plugin? I conducted a comparative test but did not observe a clear advantage of jvector in terms of performance and resource usage. Does this align with expectations?
On the SIFT 1M dataset, I can see the benefits of jvector. However, at the 100M scale, jvector's performance is an order of magnitude worse than the on-disk approach, even under memory-constrained conditions with high page misses.

Additionally, I observed that during query stress testing under memory constraints, jvector's disk I/O throughput is an order of magnitude higher than the on-disk method, along with higher JVM heap usage and storage space consumption by an order of magnitude.
My understanding is that jvector should demonstrate better competitive advantages at ultra-large data scales. However, the results above do not support this. Are there any other explanations or suggestions?

[akash-shankaran]

Hi @jinjunzh - thanks for bringing this to our attention. This is certainly not the expectations. I wonder if there is some out of box settings that we can adjust, to improve performance at this scale.

Can you share more details about your experiment:

1. machine specs - disk, memory, cores etc?
2. opensearch parameters - What is the dataset, and the parameters used to construct the indices? Do you force merge to single segment before searching?
3. What is the version of the plugin you're using for this test?
4. Can you share heap usage, and storage increase data?

If you have used block store for your tests (EBS), consider using NVME disk with JVector as that will drastically change perf. The higher I/O that you're observing should come down due to this.

To answer some of your questions:

Is there any benchmark performance comparison between the current jvector plugin and the on-disk mode with knn plugin?

We have not done such studies so far.

cc: @sam-herman