Proposal: Remove Single Query Mode for Better Flexibility

[mesibo]

Hi, I am filing this issue to propose simplifying ann_benchmarks by maintaining only one mode instead of the current single and batch modes and presenting unified benchmarking results independent of mode (currently shown separately). This is based on our discussion on: #568

Currently, the single query mode feeds queries one by one to the algorithm and expects results synchronously, while in batch mode, ann_benchmarks feeds all queries at once to the algorithm and expects results synchronously. The problem arises when an algorithm processes vectors in parallel. In this case, single query mode becomes unusable because it requires immediate output for each query.

If we remove single query mode and keep only batch mode, algorithms would have the flexibility to process queries however they prefer—sequentially or in parallel—while still returning results in the same expected format.

This should not matter for ann_benchmarks since it is essentially the same operation—only the task of feeding query vectors moves from ann_benchmarks to the algorithm, allowing each algorithm to decide on its optimal processing strategy.

This change would also simplify ann_benchmarks as it does not need to support two modes and can present a unified benchmarking view across all algorithms, independent of their internal processing. Since only the final results matter for benchmarking, how the query should be processed should be determined by the algorithm implementation.

The same approach has been used for index building, where all vectors are fed in bulk to the fit() function, and we are proposing the same for queries too. Looking forward to your insights and perspective. We have made some minor changes for this, let me know if we can submit.

On a side note, we have found that batch mode currently does not generate all plots and reports.

[erikbern]

Not entirely sure that batch mode is representative of real-world use cases. Single-vector queries are arguably a more general operation so that's why we've been using those.

[mesibo]

Thanks Erik. While single-vector queries are common, displaying results separately for single and batch modes doesn't show the complete picture. So if some algorithms do not support single query methods and only batch, ann_benchmarks shows both benchmarks differently which doesn't represent the true picture. Or else ann_benchmarks need to run all algorithms in both modes, which is unnecessary.

Also, the ANN landscape is evolving, more and more modern ANN libraries are implementing multi-core parallel features which can't use single query mode. The major shortcoming of the single query mode is that it is synchronous and queries are executed in sequence, so even if the algorithm supports parallel processing, it is artificially limited from its true performance, as I explained in a separate issue #568.

Moreover, from ann_benchmarks perspective, what matters is the result and timing and not how queries are executed. So we suggest letting each algorithm handle queries in its optimal way rather than having ann_benchmarks control this. Those algorithms can do parallel processing will handle it paralley, those who can't, the change is minimal - just moving from:

self.p.query(vec, k=n)

to:

return np.array([self.p.query(vec, k=n)[0] for vec in v])

We have implemented these changes and would be happy to contribute the code, although the changes are trivial. We're also working on additional performance criteria like filtering performance.