SOLR-17319 : Combined Query Feature for Multi Query Execution by ercsonusharma · Pull Request #3418 · apache/solr

ercsonusharma · 2025-07-04T09:19:21Z

https://issues.apache.org/jira/browse/SOLR-17319

Description

This feature aims to execute multiple queries of multiple kinds across multiple shards of a collection and combine their result basis an algorithm (like Reciprocal Rank Fusion). It also help resolve the issues being discussed w.r.t the previous PR, mainly around across shard documents merging. It provides more flexibility in terms of querying extending JSON Query DSL ultimately enabling Hybrid Search in a pure way solving the shortcomings.

Note: This feature is currently unsupported for non-distributed and grouping query.

Solution

Extended the QueryComponent to create new CombinedQueryComponent and ResponseBuilder to create new CombinedQueryResponseBuilder supports multiple response builders to hold the state and execute multiple queries.
In JSON Query DSL, a parameter is added to identity Combined Query request and basis that it invokes the new CombinedQueryComponent
CombinedQueryComponent have multiple response builders assigned for each query. These queries are first executed at the SolrSearchIndexer level and combined them using RRF for now.
At Shard level also, the responses for the multiple queries are merged.

Tests

Added tests for testing the RRF logic independently.
Added tests for across search index and distributed requests.
Added tests to assert existing behaviour of search handler's QueryComponent as well as for the newly added CombinedQueryComponent basis the flag in json query DSL.

Checklist

Please review the following and check all that apply:

I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
I have created a Jira issue and added the issue ID to my pull request title.
I have given Solr maintainers access to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
I have developed this patch against the main branch.
I have run ./gradlew check.
I have added tests for my changes.
I have added documentation for the Reference Guide

solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java

solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java

solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java

ercsonusharma · 2025-07-09T17:21:12Z

@alessandrobenedetti @dsmiley, please help review it whenever you can. Thanks!

solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java

solr/core/src/java/org/apache/solr/handler/component/combine/ReciprocalRankFusion.java

solr/core/src/java/org/apache/solr/search/combine/QueryAndResponseCombiner.java

solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java

solr/core/src/test/org/apache/solr/handler/component/CombinedQueryComponentTest.java

solr/core/src/java/org/apache/solr/search/combine/ReciprocalRankFusion.java

dsmiley

Really glad to see this work began by acknowledging the existing work and trying to address the pitfalls!

solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java

solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java

solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java

solr/core/src/java/org/apache/solr/search/combine/QueryAndResponseCombiner.java

alessandrobenedetti · 2025-07-10T09:54:16Z

Hi @ercsonusharma , thanks for resurrecting this, didn't have time to dedicate to the feature in the last few months, good to see some movement!

In the next couple of weeks, I should be able to give it a go and review it!

solr/core/src/java/org/apache/solr/handler/component/CombinedQuerySearchHandler.java

solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java

dsmiley

Thank you for the test; I like it!

I've been neglecting your PR (and of others that are non-trivial), with my attention on critical 10.0 backwards compatibility changes. But I should be getting to this...

dsmiley · 2025-12-05T05:07:34Z

solr/core/src/java/org/apache/solr/handler/component/combine/package-info.java

+ * This contains the classes to combine the scores from search index results. Multiple
+ * implementation of algorithms can be added to support them.
+ */
+package org.apache.solr.handler.component.combine;


I was thinking the CombinedQuery* classes would come here too, thus all classes associated with this are together. WDYT?

The original intention was to group the *Handler and *Component classes within the same package, as done previously.
It will affect this where we might deviate from usual convention of having solr.<ComponentName> & solr.<HandlerName>

dsmiley · 2025-12-05T15:16:20Z

BTW I'm busy tweaking some things and preparing a ~final state that's mergeable.
One thing I'm doing is reducing the forcedDistrib API as it's rather internal.
Also merging from main.
Adding changelog.
Updating javadocs; usually to simplify / remove boilerplate likely AI written stuff

Refactor SearchHandler to be more extensible and have a clearer flow. Improve BaseDistributedSearchTestCase.query to not set "distrib" on the control client request.

dsmiley · 2025-12-07T06:17:26Z

Please take a look Sonu. I also refactored SearchHandler a bit to have a clearer flow / readability and also extensibility. I'd like to merge the refactoring portions of this PR in one go, then the rest of this here. Like... I could extract all that to be under SOLR-17982.

A TODO I didn't get to tonight is testing the different forceDistrib API with a custom SearchHandler subclass. Both testForceDistrib methods are there but need to be modified. I suspect shards.info may reflect what we want to see.

solr/core/src/java/org/apache/solr/handler/component/CombinedQuerySearchHandler.java

ercsonusharma · 2025-12-07T12:45:33Z

Thanks for all the changes! So, the change would limit, having a new component only can't enable force-distributed behaviour on the existing SearchHandler.

I'd like to merge the refactoring portions of this PR in one go, then the rest of this here.

Sounds good to me.

I am still not sure if shardinfo reflects on the process method being called.

ercsonusharma · 2025-12-13T16:50:56Z

@dsmiley - Thanks for the PR! It looks good on my end. What do you think the next step should be, considering the other PR has been open for quite a few days now?

dsmiley · 2025-12-14T04:35:39Z

#3931 should merge first (say in ~2 days), then this one will be simpler and it's ready to go. I expect I should merge this one by the end of next week..

dsmiley

Sonu, can you please update this from main? It will simplify this PR a good deal. It should be ready for me to merge Friday, which I'm saying to give anyone else a moment to look before merging.

…ned_query

ercsonusharma · 2025-12-17T16:47:43Z

Sounds good, Thank you @dsmiley - done with update from main. I would appreciate it if you could trigger the workflows in the meantime.

ercsonusharma · 2025-12-18T06:24:48Z

Workflow failures seem likely to be infrastructure-related and a flaky SolrCloud test case. Re-triggering the workflow may help it pass.

ercsonusharma · 2025-12-19T14:33:27Z

@dsmiley - I see it failed again after re-triggering the same workflows. Luckily, I found a tiny cleanup to commit. Can you help approve the workflow?

After the latest run: all passed except "Solr Tests using Crave" which is struggling to find disk space.

New CombinedQuerySearchHandler etc. for implementing hybrid search with reciprocal rank fusion (RRF). See "JSON Combined Query DSL" in ref guide, and params prefixed with "combiner". QueryComponent: refactorings to enable a subclass to customize merging shard results. --------- Co-authored-by: Sonu Sharma <sonu_sharma2@apple.com> Co-authored-by: Christine Poerschke <cpoerschke@apache.org> Co-authored-by: David Smiley <dsmiley@apache.org>

New CombinedQuerySearchHandler etc. for implementing hybrid search with reciprocal rank fusion (RRF). See "JSON Combined Query DSL" in ref guide, and params prefixed with "combiner". QueryComponent: refactorings to enable a subclass to customize merging shard results. --------- Co-authored-by: Sonu Sharma <sonu_sharma2@apple.com> Co-authored-by: Christine Poerschke <cpoerschke@apache.org> Co-authored-by: David Smiley <dsmiley@apache.org> (cherry picked from commit 42cd889)

Sonu Sharma added 4 commits July 4, 2025 14:24

Combined Query Feature for Multi Query Execution

bf3cd5d

Tests: Combined Query Feature for Multi Query Execution

182bec9

Tests: Combined Query Feature for Multi Query Execution

b884f0e

Tests: Combined Query Feature for Multi Query Execution

29e8aea

github-actions bot added client:solrj tests cat:search module:clustering labels Jul 4, 2025

Improve: Fix typo

c113799

cpoerschke reviewed Jul 4, 2025

View reviewed changes

solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java Outdated Show resolved Hide resolved

cpoerschke reviewed Jul 4, 2025

View reviewed changes

solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java Outdated Show resolved Hide resolved

cpoerschke reviewed Jul 4, 2025

View reviewed changes

solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java Show resolved Hide resolved

ercsonusharma added 2 commits July 4, 2025 22:58

Tests: Fix errors

3600ed3

Review comments: implementation

9b0c76e

atris requested changes Jul 9, 2025

View reviewed changes

dsmiley reviewed Jul 9, 2025

View reviewed changes

ercsonusharma added 3 commits July 12, 2025 14:23

Code review changes

a841bc7

Code review changes

91f8e09

Code review changes

cace1f7

github-actions bot removed the module:clustering label Jul 12, 2025

ercsonusharma added 3 commits July 13, 2025 21:35

Code review changes

299db43

Code review changes

840070e

Improvement and fixes

d2feefc

ercsonusharma requested a review from atris July 16, 2025 18:45

cpoerschke reviewed Jul 25, 2025

View reviewed changes

solr/core/src/java/org/apache/solr/handler/component/CombinedQuerySearchHandler.java Outdated Show resolved Hide resolved

cpoerschke reviewed Jul 25, 2025

View reviewed changes

solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java Outdated Show resolved Hide resolved

cpoerschke reviewed Jul 25, 2025

View reviewed changes

solr/core/src/java/org/apache/solr/handler/component/CombinedQueryComponent.java Outdated Show resolved Hide resolved

dsmiley reviewed Dec 3, 2025

View reviewed changes

Improve forcedDistrib stuff

4e0b536

github-actions bot added the jetty-server label Dec 5, 2025

dsmiley reviewed Dec 5, 2025

View reviewed changes

dsmiley added 2 commits December 6, 2025 01:09

Merge branch 'main' into fork/ercsonusharma/feat_combined_query

6e6228b

Limit forceDistrib API surface area.

7604e94

Refactor SearchHandler to be more extensible and have a clearer flow. Improve BaseDistributedSearchTestCase.query to not set "distrib" on the control client request.

github-actions bot added the test-framework label Dec 7, 2025

ercsonusharma commented Dec 7, 2025

View reviewed changes

solr/core/src/java/org/apache/solr/handler/component/CombinedQuerySearchHandler.java Show resolved Hide resolved

ercsonusharma and others added 2 commits December 7, 2025 20:45

test fix for sort tie break

ed0ad98

test ForcedDistribSearchHandler

07ce63e

dsmiley mentioned this pull request Dec 9, 2025

SOLR-17982: setForcedDistrib & SearchHandler refactor #3931

Merged

dsmiley approved these changes Dec 17, 2025

View reviewed changes

dsmiley requested a review from atris December 17, 2025 14:33

Merge branch 'main' of https://github.com/apache/solr into feat_combi…

1c76210

…ned_query

github-actions bot removed test-framework jetty-server labels Dec 17, 2025

little clean up; fresh workflow trigger

c560cd5

update changelog

f7f48ff

dsmiley merged commit 42cd889 into apache:main Dec 19, 2025
2 checks passed

ercsonusharma deleted the feat_combined_query branch December 19, 2025 17:49

Conversation

ercsonusharma commented Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Solution

Tests

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ercsonusharma commented Jul 9, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dsmiley left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alessandrobenedetti commented Jul 10, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dsmiley left a comment

Choose a reason for hiding this comment

Uh oh!

dsmiley Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

ercsonusharma Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dsmiley commented Dec 5, 2025

Uh oh!

dsmiley commented Dec 7, 2025

Uh oh!

Uh oh!

ercsonusharma commented Dec 7, 2025

Uh oh!

ercsonusharma commented Dec 13, 2025

Uh oh!

dsmiley commented Dec 14, 2025

Uh oh!

dsmiley left a comment

Choose a reason for hiding this comment

Uh oh!

ercsonusharma commented Dec 17, 2025

Uh oh!

ercsonusharma commented Dec 18, 2025

Uh oh!

ercsonusharma commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ercsonusharma commented Jul 4, 2025 •

edited

Loading

ercsonusharma Dec 5, 2025 •

edited

Loading

ercsonusharma commented Dec 19, 2025 •

edited

Loading