Improve query rewrite for search context #7164
Open
+1,054
−43
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When gathering the initial search context for a repo, we use the 'fast' LLM to create a keyword search from the user's question. In evals, we found that this query rewrite made things worse for simple questions, so we only enabled it for longer or non-English questions.. I recently dug into this, and figured out this was because the rewrite prompt was not performing well.
This PR improves the rewrite prompt so we can remove the special-casing around short English questions. Benefits:
1556.10ms
677.25ms
Downside: single-sentence Chat questions with an @-repo mention become a bit slower, as they now rewrite the query.
FAQ: why are you working on this? We're reworking all of Chat!
This didn't take me much time at all. Also I need this to unblock my PR to move to Cody context to NLS search under-the-hood and eliminate the confusing duplication.
Test plan
Updated unit tests. I reran the "Tool Call" evals locally and checked that there is no regression. I was actually hoping for an improvement, but even without one this feels worth it.