Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve query rewrite for search context #7164

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

jtibshirani
Copy link
Member

@jtibshirani jtibshirani commented Feb 20, 2025

When gathering the initial search context for a repo, we use the 'fast' LLM to create a keyword search from the user's question. In evals, we found that this query rewrite made things worse for simple questions, so we only enabled it for longer or non-English questions.. I recently dug into this, and figured out this was because the rewrite prompt was not performing well.

This PR improves the rewrite prompt so we can remove the special-casing around short English questions. Benefits:

  • Removes a "gotcha" that was making the mental model more complicated.
  • It will let us switch to NLS search instead to power context search, which always expects keyword-style queries.
  • The improved query rewrite is significantly faster. Here is the average rewrite time during the "Tool Call" eval:
    • Before: 1556.10ms
    • After: 677.25ms

Downside: single-sentence Chat questions with an @-repo mention become a bit slower, as they now rewrite the query.

FAQ: why are you working on this? We're reworking all of Chat!
This didn't take me much time at all. Also I need this to unblock my PR to move to Cody context to NLS search under-the-hood and eliminate the confusing duplication.

Test plan

Updated unit tests. I reran the "Tool Call" evals locally and checked that there is no regression. I was actually hoping for an improvement, but even without one this feels worth it.

Screenshot 2025-02-20 at 11 35 42 AM

@jtibshirani jtibshirani requested review from a team, abeatrix and janhartman February 20, 2025 19:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant