Update question router evaluation rake task to return the answer message #608
+11
−2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…o also return the answer message The evaluation:generate_question_routing_response task now returns the answer message in its output. This is needed for downstream evaluation of the question router behaviour.
…ned answer Adjusted the specs for evaluation:generate_question_routing_response to verify that the 'answer' field is returned correctly. 'answer' is nil for genuine_rag classification and includes the message for other classifications. This ensures tests match the updated task behavior.
kevindew
reviewed
Oct 14, 2025
spec/lib/tasks/evaluation_spec.rb
Outdated
| end | ||
| end | ||
|
|
||
| it "the answer message is null when classification is genuine_rag" do |
There was a problem hiding this comment.
I don't think you need this test as we're not actually testing a conditional in the code.
But a tip for future: For these RSpec tests use the it method to form a sentence. So for this it would probably be it "has a nil answer for a genuine_rag classification"
spec/lib/tasks/evaluation_spec.rb
Outdated
| it "outputs the response as JSON to stdout" do | ||
| ClimateControl.modify(INPUT: input) do | ||
| answer = build(:answer, question_routing_label: "genuine_rag", question_routing_confidence_score: 0.2) | ||
| answer = build(:answer, question_routing_label: "unclear_intent", question_routing_confidence_score: 0.2, message: "Sorry, can you say that again?") |
There was a problem hiding this comment.
Suggested change
| answer = build(:answer, question_routing_label: "unclear_intent", question_routing_confidence_score: 0.2, message: "Sorry, can you say that again?") | |
| answer = build(:answer, | |
| question_routing_label: "unclear_intent", | |
| question_routing_confidence_score: 0.2, | |
| message: "Sorry, can you say that again?") |
Just because this line has got a little long
spec/lib/tasks/evaluation_spec.rb
Outdated
Comment on lines
332
to
333
| expect { Rake::Task[task_name].invoke("openai") } | ||
| .to output("{\"classification\":\"genuine_rag\",\"confidence_score\":0.2}\n").to_stdout | ||
| .to output("{\"classification\":\"unclear_intent\",\"confidence_score\":0.2,\"answer\":\"Sorry, can you say that again?\"}\n").to_stdout |
There was a problem hiding this comment.
Suggested change
| expect { Rake::Task[task_name].invoke("openai") } | |
| .to output("{\"classification\":\"genuine_rag\",\"confidence_score\":0.2}\n").to_stdout | |
| .to output("{\"classification\":\"unclear_intent\",\"confidence_score\":0.2,\"answer\":\"Sorry, can you say that again?\"}\n").to_stdout | |
| expected_output = { | |
| classification: "unclear_intent", | |
| confidence_score: 0.2, | |
| answer: "Sorry, can you say that again?", | |
| } | |
| expect { Rake::Task[task_name].invoke("openai") } | |
| .to output("#{expected_output.to_json}\n").to_stdout |
Sorry this suggestion hasn't quite worked out properly because you only changed one line, but I think the line had got a bit long so it's best to pull the data out and convert to json
…onse Remove unnecessary test and shorten long lines following reviewer's comments
|
Thanks @kevindew. I think I have implemented the changes you suggested. |
kevindew
approved these changes
Oct 14, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is so that the downstream evaluation pipeline works as expected as for alphagov/govuk-chat-evaluation#73