AISI Inspect integration #6
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…use. Fixed embedding dimensionality mismatch by specifying embed model name.
…ion is not yet ready.
…xt in solver. Revised nan handling in reducer.
…ics selected in yaml config.
…ue from Inspect plan.
xeon27
approved these changes
Jul 5, 2024
| @@ -102,3 +111,70 @@ def get_cfg(self): | |||
| if self._cfg is None: | |||
| raise ValueError("System config not set.") | |||
| return self._cfg.as_dict() | |||
|
|
|||
| def get_inspect_tool( | |||
There was a problem hiding this comment.
What's the purpose of implementing both the get_inspect_tool and the get_inspect_solver. Do we need both, or just one of these is sufficient?
| async def solve(state: TaskState, generate: Generate) -> TaskState: | ||
| query = state.user_prompt.text | ||
| async with concurrency("document_search", max_concurrency): | ||
| response = self.invoke(query, documents) |
There was a problem hiding this comment.
The response here consists of both the retrieved context and the generated answer, why are we passing it again through the chain_of_thought(), generate() and self_critique() pipeline?
veval/metrics/template.py
Outdated
| metric_function=inspect_metric_fn, | ||
| )() | ||
| for ragas_feature_name in RAGAS_FEATURE_NAMES |
There was a problem hiding this comment.
The current implementation calculates all available metrics irrespective of task. Modify this to restrict to only those metrics which are specified in the task config (yaml file).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR Type
Feature
Short Description
Integrate AISI Inspect for RAG and for RAGAS Evaluation.
To run these examples, install inspect-ai and run: