Skip to content

Issues: UKGovernmentBEIS/inspect_ai

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

32 Open 190 Closed
32 Open 190 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Inspect sometimes hangs on extremely large runs bug Something isn't working
#849 opened Nov 15, 2024 by MSchmatzAISI
@MSchmatzAISI
normalize_number fails on exponent / fractional input
#848 opened Nov 14, 2024 by evanmiller-anthropic
Multi-processing w/ s3 logs, unless you use spawn to launch your subprocesses
#840 opened Nov 13, 2024 by max-kaufmann
match scorer doesn't handle answers with percent sign
#838 opened Nov 13, 2024 by fastfedora
1
Inspect process uses large amounts of memory w/ DockerSandbox, likely because large exec() outputs are loaded into memory
#836 opened Nov 12, 2024 by max-kaufmann
Validation error when reading Eval log
#834 opened Nov 12, 2024 by us2547
@dragonstyle
5
Question about running code on eval results
#828 opened Nov 10, 2024 by AarushSah
5
Log a "Sample error detected" if a sample has an error, so that we can see it in the console when we are running an eval
#824 opened Nov 8, 2024 by max-kaufmann
For certain Model API calls, if they error we don't have the API call logged to the transcript.
#823 opened Nov 8, 2024 by max-kaufmann
1
A scorer to compute costs based on model usages (for eval model, grader model, etc.)?
#795 opened Nov 3, 2024 by grigory93
[Question] Custom metric type
#775 opened Oct 30, 2024 by us2547
2
@dragonstyle
6
Adding a scorer without rerunning existing scorers.
#770 opened Oct 27, 2024 by sohaibimran7
2
Failed epoch doesn't stop rest of task's epochs
#746 opened Oct 23, 2024 by craigwalton-dsit
Web browser struggles with large pages web browser
#740 opened Oct 21, 2024 by art-dsit
1
plot_results(): Are there any frameworks that allow summarising and visualising inspect logs?
#704 opened Oct 16, 2024 by sohaibimran7
2
support for web browser navigations that open new windows web browser
#681 opened Oct 10, 2024 by jjallaire-aisi
@MariaIzobava
Error running inspect score with a scoring function that uses a task-level sandbox
#679 opened Oct 10, 2024 by bienehito
[Feature Request] Support live statistics in progress bar
#674 opened Oct 9, 2024 by tw-aisi
1
[Feature request] command line flag to capture the values of locals in the printed traceback
#673 opened Oct 9, 2024 by JasonGross
1
.json file is missing all messages when an error is thrown
#672 opened Oct 9, 2024 by JasonGross
1
Web Browser tool cannot see checkbox state web browser
#636 opened Oct 3, 2024 by sdtblckgov
@MariaIzobava
Feature Request: Support Anthropic on Vertex models.
#574 opened Sep 30, 2024 by simonbutt
1
[feature request] live/realtime inspect view
#559 opened Sep 28, 2024 by JasonGross
1
[feature request] Support the -F flag for inspect eval
#551 opened Sep 27, 2024 by tw-aisi
1
[feature request] support passing -H / --host to docker sandbox
#540 opened Sep 27, 2024 by JasonGross
1
ProTip! Mix and match filters to narrow down what you’re looking for.