Skip to content

Issues: aws-samples/foundation-model-benchmarking-tool

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
36 Open 30 Closed
36 Open 30 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Add Autoscaling functionality to FMBench enhancement New feature or request
#10 opened Feb 6, 2024 by madhurprash
Code cleanup needed to replace the notebooks with regular python files help wanted Extra attention is needed
#20 opened Feb 15, 2024 by aarora79
@aarora79
Add documentation for the config file on readthedocs.io good first issue Good for newcomers
#21 opened Feb 15, 2024 by aarora79
@4mp
Add support for models hosted on platforms other than SageMaker enhancement New feature or request
#22 opened Feb 15, 2024 by aarora79
The README gives an impression that this tool is comparing instance types when it should be serving stacks documentation Improvements or additions to documentation
#25 opened Feb 15, 2024 by aarora79
@aarora79
The business summary plot in the report needs to have a caption for disclaimer bug Something isn't working
#26 opened Feb 15, 2024 by aarora79
Allow different experiments in the same config file to use different regions good first issue Good for newcomers
#27 opened Feb 15, 2024 by aarora79
@4mp
Provide config file for FLAN-T5 out of the box enhancement New feature or request
#28 opened Feb 15, 2024 by aarora79
@madhurprash
Try FMBench with instance count set to > 1 to see how scaling impacts latency and transactions per minute enhancement New feature or request
#29 opened Feb 17, 2024 by aarora79
Compare different models for the same dataset enhancement New feature or request
#30 opened Feb 17, 2024 by aarora79
@aarora79
Need for Integration of FMBench Tool with External 3rd party Synthetic Data Generation Tools for Benchmarking usecase.
#32 opened Mar 16, 2024 by akramIOT
FMBench to support benchmarking for embedding models enhancement New feature or request
#33 opened Mar 25, 2024 by madhurprash
@madhurprash
Reports only show information per instance instead of per endpoint
#114 opened Jun 7, 2024 by jimburtoft
Remove the config_filepath.txt as a config reference for developer workflow
#150 opened Jul 18, 2024 by madhurprash
@madhurprash
TypeError: unsupported format string passed to NoneType.__format__
#154 opened Jul 24, 2024 by prasad-nair-amd
UI Support for FMBench
#155 opened Jul 31, 2024 by madhurprash
@madhurprash
Add retry handling for throttling errors during inference
#157 opened Aug 1, 2024 by madhurprash
add hallucination metric to model evals on FMBench
#170 opened Aug 14, 2024 by madhurprash
Add support for non AWS models to be benchmarked
#203 opened Sep 28, 2024 by madhurprash
1
@madhurprash
miss warm up phase
#213 opened Oct 8, 2024 by lxning
Allow backup/ failover regions to be specified
#232 opened Nov 7, 2024 by jimburtoft
Fails when testing Haiku on AWS Bedrock
#247 opened Nov 28, 2024 by NidPlays
parameterize configuration files
#261 opened Jan 8, 2025 by madhurprash
@madhurprash
Experiment config file's model name doesn't match with pricing_fallback.yml file's model name
#267 opened Jan 14, 2025 by Yiwen-Zhang
The pricing fetching logic only works for EC2
#268 opened Jan 14, 2025 by Yiwen-Zhang
ProTip! What’s not been updated in a month: updated:<2025-02-09.