Skip to content

Issues: aws-samples/foundation-model-benchmarking-tool

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
36 Open 30 Closed
36 Open 30 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Loosen dependencies for more flexible installation
#105 opened May 31, 2024 by athewsey
@aarora79
1
Add Autoscaling functionality to FMBench enhancement New feature or request
#10 opened Feb 6, 2024 by madhurprash
Code cleanup needed to replace the notebooks with regular python files help wanted Extra attention is needed
#20 opened Feb 15, 2024 by aarora79
@aarora79
Add documentation for the config file on readthedocs.io good first issue Good for newcomers
#21 opened Feb 15, 2024 by aarora79
@4mp
Add support for models hosted on platforms other than SageMaker enhancement New feature or request
#22 opened Feb 15, 2024 by aarora79
The README gives an impression that this tool is comparing instance types when it should be serving stacks documentation Improvements or additions to documentation
#25 opened Feb 15, 2024 by aarora79
@aarora79
The business summary plot in the report needs to have a caption for disclaimer bug Something isn't working
#26 opened Feb 15, 2024 by aarora79
Allow different experiments in the same config file to use different regions good first issue Good for newcomers
#27 opened Feb 15, 2024 by aarora79
@4mp
Provide config file for FLAN-T5 out of the box enhancement New feature or request
#28 opened Feb 15, 2024 by aarora79
@madhurprash
Try FMBench with instance count set to > 1 to see how scaling impacts latency and transactions per minute enhancement New feature or request
#29 opened Feb 17, 2024 by aarora79
Compare different models for the same dataset enhancement New feature or request
#30 opened Feb 17, 2024 by aarora79
@aarora79
Need for Integration of FMBench Tool with External 3rd party Synthetic Data Generation Tools for Benchmarking usecase.
#32 opened Mar 16, 2024 by akramIOT
Add support for different payload formats for bring your own datasets for that might be needed for different inference containers enhancement New feature or request
#5 opened Jan 26, 2024 by madhurprash
@madhurprash
1
FMBench to support benchmarking for embedding models enhancement New feature or request
#33 opened Mar 25, 2024 by madhurprash
@madhurprash
Reports only show information per instance instead of per endpoint
#114 opened Jun 7, 2024 by jimburtoft
Is it possible to run FMBench on an EC2 instance on endpoints currently not supported on Sagemaker.
#120 opened Jun 14, 2024 by prasad-nair-amd
@aarora79
5
Remove the config_filepath.txt as a config reference for developer workflow
#150 opened Jul 18, 2024 by madhurprash
@madhurprash
Unable to locate credential issue
#153 opened Jul 23, 2024 by prasadwrites
1
TypeError: unsupported format string passed to NoneType.__format__
#154 opened Jul 24, 2024 by prasad-nair-amd
UI Support for FMBench
#155 opened Jul 31, 2024 by madhurprash
@madhurprash
Add retry handling for throttling errors during inference
#157 opened Aug 1, 2024 by madhurprash
add hallucination metric to model evals on FMBench
#170 opened Aug 14, 2024 by madhurprash
Add support for non AWS models to be benchmarked
#203 opened Sep 28, 2024 by madhurprash
1
@madhurprash
miss warm up phase
#213 opened Oct 8, 2024 by lxning
Support speculative decoding
#289 opened Mar 4, 2025 by myang-tech42
ProTip! Follow long discussions with comments:>50.