Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
.Net: Fix default value for max_tokens (#2743)
### Motivation and Context ####⚠️ Breaking change. When `Kernel.InvokeSemanticFunctionAsync` is used, `max_tokens` is set to `256` by default. However if you call `Kernel.CreateSemanticFunction`, `max_tokens` is set to `null` by default so the model default value is used. The reason we are setting `max_tokens` to 256 is because OpenAI text completion models have a default of 16. However OpenAI chat completion models have a default of infinity. So the current default makes sense for text completions but not for chat completions. To fix this, the default value of `max_tokens` will always be set to `null`. #### Troubleshooting Enable informational logging and check for token usage logs e.g., ``` Action: GetCompletionsAsync. Azure OpenAI Deployment Name: text-davinci-003. Prompt tokens: 14. Completion tokens: 16. Total tokens: 30. ``` #### Here's the documentation for OpenAI: https://platform.openai.com/docs/api-reference/completions/create#max_tokens `max_tokens` `integer or null` `Optional` `Defaults to 16` The maximum number of [tokens](https://platform.openai.com/tokenizer) to generate in the completion. https://platform.openai.com/docs/api-reference/chat/create#max_tokens `max_tokens` `integer or null` `Optional` `Defaults to inf` The total length of input tokens and generated tokens is limited by the model's context length. Resolves: #2738 ### Description The property `max_tokens` is an optional so we should not set a default value in SK. This change makes SK consistent in this approach. ### Contribution Checklist - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄
- Loading branch information