Skip to content

Commit

Permalink
.Net: Fix default value for max_tokens (#2743)
Browse files Browse the repository at this point in the history
### Motivation and Context

#### ⚠️ Breaking change.

When `Kernel.InvokeSemanticFunctionAsync` is used, `max_tokens` is set
to `256` by default. However if you call
`Kernel.CreateSemanticFunction`, `max_tokens` is set to `null` by
default so the model default value is used.

The reason we are setting `max_tokens` to 256 is because OpenAI text
completion models have a default of 16. However OpenAI chat completion
models have a default of infinity. So the current default makes sense
for text completions but not for chat completions.

To fix this, the default value of `max_tokens` will always be set to
`null`.

#### Troubleshooting

Enable informational logging and check for token usage logs e.g.,

```
    Action: GetCompletionsAsync. Azure OpenAI Deployment Name: text-davinci-003.
    Prompt tokens: 14. Completion tokens: 16. Total tokens: 30.
```

#### Here's the documentation for OpenAI: 


https://platform.openai.com/docs/api-reference/completions/create#max_tokens
  `max_tokens` `integer or null` `Optional` `Defaults to 16`
The maximum number of [tokens](https://platform.openai.com/tokenizer) to
generate in the completion.

  https://platform.openai.com/docs/api-reference/chat/create#max_tokens
  `max_tokens` `integer or null` `Optional` `Defaults to inf`
The total length of input tokens and generated tokens is limited by the
model's context length.

Resolves: #2738

### Description

The property `max_tokens` is an optional so we should not set a default
value in SK. This change makes SK consistent in this approach.

### Contribution Checklist

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
  • Loading branch information
markwallace-microsoft authored Sep 16, 2023
1 parent ee68b65 commit 8ea8573
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ public static Task<SKContext> InvokeSemanticFunctionAsync(
string? functionName = null,
string? skillName = null,
string? description = null,
int maxTokens = 256,
int? maxTokens = null,
double temperature = 0,
double topP = 0,
double presencePenalty = 0,
Expand Down

0 comments on commit 8ea8573

Please sign in to comment.