whisper : improve handling of prompts

[ggerganov]

fix #1960, #1961, #1962

Changed whisper_tokenize() to return the negative number of required tokens when the token buffer is not big enough. This can be used to determine the necessary tokens for a given text:

int n_needed = -whisper_tokenize(ctx, "some text", NULL, 0);

Also improve docs by clarifying that the Whisper models can process prompts of up to only n_text_ctx/2 tokens which is 224 tokens. So there is no point to provide longer prompts as they would be truncated:

whisper.cpp/whisper.cpp

Lines 5474 to 5480 in 48a1452

	// if we have already generated some text, use it as a prompt to condition the next generation
	if (!prompt_past.empty() && t_cur < 0.5f && params.n_max_text_ctx > 0) {
	int n_take = std::min(std::min(params.n_max_text_ctx, whisper_n_text_ctx(ctx)/2), int(prompt_past.size()));
	prompt = { whisper_token_prev(ctx) };
	prompt.insert(prompt.begin() + 1, prompt_past.end() - n_take, prompt_past.end());
	}

[ggerganov]

@sindresorhus lmk if this change would work for you

[sindresorhus]

Thanks for looking into this. That would work, although I still think wrapping it up into a separate function would make for a better API and make it more discoverable for consumers. I would never have thought of using whisper_tokenize like that.

// Some docs
int whisper_token_count(struct whisper_context * ctx, const char * text) {
   return -whisper_tokenize(ctx, text, NULL, 0);
}

[sindresorhus]

fix #1960, #1961, #1962

By the way, you need to make that:

fix #1960, fix #1961, fix #1962

For GitHub to do the right thing and close them all when this PR is merged.

[sindresorhus]

I would document the 224 number here for quick reference.

Same in

whisper.cpp/whisper.h

Line 507 in 5c2c07d

// maximum of whisper_n_text_ctx()/2 tokens are used