Discussion 595 cost cents default

[stephan-buckmaster]

This change adds defaults for language_models.input_token_cost_cents and output_token_cost_cents, and also adds those fields to the UI for Language Models.

[krschacht]

Good catch about the UI not allowing save!

One thing I was undecided on was if we should actually support nil. My preliminary thinking was: $0 means the tokens are free, which is appropriate for local models, and should be reflected in the interface when it shows you the cost of a conversation as $0. But nil means the token cost is unknown so this too could be reflected in the interface. In this case it shouldn’t show $0, instead it should show no information or maybe we have it explicitly show “unknown.”

[stephan-buckmaster]

That is a interesting twist! Would it be solved by a adding a flag, "supports cost per token" Tough choice

…

On Sun, Jan 26, 2025, 3:26 PM Keith Schacht ***@***.***> wrote: Good catch about the UI not allowing save! One thing I was undecided on was if we should actually support nil. My preliminary thinking was: $0 means the tokens are free, which is appropriate for local models, and should be reflected in the interface when it shows you the cost of a conversation as $0. But nil means the token cost is unknown so this too could be reflected in the interface. In this case it shouldn’t show $0, instead it should show no information or maybe we have it explicitly show “unknown.” — Reply to this email directly, view it on GitHub <#607 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACAV3TKW6ASAL5W5SA4XAD2MVVKNAVCNFSM6AAAAABV42WFYKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJUGYZTEMBWGM> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[krschacht]

Rather than adding another state tracker, I think it's reasonable for us to just use "nil" to indicate that state. Because it may be that this LLM doesn't support cost tracking, or maybe it does but someone just hasn't bothered to put the $ figure into the settings yet. We can actually be agnostic as to which of those situations we're in, we can just assume "nil" means we won't do any cost math — we won't assume $0.

[stephan-buckmaster]

It looks like my pull-request might need some more adjustments. I'm open to any further feedback or suggestions regarding the UI/behavior — please feel free to describe them in more depth as you see fit.

…

Message ID: ***@***.***>

[krschacht]

What if we just keep: (a) adding the fields to the form, (b) updating the permit to include these fields — but remove the migration and remove the validation. So basically, we'll just revert to the short-circuit (return if assistant.language_model.output_token_cost_cents.blank?)

I think that might cover us. I didn't test this to be certain, but when those token costs are nil does the app just gracefully handle it? Hopefully no costs appear anywhere for that language model.