Is llama.cpp designed to be consumed via CLI or C programs?

[segmond]

The reason I ask is because, I often see someone ask a question and they are sent to a sample C implementation instead of being given a command line command they can use to achieve their goals.

[ggerganov]

llama.cpp is designed to be used through the llama library produced by this project. The C-style interface is available in llama.h. The examples folder provides examples of using the library

[Florents-Tselai]

@ggerganov The thing is that although llama.h is C-friendly, common.h which is most times necessary, it's not.

[ggerganov]

This has been acknowledged and there is a plan to implement a higher-level API: #5215

common contains functionality shared by the llama.cpp examples. Some of it eventually becomes part of the llama library when it matures enough (#5214)

The way I see this is that having a low-level API in llama.h allows for better control, but is obviously not very user friendly. It's easier to make a low-level API, especially in the beginning when it wasn't clear what are the common use cases of the library. With time, we have better understanding of those use cases and we can support these with some abstractions. But it will always be useful to have the low-level control for more exotic applications

[bullno1]

Personally, I use llama.h in C for inference while implementing the rest myself: https://github.com/bullno1/hey.
I have my own run loop, pluggable token filter (ala grammar, guidance...), custom watcher and sampler...

I can do things like invoking a calculator in the middle of a GSM8k generation: https://github.com/bullno1/hey/blob/master/examples/calculator.c

But it will always be useful to have the low-level control for more exotic applications

Thanks for that. This is why it's the best LLM API.