How can I achieve streaming responses with Node-llama-cpp and Langchain?

[hafsalm]

How do I implement streaming responses in Langchain when integrating it with Node-llama-cpp? Are there specific features or configurations in Node-llama-cpp or Langchain that support and facilitate streaming behavior when used in conjunction?

Thank you in advance for your help and support!

[giladgd]

I'm not sure about the implementation details of the langjs module.
@nigel-daniels Can you help?

[nigel-daniels]

Currently the Langchain.js llama-cpp module based on node-llama-cpp does not have the model.stream('xxx') streaming method implemented in the LLM or Chat versions. Good news is, that is something I'm aiming to add over the next week as I have some travel time.

[hafsalm]

That’s fantastic news! It’s great to hear that you’re planning to add this feature.

[nigel-daniels]

@giladgd I have a question related to this, looking at the onToken option I can get access to the arriving tokens, is there a way to convert this to a ReadableStream or should I use something like the context.evaluate() then use the AsyncGenerator directly?

[nigel-daniels]

@giladgd np, I've run a few test, looks like the context.evaluate() will do the trick!

[nigel-daniels]

OK PR #3394 submitted on Langchain.JS to add streaming to the LLM and Chat modules, hopefully it will be in a release soon.

[hafsalm]

Awesome! 🌟 Really looking forward to its release! 🎉 Great job!

[giladgd]

Awesome :) Thanks @nigel-daniels

[nigel-daniels]

Note the first PR only supported single string submissions, I just updated it to handle multiple messages, examples will be in the updated docs. See PR# 3463.

[giladgd]

@nigel-daniels I know I predicted version 3 would be stable a long time ago... well, it has taken me way longer than I anticipated since I had to compensate for a lot of missing stuff in llama.cpp with custom implementations to make it convenient to use, but it's finally done.
I'm currently working on updating the documentation for the stable release, so all changes from now on until the release will be mostly cosmetics for documentation purposes.

If you're still up for it, I think you can begin working on the updated Langchain integration, so version 3 can be used with Langchain right away when it's released.

Version 3 also includes a project generator based on pre-made templates (via npm create --yes node-llama-cpp@beta), so you can add a template that uses Langchain to make it easy to get started.
There's also a new LlamaChat class that allows you to complete a given chat in a way that I think would make the Langchain integration significantly easier to implement.

There are a lot more planned features for node-llama-cpp, but they'll land as non-breaking features.

[linonetwo]

I find onToken?: (tokens: Token[]) => void; returns an arryr of number, How to convert it back to a single string?

There used to be a https://withcatai.github.io/node-llama-cpp/api/classes/LlamaContext#decode , but I didn't find it in beta.

This is also a beta version feedback

[brandon-e2e]

The model has a detokenize function.

[linonetwo]

@brandon-e2e Thank you, I find it.