How can I achieve streaming responses with Node-llama-cpp and Langchain?

[hafsalm]

How do I implement streaming responses in Langchain when integrating it with Node-llama-cpp? Are there specific features or configurations in Node-llama-cpp or Langchain that support and facilitate streaming behavior when used in conjunction?

Thank you in advance for your help and support!

[giladgd]

I'm not sure about the implementation details of the langjs module.
@nigel-daniels Can you help?

[nigel-daniels]

Currently the Langchain.js llama-cpp module based on node-llama-cpp does not have the model.stream('xxx') streaming method implemented in the LLM or Chat versions. Good news is, that is something I'm aiming to add over the next week as I have some travel time.

[hafsalm]

That’s fantastic news! It’s great to hear that you’re planning to add this feature.

[nigel-daniels]

@giladgd I have a question related to this, looking at the onToken option I can get access to the arriving tokens, is there a way to convert this to a ReadableStream or should I use something like the context.evaluate() then use the AsyncGenerator directly?

[nigel-daniels]

@giladgd np, I've run a few test, looks like the context.evaluate() will do the trick!

[nigel-daniels]

OK PR #3394 submitted on Langchain.JS to add streaming to the LLM and Chat modules, hopefully it will be in a release soon.

[hafsalm]

Awesome! 🌟 Really looking forward to its release! 🎉 Great job!

[giladgd]

Awesome :) Thanks @nigel-daniels

[nigel-daniels]

Note the first PR only supported single string submissions, I just updated it to handle multiple messages, examples will be in the updated docs. See PR# 3463.

[linonetwo]

I find onToken?: (tokens: Token[]) => void; returns an arryr of number, How to convert it back to a single string?

There used to be a https://withcatai.github.io/node-llama-cpp/api/classes/LlamaContext#decode , but I didn't find it in beta.

This is also a beta version feedback

[brandon-e2e]

The model has a detokenize function.

[linonetwo]

@brandon-e2e Thank you, I find it.