Seamless Session Switching

[PotatoSpudowski]

Hello fastLLaMa community! 🎉

We are excited to announce the release of a new feature in our project - Quick Context Switching between sessions. This feature is aimed at improving the efficiency of instances serving multiple sessions while ensuring that each session's state is maintained with minimal latency.

In scenarios where a single running instance serves multiple sessions, it is crucial to maintain each session's state while minimizing latency. By implementing a mechanism to save and load session states quickly, we can enhance the efficiency of a single running instance, ultimately reducing resource usage and cost.

Here's what we are saving and loading to enable quick context switching:

• KV Cache Buffer
• Random number generator state
• Embed vector state
• Last n tokens state
• Logits state
• System prompt state
• Token embeddings state
• Norm state
• Output state

These states are saved and loaded using the save_state and load_state functions, respectively.

This feature brings several benefits, including:

1. Improved efficiency in multi-session instances
2. Reduced resource usage and costs
3. Enhanced maintainability of session states

Room for improvement:
We have noticed that the saved states are close to 600 MB (high due to KV cache buffer), which indicates that there might be some room for improvement. We encourage our fellow contributors to dive into the code, investigate this issue, and feel free to raise PRs with potential optimizations.

We hope that this new feature will make your experience with our project more seamless and efficient. We’d love to hear your thoughts, feedback, and suggestions on this feature! Feel free to leave a comment below, open an issue, or even submit a pull request. Your input helps us continue to improve and grow our project. 🚀

Thank you for your support, and happy coding! 🎉

[PotatoSpudowski]

Link to demo on twitter!

[PotatoSpudowski]

PR #52 by @amitsingh19975 shaved off 100 MB 👀🔥