[Infra] Add support for an LLM service in LocalNet/DevNet infra

[Olshansk]

Objective

Increase & diversify the types of services supported in the LocalNet / DevNet infrastructure.

Origin Document

The Shannon upgrade has several goals, one of which is to support any type (i.e. general-purpose) RPC service.

Goals

• Enable other types of RPCs in our network in the early development/testing stages
• Cater to the AI market in addition to the Web3 market

Deliverables

• Identify a small, cheap and light-weight LLM model that can be part of our LocalNet / DevNet infrastructure (e.g. Llama 2 - 7B)
• Deploy the LLM service in a separate node, similar to anvil, with our infrastructure
• Make sure that model inference can be triggered via a JSON RPC call directly to the supplier node

Non-goals / Non-deliverables

• Guaranteeing correctness or efficiency of the LLM models
• Optimizing the model service's performance

General deliverables

• Comments: Add/update TODOs and comments alongside the source code so it is easier to follow.
• Testing: Add new tests (unit and/or E2E) to the test suite.
• Makefile: Add new targets to the Makefile to make the new functionality easier to use.
• Documentation: Update architectural or development READMEs; use mermaid diagrams where appropriate.

---

Creator: @Olshansk
Co-Owners: @okdas

[Olshansk]

@okdas I haven't attached an iteration to this since it's not urgent, but feel free to pick it up when you have time :)

[okdas]

Had some progress on this. The plan is to:

1. Use a small model like neural-chat-7b-v3-1.Q4_K_M.gguf.
2. Sync model into the container via https://github.com/tilt-dev/tilt-extensions/tree/master/file_sync_only.
3. Disable this by default, as the file is still kind of large and the process eats a decent amount of ram.
4. llama.cpp server.
5. Provide a curl example.
6. Optional: Add configuration for the endpoint in relayminer.
7. Optional: Add en e2e test.

^ this should probably be moved to the deliverables.. :)

[Olshansk]

^ this should probably be moved to the deliverables.. :)

What's stopping you?

[okdas]

Checkout current limitations - something we need to address: #508 (comment)

[Olshansk]

The goal here was to add support for LLMs in our infra. It did not account for SDK changes and REST support, so I'm classifying this as done.

cc @okdas @red-0ne