Google Gemini Integration

[nimrod4278]

Adding the option to use Gemini as your LLM for gpt-pilot

[senko]

Thanks for the contribution @nimrod4278 🙏, this is awesome!

Trying GPT Pilot with various LLMs, we've found that other than GPT4, they mostly stumble on our prompts because we're using JSON schemas to define the expected output. How does Gemini fare in that regard?

It would be great if you could post a few screenshot or copy/paste a log showing it working (ie. making it as far as the development steps and then starting to create files/run commants, I'd count that as a success).

[nimrod4278]

Hi Senko, Thanks for the cool project. Gemini is doing well regarding JSON schemas. I did implement a retry mechanism for it to try and fix the json if it fails parsing it. I noticed that usually it will get it right when asked to fix the JSON file given. Attached are some screenshots of it developing a very simple next js app that managed to run. [image: unnamed.png] [image: unnamed.png][image: unnamed.png][image: unnamed.png][image: unnamed.png]

…

On Wed, Jan 10, 2024 at 8:06 PM Senko Rašić ***@***.***> wrote: Thanks for the contribution @nimrod4278 <https://github.com/nimrod4278> 🙏, this is awesome! Trying GPT Pilot with various LLMs, we've found that other than GPT4, they mostly stumble on our prompts because we're using JSON schemas to define the expected output. How does Gemini fare in that regard? It would be great if you could post a few screenshot or copy/paste a log showing it working (ie. making it as far as the development steps and then starting to create files/run commants, I'd count that as a success). — Reply to this email directly, view it on GitHub <#512 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHGN5PCPPHCUK2BSPHDWPVDYN3KC7AVCNFSM6AAAAABBUOVRY2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBVGM3DIMJQGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[DjWarmonger]

@nimrod4278 Screenshots sent by email didn't work.

I am particularly interested in this as we're yet to see Gemini Ultra, which is said to outperform GPT-4. Even if slightly, it it can make or break the app.

[robin-collins]

i believe that an LLM agnostic approach should be considered now that the new king of the hill for code is claude 3, and there are also some amazing opensource coding LLMs available.

[marcuslimdw]

What about a pluggable interface (ModelManager?) that could be something like this?

class ModelManager(metaclass=ABCMeta)

    @classmethod
    def build() -> Self:
        pass

    def generate_completion(data, request_type, project):  # TODO: add types
        pass

There could then be multiple implementations OpenAiModelManager, GeminiModelManager etc. that would each own configuration (loading API keys, etc. from environment variables) and formatting requests to and responses from their APIs, including any necessary retries - I too have experienced Gemini not returning things in the format you originally requested.

[Alignment-Lab-AI]

Thanks for the contribution @nimrod4278 🙏, this is awesome!

Trying GPT Pilot with various LLMs, we've found that other than GPT4, they mostly stumble on our prompts because we're using JSON schemas to define the expected output. How does Gemini fare in that regard?

It would be great if you could post a few screenshot or copy/paste a log showing it working (ie. making it as far as the development steps and then starting to create files/run commants, I'd count that as a success).

hello, significantly delayed response, but i would be interested in training a small open source model for this

[ssnaikr]

@nimrod4278
Have you used gemini 1.5 pro?
How were the results?

[ArtikUA]

@nimrod4278

senko in Discord said: "I can't say when the core team might do it (we have a bunch of stuff on our plate) but would love merge this if it gets into mergeable state"

Could you please update it to the state that it can be merged?