Requesting support for IBM's OpenSource Granite models

[q5sys]

These open source models were just released yesterday at Red Hat Summit.
https://huggingface.co/ibm-granite
https://arxiv.org/abs/2405.04324

If this ends up being a bigger ask than I think it is, and there's something I can do to help in making this happen, let me know.

[danielhanchen]

Oh interesting!

[junzzhu]

Fine tuning for both ibm-granite/granite-3b-code-instruct and ibm-granite/granite-8b-code-base is working now as far as I checked with Llama3 Colab notebook, with training loss decreasing as expected. However, inference outputs are both useless still.

Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
Continue the fibonnaci sequence.

### Input:
1, 1, 2, 3, 5, 8

### Response:
1#<fim_prefix>A
# str
 growth
 for
 for
 for
 for





  `



  `
 `
 
 
           
 ` ` ` ` ` ` ` ` ` `                                                           9\ `<fim_prefix><fim_prefix><fim_prefix><fim_prefix>

[q5sys]

I noticed the other day when I was attempting to quantize the 34B larger models that the Granite models are 2 different types. The 3B,7B, and 8B models are llama, while the 20B and 34B are gpt-bigcode models. Not sure how that would or wouldn't affect fine tuning since i haven't looked into it yet, but I figured it was worth mentioning.

[danielhanchen]

@q5sys So if its other model types, then well error out for now.

@junzzhu Oh wait its a Code model, so finetuning on text might not work as expected. Hence the weird output

[junzzhu]

Oh wait its a Code model, so finetuning on text might not work as expected. Hence the weird output

Cool! That helps. With 7b-base model, the output is meaningful now. Thanks @danielhanchen

[danielhanchen]

Great it worked!!