Rework config to be more native / allow customization

[owtaylor]

[ Comment moved from https://github.com//issues/119#issuecomment-2902226353 ]

Use cases

Reasons that a user might want to customize the models:

• Using a different Granite autocomplete model. Right now, there are multiple reasonable options:

  granite-3.3-8b-base: this is the best Granite model for autocomplete. If someone has a fast GPU, they probably want to use this.
  granite-3.3-2b-base: worse, but faster. Best model for mid-end Macs
  granite-3.3-8b-instruct: Quality is similar to 2b-base, speed similar to 8b-base. If you have a GPU with limited memory and only have room for one Granite, this can make sense.

  In the future, it's also possible there will be multiple reasonable choices for chat models.

• Using Ollama on a different port

• Using hosted models. If you have access to an instance of VLLM running granite3.3:8b, use that instead of Ollama

• Using old models - I don't care about this one. The Granite models have a track record for improving over time, and I don't want users torturing themselves trying to figure out whether granite3.2:8b is better than granite3.3:2b - because model outputs are inherently random (even at temperature 0), you just can't tell based on a small number of prompts. If someone really wants to investigate, they can always configure the models themselves. (See below)

• Using third party models. Not our emphasis, but users (or at least Granite.Code developers...) will want to compare.

Proposed way it looks

A basic principle should be that selecting models and customizing models feels like an extension of the upstream UI rather than something alien to it.

To change your autocomplete model, you edit ~/.granite-code/models/autocomplete.yaml to change

  name: Granite.Code chat model
  version: 1.0.0
  schema: v1
  models:
-    - uses: granite.code/autocomplete@default
+    - uses: granite.code/granite-3.3:8b-base

To use a different Ollama port, you edit ~/.granite-code/models/{autocomplete,chat,embed}.yaml and add:

  name: Granite.Code chat model
  version: 1.0.0
  schema: v1
  models:
    - uses: granite.code/autocomplete@default
    - override:
       apiBase: ollama.local:11434

(OR we add a setting for this, OR we simply honor the OLLAMA_HOST environment variable - but this would be the general mechanism for overrides)

To use a hosted model, you replace ~/.granite-code/models/{autocomplete,chat,embed}.yaml with your own content.

To stop using a hosted model, you delete those files, and they will be recreated with the default content.

Notes:

• This requires a pretty simple code change to provide our own RegistryClient when unrolling the yaml file
• An alternative would be to uses: ./default-models/granite-autocomplete.yaml which avoids the code change and lets people actually open that file and see what is in there. Might be better.
• When you are using a hosted model, you want to completely replace not override: to repoint, otherwise changes to the default model will cause user's configurations to break.

[Jazzcort]

To change your autocomplete model, you edit ~/.granite-code/models/autocomplete.yaml to change

  name: Granite.Code chat model
  version: 1.0.0
  schema: v1
  models:
-    - uses: granite.code/autocomplete@default
+    - uses: granite.code/granite-3.3:8b-base

Does this mean we're going to use Continue's hub system for our default config?

Saving the default config at ./default-models/granite-autocomplete.yaml sounds great as well. We can implement a checking logic that checks the default config files during the activating phase. If the default config files don't match ours, we just simply overwrite them. We can also update these files by releasing new version of Granite.Code.

[owtaylor]

Does this mean we're going to use Continue's hub system for our default config?

The idea would be that we'd patch in our own handling instead - the look up of "slugs" from the hub is encapsulated in the Registry object passed to unrollAssistant(), and could be replaced pretty easily. But I generally think using actual files is more user-friendly - it's easier to override things if you can see what you are overriding!

[Jazzcort]

Got it! Thanks for the clarification! I'm also more toward using actual files.

[owtaylor]

@fbricon can you review this - does it seem like a good plan?

[fbricon]

We could have an Ollama API Base field in the wizard for those wanting to install granite.code on machines that don't meet the minimum requirements. The Chat needs the wizard setup to be completed, before it can be used, and there's no use installing 6.5GB of models if the user needs to move to another machine, so this needs to be done in the wizard.

So if I understand correctly, after setup, granite.code generates:

• ~/.granite-code/default-models/granite-{autocomplete,chat,embed}.yaml
• ~/.granite-code/models/{autocomplete,chat,embed}.yaml that use the aforementioned files, adding potential apiBase overrides

if granite.code/autocomplete@default is resolved to a different model during unrolling, by all granite.code installations (imagine multiple versions are used by vscode, vscodium ...) then we should be good.

The one thing that bothers me is changing the ollama api base needs to be done in multiple files.

[owtaylor]

I'd keep the two ideas about how we want to proceed separate either we hook into the unrolling machinery, and a slug like granite/autocomplete magically resolves to the right content without touching the disk, or we write out a file and we reference it with a path.

The main tradeoffs are:

• In memory: a slightly more complicated implementation
• In memory: hard for a user to know what they can override
• In memory: does better for "fighting copies of granite.code"

I think I just don't care about fighting copies of granite.code so slightly prefer the on disk solution. But the other approach is also I think fundamentally workable.

My only idea how to handle "fighting copies" plus on-disk would be to stick a transformed version of vscode.env.appName onto the (https://code.visualstudio.com/api/references/vscode-api#env ) so default-models/visual-studio-code/ vs default-models/vscodium or something like that, but I don't love that much. Also doesn't handle different profiles of the same app name.

We could have an Ollama API Base field in the wizard for those wanting to install granite.code on machines that don't meet the minimum requirements. The Chat needs the wizard setup to be completed, before it can be used, and there's no use installing 6.5GB of models if the user needs to move to another machine, so this needs to be done in the wizard.

The one thing that bothers me is changing the ollama api base needs to be done in multiple files.

Yeah, the apiBase setting needs to be done upfront before the wizard, it's clunky to have it written into multiple files, and I think "ollama on a separate machine" is a bit of a corner case. That's why I was leaning toward just supporting that via the OLLAMA_HOST environment variable, and leaving this machinery for truly custom things (changing context lengths, etc) or for entirely replacing the model with a vllm hosted model, say.