Using LocalAI embedding, allow for control of batch size with models.

[knot-split]

Describe the feature you'd like to request

Perhaps an environment variable that either sets a batch size for the embedding or else defaults to the model's config for batch size.

Describe the solution you'd like

We have tried setting a dozen different places for adjusting the physical_batch_size in various configs, either in the ccb data or else in localai or even in our model yaml files, but context chat always seems to hit the same limit no matter what we do: batch size: 512
Describe alternatives you've considered

Perhaps not using localai embedding and instead the built-in embedding from ccb doesn't have this problem, but I'm not just yet ready to try it, since it took a month for me to get the ccb+local ai on one machine and nc aio on the other.

[kyteinsky]

hello,
this should be solved in #276

adjusting the physical_batch_size in various configs, either in the ccb data or else in localai or even in our model yaml files, but context chat always seems to hit the same limit no matter what we do: batch size: 512

it would be helpful to see the error but most probably the PR linked above would solve this issue. The batch size in ccb config would not work since it configures the local embedding process, not the remote one.

[knot-split]

context_chat_backend.types.EmbeddingException: {"error":{"code":500,"message":"rpc error: code = Internal desc = input (514 tokens) is too large to process. increase the physical batch size (current batch size: 512)","type":""}}
2026-03-05 15:18:37

Okay, so this is an error I see repeatedly in the CCB logs. Docker desktop suggested--through Gordon--a number of different places to change this current batch size, but to no avail. The branch #276 shows an environment variable of CC_EM_BATCH_SIZE. I will try to set this and see if it changes anything...but is that the one you were doubting?

[kyteinsky]

CC_EM_BATCH_SIZE

that is the batch size of embedding requests submitted at a time so N chunks in one embedding request. This case seems to point to the size of the individual chunk of message being sized more than expected.

maybe it can be fixed by setting "truncate" to true in the embedding provider, if it's available,
or setting embedding_chunk_size: 1800 or similar value in the /nc_app_context_chat_backend/config.yaml file, reduced from 2000. It refers to the max no. of characters in one chunk of the embeddable text.