Skip to content

Add raw completions server mode#61

Open
Dango233 wants to merge 1 commit intoantirez:mainfrom
Dango233:codex/raw-completions
Open

Add raw completions server mode#61
Dango233 wants to merge 1 commit intoantirez:mainfrom
Dango233:codex/raw-completions

Conversation

@Dango233
Copy link
Copy Markdown

Summary

This adds an opt-in --raw-completions mode for ds4-server.

When enabled, /v1/completions treats the request prompt as an already-rendered model prompt and continues it directly, instead of wrapping it in the built-in DS4 chat template. The default behavior is unchanged.

Why

This aligns ds4's opt-in completions behavior with llama.cpp-style raw completions, where the client owns prompt rendering and the server only performs continuation.

That is useful for clients and experiments that provide their own model-specific templates, including text-completion/instruct frontends such as SillyTavern, KoboldAI/KoboldCpp-style clients, and custom completion clients that send rendered prompts directly.

Notes

  • POST /v1/chat/completions is unchanged.
  • Default /v1/completions behavior is unchanged unless --raw-completions is passed.
  • In raw mode, thinking markers or other model control tokens should be included by the client as part of the rendered prompt.

Testing

  • make ds4-server ds4_test
  • ./ds4_test --server
  • ./ds4-server --help | rg -n "raw-completions|completions"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant