Skip to content

Add support for locally hosted LLM providers (e.g., Ollama)#197

Draft
tomvothecoder wants to merge 8 commits into
E3SM-Project:dev-aifrom
tomvothecoder:ai/196-local-llm
Draft

Add support for locally hosted LLM providers (e.g., Ollama)#197
tomvothecoder wants to merge 8 commits into
E3SM-Project:dev-aifrom
tomvothecoder:ai/196-local-llm

Conversation

@tomvothecoder
Copy link
Copy Markdown
Collaborator

@tomvothecoder tomvothecoder commented May 20, 2026

Description

Add support for locally hosted LLM providers on top of dev-ai, starting with Ollama, so simulation summaries can run against internal or self-hosted inference endpoints.

This PR extends the assistant provider/configuration flow to support endpoint and model selection for local deployments, updates request handling and orchestration for local providers, adds backend test coverage for config and provider behavior, and documents local plus NERSC Spin setup.

It also narrows the assistant provider/configuration surface in the backend, keeps the internal OpenAI-compatible transport path needed by Ollama and LivAI, updates the frontend summary contract, removes unsupported environment variables from active templates, cleans up active setup/deployment docs, and refreshes backend dependency locking to drop Anthropic-specific dependency support.

Follow-up commits on this branch also harden LLM summary fallback behavior for local models by repairing missing follow-up fields and canonicalizing common shorthand citation paths, and improve the simulation summary UX by showing live generation runtime plus clearer deterministic-fallback messaging.

It also includes the implementation plan commit already present on this branch.

What Changed

Closes #196

  • limit ASSISTANT_LLM_PROVIDER to ollama and livai
  • default assistant provider to ollama
  • remove OpenAI and Anthropic config branches, env vars, and fallback reasons
  • keep Ollama/LivAI transport on the OpenAI-compatible client path
  • narrow generation_provider contract in frontend/backend types
  • update active env templates and operator/developer docs
  • refresh backend tests and uv.lock
  • harden assistant summary fallback validation for local-model output drift
  • repair missing LLM follow-ups with deterministic metadata follow-ups
  • canonicalize unambiguous citation aliases and use source_type to disambiguate name
  • show live summary runtime and clearer fallback/error messaging in simulation details UI

Checklist

  • Code follows project style guidelines
  • Self-reviewed code
  • No new warnings
  • Tests added or updated (if needed)
  • All tests pass (locally and CI/CD)
  • Documentation/comments updated (if needed)
  • Breaking change noted (if applicable)

Deployment Notes (if any)

Configure local-provider environment variables for Ollama endpoint/model selection in environments that will use hosted local inference.

No database migration expected.

Environments that still set ASSISTANT_OPENAI_* or ASSISTANT_ANTHROPIC_* must remove those settings and use only the supported ollama or livai assistant configuration.

Validation

  • make backend-test
  • uv run pytest tests/features/assistant/test_orchestrator.py
  • make frontend-lint
  • pnpm --dir frontend run type-check

@tomvothecoder tomvothecoder changed the title Add support for locally hosted LLM providers (e.g. Ollama) Remove OpenAI and Anthropic assistant provider support May 21, 2026
@tomvothecoder tomvothecoder changed the title Remove OpenAI and Anthropic assistant provider support Add support for locally hosted LLM providers (e.g., Ollama) May 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant