Skip to content

Conversation

@zhaixuejun1993
Copy link

@zhaixuejun1993 zhaixuejun1993 commented Dec 5, 2025

Enabling the OpenVINO backend fallback to Llama.cpp CPU backend mechanisms.

Below is a summary of the main process:

1. Dynamic Dimension Computation
Function: compute_cgraph_dynamic_dims()
Purpose: Determines the dynamic dimensions for each node in the computation graph. This is essential for handling nodes with variable shapes during runtime.
Process:
Traverses the computation graph.
Assigns dynamic dimension indices to nodes based on their operation type and dependencies.
Handles specific operations like [GGML_OP_VIEW], [GGML_OP_RESHAPE], and others to propagate dynamic dimensions.

2. Adding Extra Model Outputs
Function: add_extra_model_outputs_for_fallback()
Purpose: Ensures that all relevant nodes in the computation graph are included as model outputs for fallback scenarios.
Process:
Maps tensor data addresses to their corresponding nodes, excluding [GGML_OP_VIEW] nodes.
Adds nodes to the [m_model_outputs] map if they are not already present.

3. Adding Extra Model Inputs
Function: add_extra_model_inputs_for_fallback()
Purpose: Ensures that all necessary input nodes are included as model inputs for fallback scenarios.
Process:
Iterates through the source nodes of each computation graph node.
Skips nodes already in [m_model_weights] or [m_model_inputs].
Excludes intermediate nodes from [m_node_info_list].
Creates OpenVINO parameter nodes for eligible source nodes and updates the [m_inputs] and [m_model_inputs] maps.

For example:

  1. here is the split results from Llama.cpp with OpenVINO Backend (ROPE unsupport). For the sub-graph run on OpenVINO backend, which needs add new inputs and outputs node for the continue sub-graph execute.
image

@github-actions github-actions bot added the ggml label Dec 5, 2025
@zhaixuejun1993 zhaixuejun1993 changed the title OpenVINO BK fallback to Llama.CPP CPU BK, verified with ROPE unsupported OpenVINO BK fallback to Llama.CPP CPU BK Dec 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant