[QNN] Hope to know how to support `v68` arch soc with QNN?

[ecccccsgo]

Hello,

I was work with some old platform like SA8295 with htp version v68. In this repo and the aihub repo, When i use w4a16 recipe to quantize the model like smolvlm/qwen, it was error with:

[QNN Partitioner Op Support]: aten.view_copy.default | True
[QNN Partitioner Op Support]: aten.linear.default | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[QNN Partitioner Op Support]: aten.permute_copy.default | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[QNN Partitioner Op Support]: aten.permute_copy.default | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[QNN Partitioner Op Support]: aten.view_copy.default | True
[ERROR] [Qnn ExecuTorch]:  <E> [4294967295] has incorrect Value 68, expected >= 73.

[ERROR] [Qnn ExecuTorch]:  <E> QnnBackend_validateOpConfig failed 3110

[ERROR] [Qnn ExecuTorch]:  <E> Failed to validate op aten_native_layer_norm_default_24 with error 0xc26

But in some paper like AutoNeural, they use the ViT: W8A16, language model:W4A16 (page 10, in the table2).

it seems the v68 arch support this type of ops. I hope you can give me some infomation about how I can use w4a16 in llm/lvm to support larger model. :)

cc @cccclai @winskuo-quic @shewu-quic @haowhsu-quic @DannyYuyang-quic @cbilgin

[ecccccsgo]

seems to find some info to do some experiments: #15410

[shewu-quic]

Hi @ecccccsgo,
According to your error log, the issue is caused by a layer_norm op validation failure.
By default, we use 16a16w for 16a layer_norm to maintain accuracy. However, as stated in the QNN documentation, 16a16w requires at least arch V73.
If you need to run this on a V68 device, you can try the following adjustment.

Patch to run Qwen 2.5 0.5b on V68 (click to expand)

diff --git a/backends/qualcomm/quantizer/custom_annotation.py b/backends/qualcomm/quantizer/custom_annotation.py
index 9558aeced2..aef4afeadb 100644
--- a/backends/qualcomm/quantizer/custom_annotation.py
+++ b/backends/qualcomm/quantizer/custom_annotation.py
@@ -10,6 +10,7 @@ import torch
 from executorch.backends.qualcomm.quantizer.quantizer import (
     get_16a8w_qnn_ptq_config,
     get_16a8w_qnn_qat_config,
+    get_16a4w_qnn_ptq_config,
     get_8a8w_qnn_ptq_config,
     get_8a8w_qnn_qat_config,
     get_ptq_per_channel_quant_config,
@@ -499,3 +500,31 @@ def get_custom_quant_ios_dtype(
         u.target for u in list(node.users.keys())
     ] + [node.target]:
         return sharding_dtype
+
+
+def custom_annotation_16a4w_layer_norm(gm):
+    use_16a4w_config = get_16a4w_qnn_ptq_config()
+    use_16a4w_config.weight = QuantizationSpec(
+        dtype=torch.uint8,
+        quant_min=0,
+        quant_max=15,
+        qscheme=torch.per_tensor_symmetric,
+        ch_axis=0,
+        observer_or_fake_quant_ctr=use_16a4w_config.weight.observer_or_fake_quant_ctr,
+    )
+    for node in gm.graph.nodes:
+        if node.target != torch.ops.aten.layer_norm.default:
+            continue
+        act_node = node.args[0]
+        weight_node = node.args[2]
+        bias_node = None
+        input_qspec_map = {act_node: use_16a4w_config.input_activation, weight_node: use_16a4w_config.weight}
+        if len(node.args) > 2:
+            bias_node = node.args[3]
+            input_qspec_map[bias_node] = use_16a4w_config.bias
+        
+        node.meta[Q_ANNOTATION_KEY] = QuantizationAnnotation(
+            input_qspec_map=input_qspec_map,
+            output_qspec=use_16a4w_config.output_activation,
+            _annotated=True,
+        )
diff --git a/examples/qualcomm/oss_scripts/llama/static_llm_quant_recipe.py b/examples/qualcomm/oss_scripts/llama/static_llm_quant_recipe.py
index ca06a89142..fdeffa79d6 100644
--- a/examples/qualcomm/oss_scripts/llama/static_llm_quant_recipe.py
+++ b/examples/qualcomm/oss_scripts/llama/static_llm_quant_recipe.py
@@ -7,7 +7,7 @@
 from typing import Optional
 
 import torch
-from executorch.backends.qualcomm.quantizer.custom_annotation import annotate_kv_8bit
+from executorch.backends.qualcomm.quantizer.custom_annotation import annotate_kv_8bit, custom_annotate_matmul_16a8w, custom_annotation_16a4w_layer_norm
 from executorch.backends.qualcomm.quantizer.quant_recipe import (
     QuantGranularity,
     QuantRecipe,
@@ -485,16 +485,17 @@ class Qwen2_5_0_5BQuantRecipe(StaticLLMQuantRecipe):
             act_observer=MinMaxObserver,
             granularity=QuantGranularity.PER_TENSOR,
             verbose=verbose,
-        ).add_node_target(
-            {
-                torch.ops.aten.conv2d.default,
-            },
-            QuantDtype.use_16a4w_block,
-            False,
-            act_observer=MinMaxObserver,
-            granularity=QuantGranularity.PER_BLOCK,
-            extra_kwargs={"block_size": (1, 16, 1, 1)},
+        # ).add_node_target(
+        #     {
+        #         torch.ops.aten.conv2d.default,
+        #     },
+        #     QuantDtype.use_16a4w_block,
+        #     False,
+        #     act_observer=MinMaxObserver,
+        #     granularity=QuantGranularity.PER_BLOCK,
+        #     extra_kwargs={"block_size": (1, 16, 1, 1)},
         )
+        self.recipe.custom_quant_annotations.extend([custom_annotation_16a4w_layer_norm,custom_annotate_matmul_16a8w, annotate_kv_8bit])
 
 
 class Qwen2_5_1_5BQuantRecipe(StaticLLMQuantRecipe):
``

[cccclai]

@shewu-quic thanks for the detailed patch! Feel like it was asked more than once, is it possible to expand the llm quantization recipe by SoC?

[shewu-quic]

Yes, it’s a good idea to begin expanding the LLM quantization recipe by SOC. Butt I think we’ve already enabled validation in the quantizer, and I’m currently looking into whether it’s possible to automatically update the config if it’s not supported by the given SOC.