Skip to content

Torch PTQ ONNX export example add for windows + small fixes to existing torch onnx export path #1027

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
hthadicherla wants to merge 4 commits into
base: main
Choose a base branch
from
+1,054 −69
Open

Torch PTQ ONNX export example add for windows + small fixes to existing torch onnx export path #1027

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0b83d06
Added torch ptq followed by onnx export followd by GQA surgery exampl…
hthadicherla Mar 11, 2026
036942c
Fixed no of args passed to the backward function
hthadicherla Mar 12, 2026
f84cdb1
reverted benchmark.py to its original state
hthadicherla Mar 12, 2026
23ee38e
fixed export utils to pick up head dim from config by default and onl…
hthadicherla Mar 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions examples/windows/torch_onnx/llm_export/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# LLM Export (Windows)

Export LLMs from PyTorch to ONNX with quantization and GQA surgery.

## Supported Precisions

- `nvfp4` — NVIDIA FP4 quantization
- `int4_awq` — INT4 AWQ quantization
- `int8_sq` — INT8 SmoothQuant

## Usage

### NVFP4

```bash
python llm_export.py --hf_model_path "meta-llama/Llama-3.2-3B-Instruct" --dtype nvfp4 --output_dir ./llama3.2-3b-nvfp4
```

### INT4 AWQ

```bash
python llm_export.py --hf_model_path "meta-llama/Llama-3.2-3B-Instruct" --dtype int4_awq --output_dir ./llama3.2-3b-int4
```

### INT8 SmoothQuant

```bash
python llm_export.py --hf_model_path "Qwen/Qwen2.5-3B-Instruct" --dtype int8_sq --output_dir ./qwen-3b-int8
```

## Options

| Argument | Description |
|---|---|
| `--hf_model_path` | HuggingFace model name or local path |
| `--dtype` | Quantization precision (`fp16`, `fp8`, `int4_awq`, `int8_sq`, `nvfp4`) |
| `--output_dir` | Directory to save the exported ONNX model |
| `--calib_size` | Calibration dataset size (default: 512) |
| `--save_original` | Save the pre-surgery ONNX for debugging |
| `--trust_remote_code` | Trust remote code when loading from HuggingFace |
| `--onnx_path` | Skip export, run surgery on an existing ONNX |
| `--config_path` | Path to config.json if not alongside the model |
Loading
Loading