Proposal: C++ header-only Product Quantization and SVD Procrustes OPQ

[cluster2600]

Motivation

zvec has Python-level OPQ (python/zvec/backends/opq.py) using QR decomposition, but no C++ PQ/OPQ primitives. A header-only C++ implementation would:

• Enable PQ in the native index pipeline (no Python overhead)
• Improve OPQ rotation quality via SVD Procrustes (vs current QR approach)
• Have zero new dependencies (self-contained Jacobi SVD solver)

Proposed approach

Two new headers under src/ailego/:

product_quantizer.h

• k-means training for codebook learning
• Encode: float32 vector → uint8 codes
• Decode: uint8 codes → float32 approximation
• Asymmetric distance computation
• Distortion measurement

opq.h

• SVD-based Orthogonal Procrustes rotation
• Self-contained Jacobi SVD (no LAPACK dependency)
• learn_rotation(data, pq) → rotation matrix
• Compatible with product_quantizer.h

Also update the Python OPQ's _learn_rotation to use SVD Procrustes instead of QR decomposition for better rotation quality.

Questions for maintainers

1. Is src/ailego/ the right home for these headers, or would you prefer a subdirectory like src/ailego/quantization/?
2. Any existing C++ PQ work in progress that this might overlap with?
3. Should the Python OPQ upgrade (QR → SVD) be a separate PR?

Draft implementation: #173

[richyreachy]

Thanks @cluster2600 . PQ ,as we planned, will be supported by using zvec IndexConverter as well as a type of metric, which may need a unified design and implementation.

[cluster2600]

Thanks for the context @richyreachy! The IndexConverter approach sounds like a clean way to integrate PQ. Excited to see the unified design come together — happy to test things out or contribute when it's ready.

[richyreachy]

I close this for now. We can refer back to it once the related issue is on the table. Thanks.