Smooth regularization for SVD pullback to avoid gradient discontinuities near degeneracies #147
Description
Hello, developers.
In the current implementation of svd_pullback!, near-degenerate singular value differences (and sums) are handled via a hard thresholding function inv_safe:
inv_safe(a::Number, tol = defaulttol(a)) = abs(a) < tol ? zero(a) : inv(a)
While this avoids numerical explosions at exact degeneracies, it introduces discontinuities in the gradient manifold and thus might work unfavorably in an optimization problem?
So why not use a smooth Lorentzian broadening?
A smooth alternative is to replace hard thresholding with a Lorentzian broadening, as used in the tensor-network literature (see Liao et al., Phys. Rev. X 9, 031041 (2019)). Specifically, replace inv_safe's logic with
f(a, ε) = a / (a^2 + ε)
where ε > 0 is a small regularization parameter (e.g. 1e-12 which has been used in the literature).
This function behaves like 1/a away from degeneracy, while smoothly going to zero as a → 0 which is just an approximation value of gradient like but should make optimization more stable than the current one.
I can try this alternative myself, but I would like to hear your thoughts since I am not very familiar with these numerical issues.
I am quite curious to know which approach is considered better in general situations?