definition of relative error

[j4yan]

Currently the relative error when the reference value is zero is defined using an epsilon to avoid division by zero; see https://github.com/ROCmSoftwarePlatform/composable_kernel/blob/313bbea5886850acab286f45e9d9816cf0b0dca0/library/include/ck/library/host_tensor/host_tensor.hpp#L327

The problem is that this approve results in very large relative error even if the absolute error is small. For example, (v = 1e-8, v_ref = 0, eps = 1e-10), then ref_diff = 100. We can use more reasonable definition, for example:
rel_diff = (v - v_ref) / max(|v|, |v_ref|). See https://en.wikipedia.org/wiki/Relative_change_and_difference for other alternatives.

[j4yan]

The suggested definition doesn't work when v = v_ref = 0 (most likely happens for integers), so we should check it first.

[asroy]

what about:
rel_diff = (v - v_ref) / max(|v|, |v_ref|, |eps|)?

[j4yan]

It covers the corner case I mentioned above, but why do we introduce a user-defined and hard-coded variable?

[aosewski]

In the mentioned wiki article all relative difference definitions are not defined when the f(x,y) is equal to zero. IMHO this is the special case which should be treated separately.

[j4yan]

@asroy Agree with @aosewski.