gate mode diff

[theNiemand]

Would you consider supporting softplus as an alternative activation mode (e.g., via a flag like gate_mode: "lower_bound" | "softplus")?
And, I tested fla's chunk_kda vs FlashKDA, in end to end result, lower_bound gate mode will cause repeatition in model's output.Both fla's chunk_kda with lower_bound gate mode and FlashKDA have the same problem, but when use fla's chunk_kda with softplus gate mode, it works fine.
the model used is kimi linear (https://modelscope.cn/models/moonshotai/Kimi-Linear-48B-A3B-Instruct).

[zhiyuan1i]

FlashKDA is built for models trained with lower_bound, and the softplus gate was removed because it doesn’t support lower_bound and isn’t efficient on hardware.

[theNiemand]

FlashKDA is built for models trained with lower_bound, and the softplus gate was removed because it doesn’t support lower_bound and isn’t efficient on hardware.

thanks for your reply. And how can I know for sure whether kimi linear is trained with lower_bound or not？In fla_chunk_kda，use_lower_bound is just like an optimize option(seems like a trade-off between hardware efficiency and model accuracy) in inference mode.Or if kimi linear is not trained with lower_bound, it's just will be totally wrong to inference with lower_bound?

[zhiyuan1i]

FlashKDA is built for models trained with lower_bound, and the softplus gate was removed because it doesn’t support lower_bound and isn’t efficient on hardware.

thanks for your reply. And how can I know for sure whether kimi linear is trained with lower_bound or not？In fla_chunk_kda，use_lower_bound is just like an optimize option(seems like a trade-off between hardware efficiency and model accuracy) in inference mode.Or if kimi linear is not trained with lower_bound, it's just will be totally wrong to inference with lower_bound?

Kimi Linear Has no Lower Bound, therefore, FLASHKDA could not used in Kimi Linear(W48BA3B).

The lower bound was determined during the training process; it’s not an inference configuration.

[theNiemand]

FlashKDA is built for models trained with lower_bound, and the softplus gate was removed because it doesn’t support lower_bound and isn’t efficient on hardware.

thanks for your reply. And how can I know for sure whether kimi linear is trained with lower_bound or not？In fla_chunk_kda，use_lower_bound is just like an optimize option(seems like a trade-off between hardware efficiency and model accuracy) in inference mode.Or if kimi linear is not trained with lower_bound, it's just will be totally wrong to inference with lower_bound?

Kimi Linear Has no Lower Bound, therefore, FLASHKDA could not used in Kimi Linear(W48BA3B).

The lower bound was determined during the training process; it’s not an inference configuration.

i see.Is there any opensource model was trained with lower_bound ? On huggingface, it seems that there is only kimi linear W48BA3B using KDA.