log likelihood becomes nan after some steps

[Sannndy0000]

Hi,

I had the problem that my estimator got into NaN values after some steps. Sometimes from the second step, and sometimes it trains normally for longer (>1000 steps), depending on different setups (i.e. linear layer dimension, depth, learning rate). Any idea on this?

Thank you!

[Linear95]

Hi, this problem seems to be caused by one of the two possible issues: 1. the numeral instability of logvar calculation; 2. the negative MI estimation.

For the first possible issue, make sure your logvar is upper-bounded so that logvar.exp() will not become NaN.

For the second possible issue, please check some suggestions to solve the negative estimation at #17