If you add a scene to source toggler, then delete it - it's still being source toggled. I usually reboot OBS to get it to work again.