Make sure --free_gpu_mem still works when using CKPT-based diffuser model#2367

lstein merged 5 commits intoinvoke-ai:maininvoke-ai/InvokeAI:mainfrom

addianto:bugfix/free-gpu-mem-diffuseraddianto/InvokeAI:bugfix/free-gpu-mem-diffuserCopy head branch name to clipboard