Switching from SGD to AdamW: Significant mIoU boost but concerns about Deformable Offsets stability

[lxll-0904]

Hello @HuguesTHOMAS ,

First of all, thank you for this excellent open-source work. It has been a solid baseline for my research.

I have a question regarding the choice of optimizer and its impact on the Deformable KPConv module.

Observation: I noticed that the original implementation uses SGD with momentum, and specifically applies a lower learning rate to the deformable offsets (controlled by config.deform_lr_factor).

Experiment: I recently attempted to train the model on my custom dataset using the AdamW optimizer instead of SGD. In my implementation, I applied the same learning rate to all parameters (effectively removing the specific deform_lr_factor constraint).

Result: To my surprise, the model converged much faster, and the final mIoU increased by approximately 10% (absolute points) compared to the baseline SGD training on the same dataset.

Questions:

Is this significant performance gap expected? It seems SGD might be harder to tune for certain datasets, whereas AdamW adapts better.

The main concern: Does switching to AdamW (and removing the explicit low LR constraint for offsets) pose any theoretical risks to the Deformable KPConv mechanism?

For example, could this cause the kernel points to "move too fast" or become unstable (losing their geometric physical meaning), even if the validation mIoU looks good?

I would appreciate any insights on whether this modification is considered safe or if there are potential downsides I should be aware of.

Thanks again!

[HuguesTHOMAS]

Hi @lxll-0904,

Thanks for your message. Your findings are very interesting. In my experiments I did not have the same results. Let me first tell you what I experienced on the datasets from the paper for a bit of context.

First I tried AdamW as well but did not find a significant difference so I decided to stick with the SGD optimizer. Then, when designing the deformable kernel, I faced several issues which forced me to implement corrections (for example the deformable points would all collapse to the center and the network wouldn't learn anything, which is why I had to add regularization losses. My conclusion at the time was that the segmentation loss itself was not providing a good signal for getting offset gradients, and that maybe if the convolution weights themselves had at least learned a little bit more, then the offset might be learned better. Also I thought at the time that if the offset would be moving at every step, the weights learned by the convolutions would not be right and would have to adapt everytime. For these reasons, I thought that slowing the learning rate of the dermations would help (giving the convolution weights more time to adapt as the kernel points are deformed).

This was a long time ago and I am not sure anymore but I think it helped the performances which is why I kept that idea. But your experiment proves that with a different dataset and optimizer, deform_lr_factor might not be needed.

Now to answer your questions:

• Performance gap. This is not surprising, AdamW is a more versatile optimizer. In my case SGD was well tuned so that was not making a difference, but in your case it could make a difference. Furthermore the perf gap can also be explained by the deformation being learned faster, allowing the convolution weight more time to be learned after the kernel points are more stable
• I dont think there is any theoretical risk, maybe some risk of the loss diverging if the learning rate of Adam is set too high in the beginning. Honestly the concept of deformable kernel has not been explored that much and we don't know much about it. Even for the KPConv paper, this was an addition we tried but did not have time to explore too much. Even in the followup paper KPConvX, I chose to avoid deformable kernels for more simplicity and clarity.
• I would advise to use the visualization tools I created (or make your own) to check the deformed kernels And verify that nothing is going wrong, but if the mIoU looks good, there is no reason for the deformation to be wrong

Best,
Hugues