Scalarizer Tracker

[ValerianRey]

This issue tracks the candidate methods that could maybe be implemented as Scalarizer in torchjd.scalarization (see #666).

Name	Ref	Stateful	Existing implementations	Special Remarks
Sum	-	No	(trivial)
Mean	-	No	(trivial)	Sometimes called Equal Weights (EW) in research papers.
Linear	-	No	(trivial)	Sometimes called Linear Scalarization (LS) in research papers. Name it Linear or Constant?
Random	Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learning (TMLR 2022, 200 citations)	No	LibMTL (official)	Sometimes called Random Loss Weighting (RLW) in research papers. Name it RLW or Random?
STCH (Smooth TCHebyshev)	Smooth Tchebycheff Scalarization for Multi-Objective Optimization (ICML 2024, 87 citations)	No	official, LibMTL
GLS (Geometric Loss Strategy)	MultiNet++: Multi-Stream Feature Aggregation and Geometric Loss Strategy for Multi-Task Learning (CVPR workshop 2019, 149 citations)	No	LibMTL
IMTL-L	Towards Impartial Multi-Task Learning (ICLR 2021, 279 citations)	?	official, LibMTL (maybe this is IMTL-G)	Need more investigation
FAMO	FAMO: Fast Adaptive Multitask Optimization (NeurIPS 2023, 124 citations)	?	official, LibMTL	Not sure this is a Scalarizer, need more investigation.
GradNorm	GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks (ICML 2018, 2334 citations)	Yes, trainable state	unofficial, LibMTL	Not sure this is a Scalarizer, need more investigation.