Add Intruder Dimension Mitigation for LoRA Adapters

[ved1beta]

adds intrude dimensions detection + Back projection #2907

Changes

• New utility module (src/peft/utils/intruder_dimensions.py):

  • detect_intruder_dimensions(): Identifies intruder dimensions via SVD analysis
  • mitigate_intruder_dimensions(): Scales down intruder contributions
  • project_delta_to_lora(): Projects mitigated weights back to LoRA format
  • compute_reconstruction_error(): Validates reconstruction quality

• Two mitigation paths in PeftModel:

  • reduce_intruder_dimensions(): Non-destructive (creates new adapter, preserves original)
  • merge_and_unload_with_reduced_intruder_dimensions(): Destructive (merges into base model)

• Comprehensive tests (20 test cases covering edge cases, mixed precision, device transfers)

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM

# Load fine-tuned model
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
model = PeftModel.from_pretrained(base_model, "path/to/adapter")

# Apply mitigation (non-destructive)
model = model.reduce_intruder_dimensions(
    old_adapter_name="default",
    new_adapter_name="default_mitigated",
    mitigation_lambda=0.75,
)

# Save mitigated adapter
model.save_pretrained("./mitigated_adapter")