Add fuse_gelu_sigmoid_approximation optimization pass

[Rakshitha-Ireddi]

Description

Add fuse_gelu_sigmoid_approximation optimization pass that detects and fuses the sigmoid approximation of GELU activation function.

The pattern x * sigmoid(1.702 * x) is fused into gelu(x, mode="SIGMOID_APPROXIMATION").

Motivation

The sigmoid approximation of GELU is commonly used in neural networks (e.g., some transformer variants, GPT-style models). Currently, this pattern remains decomposed as mul -> sigmoid -> mul, missing the opportunity for backend optimizations. This pass enables the backend to recognize and potentially accelerate this activation pattern.

Changes

• Add optimize_gelu_sigmoid.py with fuse_gelu_sigmoid_approximation pass
• Register the pass in __init__.py
• Add pass to the default pass pipeline after fuse_gelu_exact
• Add comprehensive tests in test_gelu_sigmoid_pass.py

Testing

• Added 8 unit tests covering:
  • Basic fusion
  • Reversed operand order
  • Different tensor shapes (1D to 4D)
  • Negative cases (wrong constant, intermediate outputs, different inputs)
  • FP16 support
  • Numerical correctness

Files Changed

File	Change
coremltools/converters/mil/mil/passes/defs/optimize_gelu_sigmoid.py	New pass implementation
coremltools/converters/mil/mil/passes/tests/test_gelu_sigmoid_pass.py	Test suite
coremltools/converters/mil/mil/passes/__init__.py	Import registration
coremltools/converters/mil/mil/passes/pass_pipeline.py	Pipeline integration

Contributors

• Ireddi Rakshitha
• Yaswanth Devavarapu

[TobyRoseman]

It's not clear to me where this pattern is being detected.

[Rakshitha-Ireddi]

I have removed the Pattern 2 reference from the docstring since only Pattern 1 (x -> mul(1.702) -> sigmoid -> mul(x)) is implemented.

[TobyRoseman]

Initial CI Run: https://gitlab.com/coremltools1/coremltools/-/pipelines/2315236133

[TobyRoseman]

The new unit tests for this PR are failing on CI. Please take a look:
https://gitlab.com/coremltools1/coremltools/-/pipelines/2315236133

[Rakshitha-Ireddi]

Fixed! The issues were:

1. An incorrect early-return check that rejected patterns where the final mul is a block output (which is the normal case when the result is returned)
2. The test's expected_output_shapes parameter was incorrectly formatted as a set {(2, 3)} instead of a dictionary {output_name: (2, 3)}

[TobyRoseman]

Updated CI:
https://gitlab.com/coremltools1/coremltools/-/pipelines/2317611329

[Rakshitha-Ireddi]

@TobyRoseman
From updated CLI, Only one job was failing.
The MIL tests are now passing. The test_py310_pytorch_executorch failure appears to be unrelated to my changes - it's an import error in coremltools.converters.mil.frontend.torch which my PR doesn't touch. My changes only affect mil/passes/defs/optimize_gelu_sigmoid.py and mil/passes/tests/test_gelu_sigmoid_pass.py

Could you please check and let me know what can i do for this.

[TobyRoseman]

@Rakshitha-Ireddi - I agree that failure is not related to your PR; I'm seeing the same failure in main. I'll investigate and let you know when I'm ready for you to rebase this branch.

[Rakshitha-Ireddi]

Thanks for checking! I will hold off on changes and wait for your update on the main branch issue.