Skip to content

[FLINK-39758] Fix DECIMAL OOB in SchemaMergingUtils#4419

Open
haruki-830 wants to merge 1 commit into
apache:masterfrom
haruki-830:FLINK-39758
Open

[FLINK-39758] Fix DECIMAL OOB in SchemaMergingUtils#4419
haruki-830 wants to merge 1 commit into
apache:masterfrom
haruki-830:FLINK-39758

Conversation

@haruki-830
Copy link
Copy Markdown
Contributor

Summary

This commit fixes the DECIMAL precision overflow (Out-Of-Bounds) issue in SchemaMergingUtils and SchemaUtils, which previously caused pipeline failures when merging DECIMAL types from multiple tables with significantly different precisions and scales.

Key Changes

  1. New createDecimalBounded Utility Method
  • Introduced createDecimalBounded method in SchemaMergingUtils to handle DECIMAL precision overflow gracefully
  • When computed precision exceeds DecimalType.MAX_PRECISION (38), the method truncates scale digits instead of throwing an exception
  • Truncation formula: lossDigits = precision - MAX_PRECISION, then resultPrecision = precision - lossDigits, resultScale = scale - lossDigits
  • Follows the same approach as Flink SQL for handling DECIMAL overflow
  1. SchemaMergingUtils Fix
  • Replaced Preconditions.checkArgument with createDecimalBounded call in mergeDecimalTypes()
  • Removed the hard failure that threw IllegalArgumentException when merged precision exceeded 38
  • SchemaUtils Fix
  • Applied the same createDecimalBounded fix in inferWiderType() for DECIMAL-to-DECIMAL merging
  • Applied the same fix in mergeExactNumericsIntoDecimal() for DECIMAL-to-integer merging
  1. Comprehensive Testing
  • Updated SchemaUtilsTest to verify that overflow cases now return bounded DECIMAL types instead of throwing exceptions
  • Added testMergingDecimalWithOutOfBoundPrecisions with 6 parameterized test cases covering various overflow scenarios
  • Refactored FlinkPipelineComposerITCase to support generic merging test infrastructure via runGenericMergingTest

Before fix: Merging DECIMAL(5, 5) and DECIMAL(38, 0) throws IllegalArgumentException (requires precision 43, max 38)
After fix: Merging DECIMAL(5, 5) and DECIMAL(38, 0) returns DECIMAL(38, 0)

JIRA Reference

https://issues.apache.org/jira/browse/FLINK-39758

@haruki-830
Copy link
Copy Markdown
Contributor Author

@lvyanquan could you please help review this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant