[CALCITE-7187] Java UDF byte arrays cannot be mapped to VARBINARY

[macroguo-ghy]

Jira Link

CALCITE-7187

Changes Proposed

This PR fixes Java UDF byte[] handling so Calcite maps SQL VARBINARY
values consistently during enumerable code generation.

Reproduction:

select "adhoc".bytearray(cast('test' as varchar)) = x'74657374' as c;
values "adhoc".bytearray_length(x'74657374');

Changes in this PR:

• add byte[] <-> ByteString conversion helpers in SqlFunctions
• wire those conversions through EnumUtils and BuiltInMethod
• add UdfTest coverage for byte[] return values and byte[] parameters
• document byte[] and ByteString as supported Java representations of SQL
  VARBINARY for Java UDFs

[Copilot]

Pull request overview

Fixes Calcite enumerable code generation so Java UDF byte[] parameters/return values are consistently mapped to SQL VARBINARY (aligned with existing ByteString behavior), with tests and documentation updates.

Changes:

• Added byte[] ↔ ByteString conversion helpers in SqlFunctions and exposed them via BuiltInMethod.
• Wired conversions into enumerable expression conversion paths (EnumUtils) for UDF argument/return handling.
• Added UDF test coverage and documented supported Java representations for VARBINARY.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
testkit/src/main/java/org/apache/calcite/util/Smalls.java	Adds UDF helpers returning/accepting byte[] for test scenarios.
core/src/test/java/org/apache/calcite/test/UdfTest.java	Registers new UDFs and adds regression tests for byte[] VARBINARY interop.
core/src/main/java/org/apache/calcite/runtime/SqlFunctions.java	Introduces conversion helpers between byte[] and ByteString.
core/src/main/java/org/apache/calcite/util/BuiltInMethod.java	Adds reflective method handles for the new conversion helpers.
core/src/main/java/org/apache/calcite/adapter/enumerable/EnumUtils.java	Applies conversions during enumerable codegen type adaptation.
site/_docs/reference.md	Documents supported Java types for SQL VARBINARY in Java UDFs.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

ByteArrayLengthFunction.eval does not handle null input; if a SQL VARBINARY argument is NULL (or conversion yields null), this will throw a NullPointerException. Consider changing the signature to return a boxed Integer and return null when bytes is null (or otherwise define/implement the desired NULL semantics).

Suggested change

	public static int eval(byte[] bytes) {
	public static @Nullable Integer eval(@Nullable byte[] bytes) {
	if (bytes == null) {
	return null;
	}

[macroguo-ghy]

UDF in Smalls is just for test

[Copilot]

The doc mentions ByteString without qualification; since there are multiple ByteString types in the Java ecosystem, consider using the fully qualified name (org.apache.calcite.avatica.util.ByteString) or linking to the API docs to avoid ambiguity. Also, the sentence about Byte[] could be misread as "not supported at all"; if the intent is "not supported as a VARBINARY representation", consider stating that explicitly.

Suggested change

	For Java UDFs, `byte[]` and `ByteString` are supported Java representations of
	SQL `VARBINARY` values for parameters and return types. Boxed byte arrays
	(`Byte[]`) are not supported.
	For Java UDFs, `byte[]` and [`ByteString`]({{ site.apiRoot }}/org/apache/calcite/avatica/util/ByteString.html) are supported Java representations of
	SQL `VARBINARY` values for parameters and return types. Boxed byte arrays
	(`Byte[]`) are not supported as Java representations of SQL `VARBINARY` parameters or return types.

[sonarqubecloud]

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
63.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud