generated from amazon-archives/__template_Custom
-
Notifications
You must be signed in to change notification settings - Fork 188
Add time conversion functions for convert command #5210
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
ritvibhatt
wants to merge
22
commits into
opensearch-project:main
Choose a base branch
from
ritvibhatt:convert-time-functions
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
22 commits
Select commit
Hold shift + click to select a range
efa81ce
add time conversion functions
ritvibhatt 13d50d0
add integ tests
ritvibhatt eae36db
fix number of arguments
ritvibhatt 73f4088
fix timeformat parsing in integ tests
ritvibhatt 82fd69f
update comments
ritvibhatt b45ff6d
apply spotless
ritvibhatt becd73e
update convert command doc
ritvibhatt 93f7768
sql cli test fix
ritvibhatt 80b16a9
update utils
ritvibhatt fdc0b67
fix integ tests
ritvibhatt da6cfeb
add explain tests
ritvibhatt 5ac49be
fix test
ritvibhatt 215b59b
apply spotless
ritvibhatt 0ef7745
fix null timeformat
ritvibhatt bb8ebf1
fix tests
ritvibhatt 2b10e9a
empty
ritvibhatt 6f6162e
update docs
ritvibhatt afa3915
empty
ritvibhatt 4364539
fix parsing
ritvibhatt 8d3b734
update docs table format
ritvibhatt 39e95ec
update doc example
ritvibhatt 7394650
fix ctime fractional seconds
ritvibhatt File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
105 changes: 105 additions & 0 deletions
105
core/src/main/java/org/opensearch/sql/expression/function/udf/CTimeConvertFunction.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,105 @@ | ||
| /* | ||
| * Copyright OpenSearch Contributors | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| package org.opensearch.sql.expression.function.udf; | ||
|
|
||
| import java.time.Instant; | ||
| import java.time.ZoneId; | ||
| import java.time.ZonedDateTime; | ||
| import java.util.List; | ||
| import org.apache.calcite.adapter.enumerable.NotNullImplementor; | ||
| import org.apache.calcite.adapter.enumerable.NullPolicy; | ||
| import org.apache.calcite.adapter.enumerable.RexToLixTranslator; | ||
| import org.apache.calcite.linq4j.tree.Expression; | ||
| import org.apache.calcite.linq4j.tree.Expressions; | ||
| import org.apache.calcite.rex.RexCall; | ||
| import org.apache.calcite.sql.type.SqlReturnTypeInference; | ||
| import org.opensearch.sql.calcite.utils.PPLOperandTypes; | ||
| import org.opensearch.sql.calcite.utils.PPLReturnTypes; | ||
| import org.opensearch.sql.expression.datetime.StrftimeFormatterUtil; | ||
| import org.opensearch.sql.expression.function.ImplementorUDF; | ||
| import org.opensearch.sql.expression.function.UDFOperandMetadata; | ||
|
|
||
| /** | ||
| * PPL ctime() conversion function. Converts UNIX epoch timestamps to human-readable time strings | ||
| * using strftime format specifiers. Default format: {@code %m/%d/%Y %H:%M:%S}. | ||
| */ | ||
| public class CTimeConvertFunction extends ImplementorUDF { | ||
|
|
||
| private static final String DEFAULT_FORMAT = "%m/%d/%Y %H:%M:%S"; | ||
|
|
||
| public CTimeConvertFunction() { | ||
| super(new CTimeImplementor(), NullPolicy.ANY); | ||
| } | ||
|
|
||
| @Override | ||
| public SqlReturnTypeInference getReturnTypeInference() { | ||
| return PPLReturnTypes.STRING_FORCE_NULLABLE; | ||
| } | ||
|
|
||
| @Override | ||
| public UDFOperandMetadata getOperandMetadata() { | ||
| return PPLOperandTypes.ANY_OPTIONAL_STRING; | ||
| } | ||
|
|
||
| public static class CTimeImplementor implements NotNullImplementor { | ||
| @Override | ||
| public Expression implement( | ||
| RexToLixTranslator translator, RexCall call, List<Expression> translatedOperands) { | ||
| if (translatedOperands.isEmpty()) { | ||
| return Expressions.constant(null, String.class); | ||
| } | ||
| Expression fieldValue = Expressions.box(translatedOperands.get(0)); | ||
| if (translatedOperands.size() == 1) { | ||
| return Expressions.call(CTimeConvertFunction.class, "convert", fieldValue); | ||
| } | ||
| Expression timeFormat = Expressions.box(translatedOperands.get(1)); | ||
| return Expressions.call( | ||
| CTimeConvertFunction.class, "convertWithFormat", fieldValue, timeFormat); | ||
| } | ||
| } | ||
|
|
||
| public static String convert(Object value) { | ||
| return convertWithFormat(value, null); | ||
| } | ||
|
|
||
| public static String convertWithFormat(Object value, Object timeFormatObj) { | ||
| Double timestamp = toEpochSeconds(value); | ||
| if (timestamp == null) { | ||
| return null; | ||
| } | ||
| String format = (timeFormatObj != null) ? timeFormatObj.toString().trim() : DEFAULT_FORMAT; | ||
| if (format.isEmpty()) { | ||
| return null; | ||
| } | ||
| try { | ||
| long seconds = timestamp.longValue(); | ||
| int nanos = (int) ((timestamp - seconds) * 1_000_000_000); | ||
| Instant instant = Instant.ofEpochSecond(seconds, nanos); | ||
| ZonedDateTime zdt = ZonedDateTime.ofInstant(instant, ZoneId.of("UTC")); | ||
| return StrftimeFormatterUtil.formatZonedDateTime(zdt, format).stringValue(); | ||
| } catch (Exception e) { | ||
| return null; | ||
| } | ||
| } | ||
|
|
||
| public static Double toEpochSeconds(Object value) { | ||
| if (value == null) { | ||
| return null; | ||
| } | ||
| if (value instanceof Number) { | ||
| return ((Number) value).doubleValue(); | ||
| } | ||
| String str = value.toString().trim(); | ||
| if (str.isEmpty()) { | ||
| return null; | ||
| } | ||
| try { | ||
| return Double.parseDouble(str); | ||
| } catch (NumberFormatException e) { | ||
| return null; | ||
| } | ||
| } | ||
| } |
55 changes: 55 additions & 0 deletions
55
core/src/main/java/org/opensearch/sql/expression/function/udf/Dur2SecConvertFunction.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| /* | ||
| * Copyright OpenSearch Contributors | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| package org.opensearch.sql.expression.function.udf; | ||
|
|
||
| import java.util.regex.Matcher; | ||
| import java.util.regex.Pattern; | ||
|
|
||
| /** PPL dur2sec() conversion function. Converts duration format {@code [D+]HH:MM:SS} to seconds */ | ||
| public class Dur2SecConvertFunction extends BaseConversionUDF { | ||
|
|
||
| public static final Dur2SecConvertFunction INSTANCE = new Dur2SecConvertFunction(); | ||
|
|
||
| // Matches [D+]HH:MM:SS — optional days prefix with + separator | ||
| private static final Pattern DURATION_PATTERN = | ||
| Pattern.compile("^(?:(\\d+)\\+)?(\\d{1,2}):(\\d{1,2}):(\\d{1,2})$"); | ||
|
|
||
| public Dur2SecConvertFunction() { | ||
| super(Dur2SecConvertFunction.class); | ||
| } | ||
|
|
||
| public static Object convert(Object value) { | ||
| return INSTANCE.convertValue(value); | ||
| } | ||
|
|
||
| @Override | ||
| protected Object applyConversion(String preprocessedValue) { | ||
| Double existingSeconds = tryParseDouble(preprocessedValue); | ||
| if (existingSeconds != null) { | ||
| return existingSeconds; | ||
| } | ||
|
|
||
| Matcher matcher = DURATION_PATTERN.matcher(preprocessedValue); | ||
| if (!matcher.matches()) { | ||
| return null; | ||
| } | ||
|
|
||
| try { | ||
| int days = matcher.group(1) != null ? Integer.parseInt(matcher.group(1)) : 0; | ||
| int hours = Integer.parseInt(matcher.group(2)); | ||
| int minutes = Integer.parseInt(matcher.group(3)); | ||
| int seconds = Integer.parseInt(matcher.group(4)); | ||
|
|
||
| if (hours >= 24 || minutes >= 60 || seconds >= 60) { | ||
| return null; | ||
| } | ||
|
|
||
| return (double) (days * 86400 + hours * 3600 + minutes * 60 + seconds); | ||
| } catch (NumberFormatException e) { | ||
| return null; | ||
| } | ||
| } | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All these have to a UDF?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I couldn't find any Calcite operators that handled parsing time the formats that was needed for these functions