VariantGet RFC

Signed-off-by: Adam Gutglick <adam@spiraldb.com>

Author: AdamGS

rfcs/0057-variant-get-expr.md

@@ -0,0 +1,109 @@
+- Start Date: 2026-05-05
+- Authors: @AdamGS
+- RFC PR: [vortex-data/rfcs#57](https://github.com/vortex-data/rfcs/pull/57)
+
+# VariantGet Expression
+
+## Summary
+
+Introduce a new `VariantGet` expression that extracts useable data from variant arrays.
+
+## Motivation
+
+As described in the [Variant RFC](https://github.com/vortex-data/rfcs/blob/develop/rfcs/0015-variant-type.md),
+variants arrays are useful for many use cases, but in order to actually use the data a fully typed array is required.
+
+## Design
+
+### Definition
+
+A new VariantGet expression is required, the expression has two inputs:
+
+1. Path to the required child - similar to JSONPath, combination of names and indexes.
+2. Optional dtype, if None - the return type is `None`, the expression's return type is `Variant`.
+
+### Array
+
+The canonical Variant array will add an additional child, representing optional shredded data, it will now have:
+
+1. Validity
+2. Core storage - containing the raw unshredded data, which can be encoded in any way the child array's encoding.
+3. A shredded child.
+
+The "core storage" child may also have its own shredded child.
+
+### Execution
+
+When executing the expression on a variant array, it will pull out recursively shredded data until the path is exhausted OR the path reached a child path that isn't shredded. As we traverse the chain of shredded children along the path, we'll need to make sure to keep track of their validity, as the leaf child's validity is an OR of all of them.
+
+At this point, we have 3 possible cases:
+
+1. Perfectly shredded - there's a fully shredded child at this path. If it matches the expected type or can be casted into it, we can just return it.
+2. Partially shredded - data for this path exists in both the shredded child AND in some unshredded values, which we can merge according to the expected type.
+3. Unshredded - No shredded child at this path, we try and extract the relevant value from the unshredded values which are unchanged from the original array.
+
+The important invariant is that `VariantGet` changes the typed child selected for the requested
+path, but it does not rewrite the raw unshredded data. The raw storage continues to represent the
+same original variant values and can still be used by later `VariantGet` expressions for paths that
+were not shredded.
+
+```text
+Variant array before VariantGet("$.a.b", i64)
+
++--------------------------------------------------------------+
+| validity                                                     |
+| raw unshredded data --------------------------------------+  |
+| shredded children                                        |  |
+|   $.a.b: utf8 / missing / partially materialized         |  |
+|   $.x.y: bool                                            |  |
++----------------------------------------------------------|---+
+                                                           |
+VariantGet("$.a.b", i64)                                  | unchanged
+                                                           |
++----------------------------------------------------------|---+
+| validity for rows where $.a.b can be read as i64         |  |
+| raw unshredded data <------------------------------------+  |
+| typed child: i64 values for $.a.b                           |
+|   built from shredded data, raw data, or a merge of both     |
++--------------------------------------------------------------+
+```
+
+## Compatibility
+
+TODO: Explain compatibility concerns.
+
+- Does this change the file format or wire format? Is it backward or forward compatible?
+- Does this break any public APIs? If so, what is the migration path?
+- Are there performance implications?
+
+If there are no compatibility concerns, briefly state why.
+
+## Drawbacks
+
+TODO: Explain the cost of this change.
+
+- Why should we not do this?
+- What is the maintenance cost?
+- Does this add complexity that could be avoided?
+
+## Alternatives
+
+We can make the dtype parameter required, but I do think that the optional one keeps execution more flexible and opens up
+opportunities for different usage, which is useful for compute engines that have more flexible type systems or that might want
+to process the raw byte data themselves.
+
+## Prior Art
+
+See the [Variant RFC](https://github.com/vortex-data/rfcs/blob/develop/rfcs/0015-variant-type.md).
+
+## Unresolved Questions
+
+TODO: List open questions for the RFC process.
+
+- What parts of the design still need to be resolved?
+- What is explicitly out of scope?
+- What can be deferred to implementation?
+
+## Future Possibilities
+
+TODO: Capture natural extensions or follow-on work that are out of scope for this RFC.