Add float16 (half-precision) vector support #117
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit adds comprehensive support for float16 (F16) vectors to both MySQL and HTTP clients in the SingleStoreDB Python SDK. Changes: - Add FLOAT16 = 7 as the 7th vector type constant - Add FIELD_TYPE constants: FLOAT16_VECTOR_JSON (2007) and FLOAT16_VECTOR (3007) - Update protocol parser to recognize and handle FLOAT16 vector metadata - Add float16_vector_json_or_none() and float16_vector_or_none() converters - Register float16 converters in the converters dictionary (types 2007 and 3007) - Add FLOAT16 vector types to TEXT_TYPES set for proper type handling - Update C accelerator with full float16 support: - Add FLOAT16 constants and PyStrings struct member - Update type arrays with 'e' format (2 bytes) for struct.unpack - Add float16 to JSON and binary vector case statements - Initialize numpy dtype kwargs for float16 - Add comprehensive tests: - Create f16_vectors test table with 3 test rows - Implement test_f16_vectors() method following existing patterns - Use assert_array_almost_equal with decimal=2 for float16 precision Technical notes: - Float16 has ~3 decimal digits of precision (vs ~7 for float32) - Uses struct format 'e' for half-precision (2 bytes per element) - Supports both JSON and binary wire formats - All pre-commit hooks passed (flake8, autopep8, mypy) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Extends the UDF framework to handle float16/half-precision types: - Add F16 vector type constant and export - Map float16 to FLOAT SQL type in signatures - Add Float16Array/HalfArray type aliases - Implement F16 numpy and struct format conversions - Add comprehensive float16 UDF tests This complements protocol-level float16 vector support (5377083). Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
kesmit13
requested review from
Copilot
and removed request for
srinathnarayanan
There was a problem hiding this comment.
Pull request overview
This PR adds comprehensive support for float16 (F16/half-precision) vectors to the SingleStoreDB Python SDK, extending the existing vector type support to include 16-bit floating-point values with approximately 3 decimal digits of precision.
Key changes:
- Added FLOAT16 vector type constant and field type constants (2007 for JSON, 3007 for binary)
- Implemented float16 converters for both JSON and binary wire formats using struct format 'e' (2 bytes per element)
- Extended C accelerator with full float16 support including proper array indexing and numpy dtype initialization
- Added comprehensive test coverage with appropriate precision tolerances (decimal=2 for ~3 digit precision)
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| singlestoredb/mysql/constants/VECTOR_TYPE.py | Added FLOAT16 = 7 constant for the new vector type |
| singlestoredb/mysql/constants/FIELD_TYPE.py | Added FLOAT16_VECTOR_JSON (2007) and FLOAT16_VECTOR (3007) field type constants |
| singlestoredb/mysql/protocol.py | Extended protocol parser to recognize and handle FLOAT16 vector metadata |
| singlestoredb/mysql/connection.py | Added FLOAT16 vector types to TEXT_TYPES set for proper type handling |
| singlestoredb/converters.py | Implemented float16_vector_json_or_none() and float16_vector_or_none() converters and registered them for types 2007 and 3007 |
| accel.c | Updated C accelerator with FLOAT16 constants, extended type arrays with 'e' format (2 bytes), added case statements, and initialized numpy dtype |
| singlestoredb/functions/utils.py | Added F16 to VectorTypes enum and implemented conversion functions using 'f2' numpy type and 'e' struct format |
| singlestoredb/functions/signature.py | Added float16 to FLOAT SQL type mapping |
| singlestoredb/functions/init.py | Exported F16 constant for public API |
| singlestoredb/functions/typing/numpy.py | Added Float16Array and HalfArray type aliases |
| singlestoredb/tests/test.sql | Created f16_vectors test table with 3 test rows containing appropriate float16 values |
| singlestoredb/tests/test_connection.py | Implemented test_f16_vectors() method with decimal=2 precision for float16 accuracy |
| singlestoredb/tests/test_udf.py | Added float16 UDF signature tests for scalar and vector types |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This commit implements a generalized version-based SQL loading system for tests and adds server version checks for float16 vector tests. Changes: - Add server version check to test_f16_vectors() that skips the test if server version < 9.1 with a descriptive message - Create test_9_1.sql containing float16 vector test data - Implement generalized version-specific SQL file loading in utils.py: - get_server_version(): Extract server version as (major, minor) tuple - find_version_specific_sql_files(): Discover test_X_Y.sql files - load_version_specific_sql(): Conditionally load SQL based on version - Update load_sql() to automatically load version-specific SQL files - Remove f16_vectors table from test.sql (now in test_9_1.sql) The new system automatically discovers and loads SQL files matching the pattern test_X_Y.sql where X is major version and Y is minor version. Files are loaded only if the server version >= the file version. This makes it easy to add version-specific test data in the future (e.g., test_9_2.sql, test_10_0.sql) without modifying Python code. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit adds comprehensive support for float16 (F16) vectors to both MySQL and HTTP clients in the SingleStoreDB Python SDK.
Changes:
Technical notes: