-
-
Notifications
You must be signed in to change notification settings - Fork 60
feat(autocomplete): Add boolean key attributes to RPC autocomplete #7642
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
This PR has a migration; here is the generated SQL for -- start migrations
-- forward migration events_analytics_platform : 0051_add_bool_keys_to_autocomplete
Local op: ALTER TABLE eap_item_co_occurring_attrs_1_local ADD COLUMN IF NOT EXISTS attributes_bool Array(String) AFTER attributes_float;
Distributed op: ALTER TABLE eap_item_co_occurring_attrs_1_dist ADD COLUMN IF NOT EXISTS attributes_bool Array(String) AFTER attributes_float;
Local op: CREATE MATERIALIZED VIEW IF NOT EXISTS eap_item_co_occurring_attrs_2_mv TO eap_item_co_occurring_attrs_1_local (organization_id UInt64, project_id UInt64, item_type UInt8, date Date CODEC (DoubleDelta, ZSTD(1)), retention_days UInt16, attribute_keys_hash Array(UInt64) MATERIALIZED arrayMap(k -> cityHash64(k), arrayDistinct(arrayConcat(attributes_string, attributes_float, attributes_bool))), attributes_string Array(String), attributes_float Array(String), attributes_bool Array(String), key_hash UInt64 MATERIALIZED cityHash64(arraySort(arrayDistinct(arrayConcat(attributes_string, attributes_float, attributes_bool))))) AS
SELECT
organization_id AS organization_id,
project_id AS project_id,
item_type as item_type,
toMonday(timestamp) AS date,
retention_days as retention_days,
arrayConcat(mapKeys(attributes_string_0), mapKeys(attributes_string_1), mapKeys(attributes_string_2), mapKeys(attributes_string_3), mapKeys(attributes_string_4), mapKeys(attributes_string_5), mapKeys(attributes_string_6), mapKeys(attributes_string_7), mapKeys(attributes_string_8), mapKeys(attributes_string_9), mapKeys(attributes_string_10), mapKeys(attributes_string_11), mapKeys(attributes_string_12), mapKeys(attributes_string_13), mapKeys(attributes_string_14), mapKeys(attributes_string_15), mapKeys(attributes_string_16), mapKeys(attributes_string_17), mapKeys(attributes_string_18), mapKeys(attributes_string_19), mapKeys(attributes_string_20), mapKeys(attributes_string_21), mapKeys(attributes_string_22), mapKeys(attributes_string_23), mapKeys(attributes_string_24), mapKeys(attributes_string_25), mapKeys(attributes_string_26), mapKeys(attributes_string_27), mapKeys(attributes_string_28), mapKeys(attributes_string_29), mapKeys(attributes_string_30), mapKeys(attributes_string_31), mapKeys(attributes_string_32), mapKeys(attributes_string_33), mapKeys(attributes_string_34), mapKeys(attributes_string_35), mapKeys(attributes_string_36), mapKeys(attributes_string_37), mapKeys(attributes_string_38), mapKeys(attributes_string_39)) AS attributes_string,
mapKeys(attributes_bool) AS attributes_bool,
arrayConcat(mapKeys(attributes_float_0), mapKeys(attributes_float_1), mapKeys(attributes_float_2), mapKeys(attributes_float_3), mapKeys(attributes_float_4), mapKeys(attributes_float_5), mapKeys(attributes_float_6), mapKeys(attributes_float_7), mapKeys(attributes_float_8), mapKeys(attributes_float_9), mapKeys(attributes_float_10), mapKeys(attributes_float_11), mapKeys(attributes_float_12), mapKeys(attributes_float_13), mapKeys(attributes_float_14), mapKeys(attributes_float_15), mapKeys(attributes_float_16), mapKeys(attributes_float_17), mapKeys(attributes_float_18), mapKeys(attributes_float_19), mapKeys(attributes_float_20), mapKeys(attributes_float_21), mapKeys(attributes_float_22), mapKeys(attributes_float_23), mapKeys(attributes_float_24), mapKeys(attributes_float_25), mapKeys(attributes_float_26), mapKeys(attributes_float_27), mapKeys(attributes_float_28), mapKeys(attributes_float_29), mapKeys(attributes_float_30), mapKeys(attributes_float_31), mapKeys(attributes_float_32), mapKeys(attributes_float_33), mapKeys(attributes_float_34), mapKeys(attributes_float_35), mapKeys(attributes_float_36), mapKeys(attributes_float_37), mapKeys(attributes_float_38), mapKeys(attributes_float_39)) AS attributes_float
FROM eap_items_1_local
;
Local op: DROP TABLE IF EXISTS eap_item_co_occurring_attrs_1_mv SYNC;
-- end forward migration events_analytics_platform : 0051_add_bool_keys_to_autocomplete
-- backward migration events_analytics_platform : 0051_add_bool_keys_to_autocomplete
Local op: CREATE MATERIALIZED VIEW IF NOT EXISTS eap_item_co_occurring_attrs_1_mv TO eap_item_co_occurring_attrs_1_local (organization_id UInt64, project_id UInt64, item_type UInt8, date Date CODEC (DoubleDelta, ZSTD(1)), retention_days UInt16, attribute_keys_hash Array(UInt64) MATERIALIZED arrayMap(k -> cityHash64(k), arrayConcat(attributes_string, attributes_float)), attributes_string Array(String), attributes_float Array(String), key_hash UInt64 MATERIALIZED cityHash64(arraySort(arrayConcat(attributes_string, attributes_float)))) AS
SELECT
organization_id AS organization_id,
project_id AS project_id,
item_type as item_type,
toMonday(timestamp) AS date,
retention_days as retention_days,
arrayConcat(mapKeys(attributes_string_0), mapKeys(attributes_string_1), mapKeys(attributes_string_2), mapKeys(attributes_string_3), mapKeys(attributes_string_4), mapKeys(attributes_string_5), mapKeys(attributes_string_6), mapKeys(attributes_string_7), mapKeys(attributes_string_8), mapKeys(attributes_string_9), mapKeys(attributes_string_10), mapKeys(attributes_string_11), mapKeys(attributes_string_12), mapKeys(attributes_string_13), mapKeys(attributes_string_14), mapKeys(attributes_string_15), mapKeys(attributes_string_16), mapKeys(attributes_string_17), mapKeys(attributes_string_18), mapKeys(attributes_string_19), mapKeys(attributes_string_20), mapKeys(attributes_string_21), mapKeys(attributes_string_22), mapKeys(attributes_string_23), mapKeys(attributes_string_24), mapKeys(attributes_string_25), mapKeys(attributes_string_26), mapKeys(attributes_string_27), mapKeys(attributes_string_28), mapKeys(attributes_string_29), mapKeys(attributes_string_30), mapKeys(attributes_string_31), mapKeys(attributes_string_32), mapKeys(attributes_string_33), mapKeys(attributes_string_34), mapKeys(attributes_string_35), mapKeys(attributes_string_36), mapKeys(attributes_string_37), mapKeys(attributes_string_38), mapKeys(attributes_string_39)) AS attributes_string,
arrayConcat(mapKeys(attributes_float_0), mapKeys(attributes_float_1), mapKeys(attributes_float_2), mapKeys(attributes_float_3), mapKeys(attributes_float_4), mapKeys(attributes_float_5), mapKeys(attributes_float_6), mapKeys(attributes_float_7), mapKeys(attributes_float_8), mapKeys(attributes_float_9), mapKeys(attributes_float_10), mapKeys(attributes_float_11), mapKeys(attributes_float_12), mapKeys(attributes_float_13), mapKeys(attributes_float_14), mapKeys(attributes_float_15), mapKeys(attributes_float_16), mapKeys(attributes_float_17), mapKeys(attributes_float_18), mapKeys(attributes_float_19), mapKeys(attributes_float_20), mapKeys(attributes_float_21), mapKeys(attributes_float_22), mapKeys(attributes_float_23), mapKeys(attributes_float_24), mapKeys(attributes_float_25), mapKeys(attributes_float_26), mapKeys(attributes_float_27), mapKeys(attributes_float_28), mapKeys(attributes_float_29), mapKeys(attributes_float_30), mapKeys(attributes_float_31), mapKeys(attributes_float_32), mapKeys(attributes_float_33), mapKeys(attributes_float_34), mapKeys(attributes_float_35), mapKeys(attributes_float_36), mapKeys(attributes_float_37), mapKeys(attributes_float_38), mapKeys(attributes_float_39)) AS attributes_float
FROM eap_items_1_local
;
Local op: DROP TABLE IF EXISTS eap_item_co_occurring_attrs_2_mv SYNC;
Distributed op: ALTER TABLE eap_item_co_occurring_attrs_1_dist DROP COLUMN IF EXISTS attributes_bool;
Local op: ALTER TABLE eap_item_co_occurring_attrs_1_local DROP COLUMN IF EXISTS attributes_bool;
-- end backward migration events_analytics_platform : 0051_add_bool_keys_to_autocomplete |
bd932a4 to
a3abbb9
Compare
snuba/snuba_migrations/events_analytics_platform/0051_add_bool_keys_to_autocomplete.py
Show resolved
Hide resolved
a3fe48c to
50317f0
Compare
...a/datasets/configuration/events_analytics_platform/storages/eap_item_co_occurring_attrs.yaml
Outdated
Show resolved
Hide resolved
f38ca57 to
56a622c
Compare
snuba/snuba_migrations/events_analytics_platform/0051_add_bool_keys_to_autocomplete.py
Outdated
Show resolved
Hide resolved
snuba/snuba_migrations/events_analytics_platform/0051_add_bool_keys_to_autocomplete.py
Outdated
Show resolved
Hide resolved
| toMonday(timestamp) AS date, | ||
| retention_days as retention_days, | ||
| arrayConcat({_attr_str_names}) AS attributes_string, | ||
| mapKeys(attributes_bool) AS attributes_bool, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Bug: The migration adds an attributes_bool column, but the storage schema YAML is not updated, and the endpoint still queries the old attributes_float column for booleans.
Severity: CRITICAL
Suggested Fix
Add the new attributes_bool column to the columns list in snuba/datasets/configuration/events_analytics_platform/storages/eap_item_co_occurring_attrs.yaml. Also, update the PROTO_TYPE_TO_ATTRIBUTE_COLUMN mapping in snuba/protos/common.py to map TYPE_BOOLEAN to "attributes_bool" so the endpoint queries the correct column.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.
Location:
snuba/snuba_migrations/events_analytics_platform/0051_add_bool_keys_to_autocomplete.py#L59
Potential issue: The migration `0051_add_bool_keys_to_autocomplete.py` adds an
`attributes_bool` column to the `eap_item_co_occurring_attrs` table and updates its
materialized view. However, the corresponding storage schema file,
`eap_item_co_occurring_attrs.yaml`, is not updated to include this new column. This will
cause a schema mismatch between the application's definition and the actual database
table, leading to failures during schema loading. Furthermore, the autocomplete endpoint
for boolean attributes (`TYPE_BOOLEAN`) is still configured to query the
`attributes_float` column, but the boolean keys will now reside in the new
`attributes_bool` column, causing autocomplete for these attributes to fail.
Did we get this right? 👍 / 👎 to inform future reviews.
Add support for boolean attribute keys in the autocomplete functionality by:
attributes_boolcolumn to autocomplete tablesattributes_bool