Self-Referential FK Ordering Enhancement

[bjester]

Target branch: release-0.9.x

Current behavior

The current deserialization behavior recurses into 'children' models when it encounters a model with a self-referential foreign key (FK). This contributes to its monolithic nature and its memory consumption by requiring the algorithm to track more data in memory to ensure comprehensive deserialization of store data.

The serialization process tracks the FK value for self-referential models in a field called _self_ref_fk. This allows it to lookup those FK records using that value.

Additionally, the Morango registry, which tracks the syncable models defined by any apps using Morango, orders them according to their cross-model FK dependencies.

Desired behavior

The deserialization algorithm would be more amenable to the streaming architecture by ordering the store records more intelligently before they enter the pipeline. Combined with the existing ordering by the Morango registry of syncable models, this should create a predictable order in which store records can be properly deserialized without broken FKs.

Enhancements

Data Model

This specifically addresses self-referential FK ordering within the same model (e.g., parent-child relationships in hierarchical data). The registry handles cross-model dependencies, but self-referential FKs need special ordering.

• A new field _self_ref_order should be added to the AbstractStore with type of Integer
• The new field should default to 0
• Django model migrations should be created to add the new field to Store and Buffer
• The SyncableModel should have a new tuple attribute morango_ordering_fields which should accept one or more field expressions in the format that Django uses for order_by()
• The SyncableModelRegistry.get_model_querysets should order its querysets according to morango_ordering_fields if set

Serialization

Update StoreUpdate transform in serialize.py to set _self_ref_order during Store record creation and updates.

• The _handle_store_create method should be updated:
  • if _self_ref_fk is None or the model is not self-referential, the order should be 0
  • otherwise, it should query for the parent's order (the Store._self_ref_order for id=_self_ref_fk)
• The _handle_store_create method should be updated:
  • to detect if the _self_ref_fk has changed, and if so, update the order like _handle_store_create

Value add

Provides more certainty to the pipeline that we process store records in the order of their relational dependency in the DB.

Possible tradeoffs

AI usage