[FEATURE] Business Ontology Layer on top of the Semantic Model

[prinkanintugle]

Feature Description

Business Ontology Layer on top of the Semantic Model (Microsoft CDM-first)

Introduce a new business ontology layer that sits on top of the existing semantic data model (SemanticModel) generated by intugle. This layer will allow users to:

1. Group and abstract the semantic model into higher-level business domains (e.g., Customer, Sales, Finance, Product).
2. Map elements of the semantic model (entities, attributes, relationships) to Microsoft Common Data Model (CDM) entities and attributes.

The goal is to provide a clear separation between:

• The technical/semantic data ontology (SemanticModel), and
• The business-facing ontology that aligns with Microsoft CDM concepts and domain terminology.

Future extensions could add support for other ontologies (e.g., FIBO), but the initial focus is Microsoft CDM.

Problem Statement

The current semantic model in intugle gives a strong representation of tables, entities, attributes, and relationships at the data level, but:

• Business stakeholders think in terms of domains (Customer, Order, Product, Invoice) and standard business entities (e.g., CDM Account, Contact, SalesOrder).
• Many teams are adopting Microsoft CDM as a canonical model to align data across applications, Power Platform, Dynamics, Fabric, and other services.

Without a dedicated business ontology / CDM mapping layer:

• Mappings to CDM are often managed in spreadsheets or ad-hoc documentation.
• There is no single consistent place in intugle to represent how local semantic entities align with CDM entities and attributes.
• Domain classification and governance across multiple projects is harder, and reuse of mappings is limited.

This feature would enable:

• Explicit, versioned mappings between the local semantic model and Microsoft CDM.
• A clean abstraction for business domains and CDM-aligned concepts on top of the semantic model.

Proposed Solution

Add a Business Ontology Layer dedicated to domain + CDM mappings on top of the existing SemanticModel with the following capabilities:

1. Business Domains & Concepts

• Define Business Domains (e.g., CustomerDomain, SalesDomain, ProductDomain).
• Define Business Concepts that represent business entities:
  • Example: Customer, Account, SalesOrder, InvoiceLine.
• Each concept can:
  • Belong to a domain.
  • Reference a CDM entity (and optionally attributes) via stable identifiers (e.g., cdm:Account, cdm:SalesOrder).

2. Mapping Layer: Semantic Model → CDM Concepts

• Provide a mapping engine that links:
  • SemanticModel entities (tables/objects) → Business Concepts → CDM Entities.
  • SemanticModel attributes (columns/fields) → CDM attributes.
• Support mapping patterns such as:
  • One semantic entity → one CDM entity.
  • Multiple semantic entities → one CDM entity (e.g., header/detail split).
  • Composite mappings (e.g., building CDM attributes from multiple fields).

Examples:

• semantic.customer → Business Concept Customer → CDM Contact
• semantic.account → Business Concept Account → CDM Account
• semantic.sales_order_header + semantic.sales_order_line → Business Concept SalesOrder → CDM SalesOrder

3. CDM Templates / Catalog

• Provide a CDM catalog or template within intugle containing:
  • Canonical CDM entities (names, IDs).
  • Core attributes for each entity.
• The CDM catalog can be:
  • Used directly when defining Business Concepts.
  • Queried to see which CDM entities are already in use / mapped.

4. Governance & Metadata

• Represent business ontology and mappings in a structured, versionable format (e.g., JSON/YAML):
  • business_ontology_cdm.json
  • semantic_to_cdm_mappings.yaml
• Capture metadata per mapping:
  • Status (proposed, approved, deprecated).
  • Domain owner / steward.
  • Notes or references (e.g., link to CDM docs, internal standards).
• Enable queries like:
  • “Which semantic entities are mapped to CDM Account?”
  • “Which CDM entities are currently unmapped?”

5. APIs & Possible CLI Support

• Python API (illustrative):

  • BusinessOntology
  • BusinessConcept
  • CDMCatalog
  • OntologyMapper or similar mapping class.

• CLI ideas:

  • intugle cdm init-ontology
  • intugle cdm map --semantic-model semantic_model.json --cdm-entity Account

• Integration with SemanticModel:

  • Ability to navigate semantic_model via business domains and CDM entities, not only technical names.

Use Case

Domain: General enterprise analytics with Microsoft-centered ecosystem (Power Platform, Fabric, Dynamics, etc.)

Workflow:

1. A SemanticModel is generated from existing data sources (warehouse tables, NoSQL-to-relational outputs, etc.).
2. The organization wants to align these local semantics with Microsoft CDM so that:
   • Data products can be more easily integrated with CDM-aware tools.
   • Business users can reason about data in CDM terms.
3. Using the new feature:
   • A data/ontology architect defines business domains (Customer, Sales, Product).
   • They create Business Concepts (e.g., Customer, Account, SalesOrder) and link them to CDM entities.
   • They map SemanticModel entities/columns to these concepts and CDM attributes.
4. Downstream:
   • Governance and cataloging tools can report which parts of the semantic model conform to CDM.
   • Data products can expose CDM-conformant contracts.
   • Documentation and semantic search can be driven by CDM and domain vocabulary, not just table/column names.

Alternative Solutions

• Maintain CDM mappings in spreadsheets or separate documentation.
  • Hard to keep in sync with the actual semantic model.
  • No programmatic enforcement or validation.
• Use external ontology/knowledge graph tools for CDM alignment.
  • Powerful, but adds another stack; integration back into intugle is custom.
• Add simple labels or tags to entities (e.g., “this is an Account”).
  • Too weak to express rich mappings and attribute-level alignment.

The proposed feature provides first-class CDM support directly inside intugle, reusing the existing semantic model and metadata framework.

Additional Context

This feature would be a key building block for:

• CDM-aligned data products.
• More meaningful business glossaries and catalogs.
• Semantic search over concepts like “Customer” or “Account” rather than technical table names.
• Once CDM is supported robustly, the same pattern could later be extended to:
• Other standard ontologies (e.g., FIBO).
• Organization-specific ontologies on top of CDM.

Examples

(API names and modules are illustrative; final design should match the existing intugle public API pattern, e.g., from intugle import SemanticModel.)

from intugle import SemanticModel, BusinessOntology, CDMCatalog, OntologyMapper

# 1. Load the existing semantic data model
semantic_model = SemanticModel.load("semantic_model.json")

# 2. Load or initialize the Microsoft CDM catalog
cdm_catalog = CDMCatalog.load_builtin("cdm_core")  # e.g., ships with intugle

# 3. Create / load a business ontology
business_ontology = BusinessOntology(name="Enterprise Business Ontology (CDM)")

# Define domains
business_ontology.add_domain(
    name="CustomerDomain",
    description="All customer and account-related concepts"
)
business_ontology.add_domain(
    name="SalesDomain",
    description="Sales orders, invoices, and related concepts"
)

# Define business concepts linked to CDM entities
customer_concept = business_ontology.add_concept(
    name="Customer",
    domain="CustomerDomain",
    cdm_entity=cdm_catalog.get_entity("Contact"),
)

account_concept = business_ontology.add_concept(
    name="Account",
    domain="CustomerDomain",
    cdm_entity=cdm_catalog.get_entity("Account"),
)

sales_order_concept = business_ontology.add_concept(
    name="SalesOrder",
    domain="SalesDomain",
    cdm_entity=cdm_catalog.get_entity("SalesOrder"),
)

# 4. Map semantic entities to business concepts / CDM
mapper = OntologyMapper(semantic_model, business_ontology, cdm_catalog)

mapper.map_entity(
    semantic_entity="customer",
    concept="Customer",
    attribute_map={
        "customer_id": "Contact.ContactId",
        "email": "Contact.Email",
        "full_name": "Contact.FullName",
    },
)

mapper.map_entity(
    semantic_entity="account",
    concept="Account",
    attribute_map={
        "account_id": "Account.AccountId",
        "account_name": "Account.Name",
        "account_balance": "Account.Balance",
    },
)

mapper.map_entity(
    semantic_entity="sales_order_header",
    concept="SalesOrder",
    attribute_map={
        "order_id": "SalesOrder.SalesOrderId",
        "order_date": "SalesOrder.OrderDate",
        "customer_id": "SalesOrder.CustomerId",
    },
)

# 5. Save ontology + mappings
business_ontology.save("business_ontology_cdm.json")
mapper.export_mappings("semantic_to_cdm_mappings.json")