Add extended table metadata retrieval and models for columns and option sets

.azdo/ci-pr.yaml

@@ -42,7 +42,7 @@ extends:
 
         - script: |
             python -m pip install --upgrade pip
-            python -m pip install flake8 black build
+            python -m pip install flake8 black build diff-cover
             python -m pip install -e .[dev]
           displayName: 'Install dependencies'
 
@@ -60,18 +60,30 @@ extends:
         - script: |
             python -m build
           displayName: 'Build package'
-        
+
         - script: |
             python -m pip install dist/*.whl
           displayName: 'Install wheel'
-        
+
         - script: |
-            pytest
+            PYTHONPATH=src pytest --junitxml=test-results.xml --cov --cov-report=xml
           displayName: 'Test with pytest'
-
+
+        - script: |
+            git fetch origin main
+            diff-cover coverage.xml --compare-branch=origin/main --fail-under=90
+          displayName: 'Diff coverage (90% for new changes)'
+
         - task: PublishTestResults@2
           condition: succeededOrFailed()
           inputs:
-            testResultsFiles: '**/test-*.xml'
+            testResultsFiles: '**/test-results.xml'
             testRunTitle: 'Python 3.12'
           displayName: 'Publish test results'
+
+        - task: PublishCodeCoverageResults@2
+          condition: succeededOrFailed()
+          inputs:
+            summaryFileLocation: '**/coverage.xml'
+            pathToSources: '$(Build.SourcesDirectory)/src'
+          displayName: 'Publish code coverage'

.claude/skills/dataverse-sdk-dev/SKILL.md

@@ -13,13 +13,39 @@ This skill provides guidance for developers working on the PowerPlatform Dataver
 
 ### API Design
 
-1. **Public methods in operation namespaces** - New public methods go in the appropriate namespace module under `src/PowerPlatform/Dataverse/operations/` (`records.py`, `query.py`, `tables.py`). The `client.py` file exposes these via namespace properties (`client.records`, `client.query`, `client.tables`). Public types and constants live in their own modules (e.g., `models/metadata.py`, `common/constants.py`)
+1. **Public methods in operation namespaces** - New public methods go in the appropriate namespace module under `src/PowerPlatform/Dataverse/operations/` (`records.py`, `query.py`, `tables.py`, `batch.py`). The `client.py` file exposes these via namespace properties (`client.records`, `client.query`, `client.tables`, `client.batch`). Public types and constants live in their own modules (e.g., `models/metadata.py`, `models/batch.py`, `common/constants.py`)
 2. **Every public method needs README example** - Public API methods must have examples in README.md
 3. **Reuse existing APIs** - Always check if an existing method can be used before making direct Web API calls
 4. **Update documentation** when adding features - Keep README and SKILL files (both copies) in sync
 5. **Consider backwards compatibility** - Avoid breaking changes
 6. **Internal vs public naming** - Modules, files, and functions not meant to be part of the public API must use a `_` prefix (e.g., `_odata.py`, `_relationships.py`). Files without the prefix (e.g., `constants.py`, `metadata.py`) are public and importable by SDK consumers
 
+### Dataverse Property Naming Rules
+
+Dataverse uses two different naming conventions for properties. Getting this wrong causes 400 errors that are hard to debug.
+
+| Property type | Name convention | Example | When used |
+|---|---|---|---|
+| **Structural** (columns) | LogicalName (always lowercase) | `new_name`, `new_priority` | `$select`, `$filter`, `$orderby`, record payload keys |
+| **Navigation** (relationships / lookups) | Navigation Property Name (usually SchemaName, PascalCase, case-sensitive) | `new_CustomerId`, `new_AgentId` | `$expand`, `@odata.bind` annotation keys |
+
+Navigation property names are case-sensitive and must match the entity's `$metadata`. Using the logical name instead of the navigation property name results in 400 Bad Request errors.
+
+**Critical rule:** The OData parser validates `@odata.bind` property names **case-sensitively** against declared navigation properties. Lowercasing `new_CustomerId@odata.bind` to `new_customerid@odata.bind` causes: `ODataException: An undeclared property 'new_customerid' which only has property annotations...`
+
+**SDK implementation:**
+
+- `_lowercase_keys()` lowercases all keys EXCEPT those containing `@odata.` (preserves navigation property casing in `@odata.bind` keys)
+- `_lowercase_list()` lowercases `$select` and `$orderby` params (structural properties)
+- `$expand` params are passed as-is (navigation properties, PascalCase)
+- `_convert_labels_to_ints()` skips `@odata.` keys entirely (they are annotations, not attributes)
+
+**When adding new code that processes record dicts or builds query parameters:**
+
+- Always use `_lowercase_keys()` for record payloads. Never manually call `.lower()` on all keys
+- Never lowercase `$expand` values or `@odata.bind` key prefixes
+- If iterating record keys, skip keys containing `@odata.` when doing attribute-level operations
+
 ### Code Style
 
 6. **No emojis** - Do not use emoji in code, comments, or output
@@ -28,3 +54,35 @@ This skill provides guidance for developers working on the PowerPlatform Dataver
 9. **Document public APIs** - Add Sphinx-style docstrings with examples for public methods
 10. **Define __all__ in module files** - Each module declares its own exports via `__all__` (e.g., `errors.py` defines `__all__ = ["HttpError", ...]`). Package `__init__.py` files should not re-export or redefine another module's `__all__`; they use `__all__ = []` to indicate no star-import exports.
 11. **Run black before committing** - Always run `python -m black <changed files>` before committing. CI will reject unformatted code. Config is in `pyproject.toml` under `[tool.black]`.
+
+### Docstring Type Annotations (Microsoft Learn Compatibility)
+
+This SDK's API reference is published on Microsoft Learn. The Learn doc pipeline parses `:type:` and `:rtype:` directives differently from standard Sphinx -- every word between `:class:` references is treated as a separate cross-reference (`<xref:word>`). Using Sphinx-style `:class:\`list\` of :class:\`str\`` produces broken `<xref:of>` links on Learn.
+
+**Rules for `:type:` and `:rtype:` directives:**
+
+- Use Python bracket notation for generic types: `list[str]`, `dict[str, typing.Any]`, `list[dict]`
+- Use `or` (without `:class:`) for union types: `str or None`, `dict or list[dict]`
+- Use bracket nesting for complex types: `collections.abc.Iterable[list[dict]]`
+- Use `~` prefix for SDK types to show short name: `list[~PowerPlatform.Dataverse.models.record.Record]`
+- `:class:` is fine for single standalone types: `:class:\`str\``, `:class:\`bool\``
+
+**Never** use `:class:\`X\` of :class:\`Y\`` or `:class:\`X\` mapping :class:\`Y\` to :class:\`Z\`` -- the words `of`, `mapping`, `to` become broken `<xref:>` links.
+
+**Correct examples:**
+
+```rst
+:type data: dict or list[dict]
+:rtype: list[str]
+:rtype: collections.abc.Iterable[list[~PowerPlatform.Dataverse.models.record.Record]]
+:type select: list[str] or None
+:type columns: dict[str, typing.Any]
+```
+
+**Wrong examples (NEVER use):**
+
+```rst
+:type data: :class:`dict` or :class:`list` of :class:`dict`
+:rtype: :class:`list` of :class:`str`
+:type columns: :class:`dict` mapping :class:`str` to :class:`typing.Any`
+```

.claude/skills/dataverse-sdk-use/SKILL.md

@@ -22,6 +22,7 @@ Use the PowerPlatform Dataverse Client Python SDK to interact with Microsoft Dat
 - `client.query` -- query and search operations
 - `client.tables` -- table metadata, columns, and relationships
 - `client.files` -- file upload operations
+- `client.batch` -- batch multiple operations into a single HTTP request
 
 ### Bulk Operations
 The SDK supports Dataverse's native bulk operations: Pass lists to `create()`, `update()` for automatic bulk processing, for `delete()`, set `use_bulk_delete` when passing lists to use bulk operation
@@ -30,6 +31,9 @@ The SDK supports Dataverse's native bulk operations: Pass lists to `create()`, `
 - Control page size with `page_size` parameter
 - Use `top` parameter to limit total records returned
 
+### DataFrame Support
+- DataFrame operations are accessed via the `client.dataframe` namespace: `client.dataframe.get()`, `client.dataframe.create()`, `client.dataframe.update()`, `client.dataframe.delete()`
+
 ## Common Operations
 
 ### Import
@@ -105,6 +109,20 @@ for page in client.records.get(
         print(f"{account['name']} - {contact.get('fullname', 'N/A')}")
 ```
 
+#### Create Records with Lookup Bindings (@odata.bind)
+```python
+# Set lookup fields using @odata.bind with PascalCase navigation property names
+# CORRECT: use the navigation property name (case-sensitive, must match $metadata)
+guid = client.records.create("new_ticket", {
+    "new_name": "TKT-001",
+    "new_CustomerId@odata.bind": f"/new_customers({customer_id})",
+    "new_AgentId@odata.bind": f"/new_agents({agent_id})",
+})
+
+# WRONG: lowercase navigation property causes 400 error
+# "new_customerid@odata.bind" -> ODataException: undeclared property 'new_customerid'
+```
+
 #### Update Records
 ```python
 # Single update
@@ -115,7 +133,7 @@ client.records.update("account", [id1, id2, id3], {"industry": "Technology"})
 ```
 
 #### Upsert Records
-Creates or updates records identified by alternate keys. Single item → PATCH; multiple items → `UpsertMultiple` bulk action.
+Creates or updates records identified by alternate keys. Single item -> PATCH; multiple items -> `UpsertMultiple` bulk action.
 > **Prerequisite**: The table must have an alternate key configured in Dataverse for the columns used in `alternate_key`. Without it, Dataverse will reject the request with a 400 error.
 ```python
 from PowerPlatform.Dataverse.models.upsert import UpsertItem
@@ -157,6 +175,42 @@ client.records.delete("account", account_id)
 client.records.delete("account", [id1, id2, id3], use_bulk_delete=True)
 ```
 
+### DataFrame Operations
+
+The SDK provides DataFrame wrappers for all CRUD operations via the `client.dataframe` namespace, using pandas DataFrames and Series as input/output.
+
+```python
+import pandas as pd
+
+# Query records -- returns a single DataFrame
+df = client.dataframe.get("account", filter="statecode eq 0", select=["name"])
+print(f"Got {len(df)} rows")
+
+# Limit results with top for large tables
+df = client.dataframe.get("account", select=["name"], top=100)
+
+# Fetch single record as one-row DataFrame
+df = client.dataframe.get("account", record_id=account_id, select=["name"])
+
+# Create records from a DataFrame (returns a Series of GUIDs)
+new_accounts = pd.DataFrame([
+    {"name": "Contoso", "telephone1": "555-0100"},
+    {"name": "Fabrikam", "telephone1": "555-0200"},
+])
+new_accounts["accountid"] = client.dataframe.create("account", new_accounts)
+
+# Update records from a DataFrame (id_column identifies the GUID column)
+new_accounts["telephone1"] = ["555-0199", "555-0299"]
+client.dataframe.update("account", new_accounts, id_column="accountid")
+
+# Clear a field by setting clear_nulls=True (by default, NaN/None fields are skipped)
+df = pd.DataFrame([{"accountid": "guid-1", "websiteurl": None}])
+client.dataframe.update("account", df, id_column="accountid", clear_nulls=True)
+
+# Delete records by passing a Series of GUIDs
+client.dataframe.delete("account", new_accounts["accountid"])
+```
+
 ### SQL Queries
 
 SQL queries are **read-only** and support limited SQL syntax. A single SELECT statement with optional WHERE, TOP (integer literal), ORDER BY (column names only), and a simple table alias after FROM is supported. But JOIN and subqueries may not be. Refer to the Dataverse documentation for the current feature set.
@@ -196,6 +250,7 @@ table_info = client.tables.create(
 #### Supported Column Types
 Types on the same line map to the same exact format under the hood
 - `"string"` or `"text"` - Single line of text
+- `"memo"` or `"multiline"` - Multiple lines of text (4000 character default)
 - `"int"` or `"integer"` - Whole number
 - `"decimal"` or `"money"` - Decimal number
 - `"float"` or `"double"` - Floating point number
@@ -229,6 +284,63 @@ for table in tables:
     print(table)
 ```
 
+#### Get Extended Table Metadata
+```python
+# Get table with column metadata
+info = client.tables.get("account", include_columns=True)
+for col in info["columns"]:
+    print(f"{col.logical_name} ({col.attribute_type})")
+
+# Get table with relationship metadata
+info = client.tables.get("account", include_relationships=True)
+
+# Get specific entity properties
+info = client.tables.get("account", select=["DisplayName", "Description"])
+```
+
+#### List Columns
+```python
+from PowerPlatform.Dataverse.models.table_info import ColumnInfo
+
+columns = client.tables.get_columns("account")
+for col in columns:
+    print(f"{col.schema_name}: {col.attribute_type} (required: {col.required_level})")
+
+# Filter to specific column types (OData syntax, fully-qualified enum)
+picklists = client.tables.get_columns(
+    "account",
+    filter="AttributeType eq Microsoft.Dynamics.CRM.AttributeTypeCode'Picklist'",
+)
+```
+
+#### Get Single Column
+```python
+col = client.tables.get_column("account", "emailaddress1")
+if col:
+    print(f"Type: {col.attribute_type}, Required: {col.required_level}")
+```
+
+#### Get Column Options (Picklist/Choice Values)
+```python
+from PowerPlatform.Dataverse.models.table_info import OptionSetInfo
+
+options = client.tables.get_column_options("account", "accountcategorycode")
+if options:
+    for opt in options.options:
+        print(f"  Value={opt.value}, Label={opt.label}")
+```
+
+#### List Table Relationships
+```python
+# All relationships
+rels = client.tables.list_relationships("account")
+
+# Specific type: "one_to_many" / "1:N", "many_to_one" / "N:1", "many_to_many" / "N:N"
+rels = client.tables.list_relationships("account", relationship_type="one_to_many")
+for rel in rels:
+    print(f"{rel['SchemaName']}: {rel.get('ReferencingEntity')}")
+```
+
 #### Delete Tables
 ```python
 client.tables.delete("new_Product")
@@ -316,6 +428,50 @@ client.files.upload(
 )
 ```
 
+### Batch Operations
+
+Use `client.batch` to send multiple operations in one HTTP request. All batch methods return `None`; results arrive via `BatchResult` after `execute()`.
+
+```python
+# Build a batch request
+batch = client.batch.new()
+batch.records.create("account", {"name": "Contoso"})
+batch.records.update("account", account_id, {"telephone1": "555-0100"})
+batch.records.get("account", account_id, select=["name"])
+batch.query.sql("SELECT TOP 5 name FROM account")
+
+result = batch.execute()
+for item in result.responses:
+    if item.is_success:
+        print(f"[OK] {item.status_code} entity_id={item.entity_id}")
+        if item.data:
+            # GET responses populate item.data with the parsed JSON record
+            print(item.data.get("name"))
+    else:
+        print(f"[ERR] {item.status_code}: {item.error_message}")
+
+# Transactional changeset (all succeed or roll back)
+with batch.changeset() as cs:
+    ref = cs.records.create("contact", {"firstname": "Alice"})
+    cs.records.update("account", account_id, {"primarycontactid@odata.bind": ref})
+
+# Continue on error
+result = batch.execute(continue_on_error=True)
+print(f"Succeeded: {len(result.succeeded)}, Failed: {len(result.failed)}")
+```
+
+**BatchResult properties:**
+- `result.responses` -- list of `BatchItemResponse` in submission order
+- `result.succeeded` -- responses with 2xx status codes
+- `result.failed` -- responses with non-2xx status codes
+- `result.has_errors` -- True if any response failed
+- `result.entity_ids` -- GUIDs from OData-EntityId headers (creates and updates)
+
+**Batch limitations:**
+- Maximum 1000 operations per batch
+- Paginated `records.get()` (without `record_id`) is not supported in batch
+- `flush_cache()` is not supported in batch
+
 ## Error Handling
 
 The SDK provides structured exceptions with detailed error information:
@@ -359,6 +515,7 @@ except ValidationError as e:
 - Check filter/expand parameters use correct case
 - Verify column names exist and are spelled correctly
 - Ensure custom columns include customization prefix
+- For `@odata.bind` errors ("undeclared property"): the navigation property name before `@odata.bind` is case-sensitive and must match the entity's `$metadata` exactly (e.g., `new_CustomerId@odata.bind` for custom lookups, `parentaccountid@odata.bind` for system lookups). The SDK preserves `@odata.bind` key casing.
 
 ## Best Practices
 
@@ -371,7 +528,7 @@ except ValidationError as e:
 5. **Use production credentials** - ClientSecretCredential or CertificateCredential for unattended operations
 6. **Error handling** - Implement retry logic for transient errors (`e.is_transient`)
 7. **Always include customization prefix** for custom tables/columns
-8. **Use lowercase** - Generally using lowercase input won't go wrong, except for custom table/column naming
+8. **Use lowercase for column names, match `$metadata` for navigation properties** - Column names in `$select`/`$filter`/record payloads use lowercase LogicalNames. Navigation properties in `$expand` and `@odata.bind` keys are case-sensitive and must match the entity's `$metadata` (PascalCase for custom lookups like `new_CustomerId`, lowercase for system lookups like `parentaccountid`)
 9. **Test in non-production environments** first
 10. **Use named constants** - Import cascade behavior constants from `PowerPlatform.Dataverse.common.constants`