feat: Enhance OData client with support for include annotations and pagination options

- Added `include_annotations` parameter to `_RecordGet` and `_RecordList` classes for OData requests.
- Updated `_BatchClient` to handle new parameters in batch operations.
- Enhanced `_ODataClient` methods to support `include_annotations`, `expand`, `page_size`, and `count` parameters.
- Modified `BatchRecordOperations` to pass new parameters in batch record retrieval and listing methods.
- Updated `RecordOperations` to include new parameters for retrieving and listing records.
- Added unit tests to validate the new functionality for batch operations and record retrieval.
- Implemented migration tool updates to handle changes in method signatures and ensure backward compatibility.

Author: Samson Gebre

.claude/skills/dataverse-sdk-use/SKILL.md

@@ -28,11 +28,12 @@ Use the PowerPlatform Dataverse Client Python SDK to interact with Microsoft Dat
 The SDK supports Dataverse's native bulk operations: Pass lists to `create()`, `update()` for automatic bulk processing, for `delete()`, set `use_bulk_delete` when passing lists to use bulk operation
 
 ### Paging
-- Control page size with `page_size` parameter
+- Control page size with `page_size` parameter on `records.list()`, `records.list_pages()`, or `QueryBuilder.page_size()`
 - Use `top` parameter to limit total records returned
-- Simple streaming: `records.list_pages(table, filter, select)` — yields one `QueryResult` per HTTP page (3 params only; use builder for advanced options)
-- Advanced streaming: `client.query.builder(table)....execute_pages()` — full builder options, one `QueryResult` per page
+- **Preferred**: `client.query.builder(table)....execute_pages()` — composable `where(col(...))` filters, formatted values, expand with nested selects, full pagination control
+- Simple streaming shortcut: `records.list_pages(table, *, filter, select, top, orderby, expand, page_size, count, include_annotations)` — string-based OData filter only, yields one `QueryResult` per page
 - `execute(by_page=True/False)` is **deprecated** and emits `UserWarning`; use `execute_pages()` instead
+- `QueryBuilder.to_dataframe()` is **deprecated**; use `.execute().to_dataframe()` instead
 
 ### QueryResult
 - Returned by `records.list()`, `records.retrieve()`, `execute()`, and each page from `list_pages()` / `execute_pages()`
@@ -98,48 +99,81 @@ contact_ids = client.records.create("contact", contacts)
 # Get single record by ID
 account = client.records.retrieve("account", account_id, select=["name", "telephone1"])
 
-# Query with filter — follows @odata.nextLink automatically (multiple HTTP requests if needed),
-# loads all matching records into memory, returns a single QueryResult.
-# Page size is Dataverse's default (~5000/page); use top to bound total records and round-trips.
-# For very large sets where memory is a concern, use records.list_pages() or execute_pages() instead.
-result = client.records.list(
-    "account",
-    select=["accountid", "name"],      # select is case-insensitive (automatically lowercased)
-    filter="statecode eq 0",           # filter must use lowercase logical names (not transformed)
-    top=100,                           # bounds both total records returned and HTTP round-trips
-)
+# Simple shortcut — use records.list() only for basic filter + select without composable logic.
+# Follows @odata.nextLink automatically and loads all matching records into memory.
+# For filtering, sorting, expansion, or formatted values, prefer client.query.builder() (see below).
+result = client.records.list("account", filter="statecode eq 0", select=["name", "accountid"])
 for record in result:
     print(record["name"])
+```
 
-# Simple streaming — page-by-page (3 params only; use builder for ordering/expand/count)
-for page in client.records.list_pages(
-    "account",
-    select=["accountid", "name"],
-    filter="statecode eq 0",
-):
-    for record in page:
-        print(record["name"])
+#### Query Builder (Preferred for Filtering, Sorting, Expand, Formatted Values)
 
-# Advanced streaming — full builder options, one QueryResult per HTTP page
+Use `client.query.builder()` for any query that goes beyond simple filter + select. It provides composable `where(col(...))` expressions, formatted value support, nested expansion, and streaming — all with a fluent API.
+
+```python
 from PowerPlatform.Dataverse.models.filters import col
+from PowerPlatform.Dataverse.models.query_builder import ExpandOption
+
+# Basic query with composable filter and sort
+result = (client.query.builder("account")
+          .select("accountid", "name", "statecode")
+          .where(col("statecode") == 0)
+          .order_by("name asc")
+          .execute())
+for record in result:
+    print(record["name"])
+
+# Composable filters — AND / OR / NOT using Python operators
+result = (client.query.builder("contact")
+          .select("fullname", "emailaddress1")
+          .where((col("statecode") == 0) & (col("emailaddress1").contains("@contoso.com")))
+          .execute())
+
+# Formatted values — display labels for option sets, currency symbols, etc.
+result = (client.query.builder("account")
+          .select("accountid", "name", "industrycode")
+          .where(col("statecode") == 0)
+          .include_formatted_values()
+          .execute())
+for record in result:
+    label = record.get("industrycode@OData.Community.Display.V1.FormattedValue")
+    print(record["name"], label)
+
+# Navigation property expansion with nested column select
+result = (client.query.builder("account")
+          .select("name")
+          .expand(ExpandOption("primarycontactid").select("fullname", "emailaddress1"))
+          .where(col("statecode") == 0)
+          .execute())
+for record in result:
+    contact = record.get("primarycontactid", {})
+    print(f"{record['name']} - {contact.get('fullname', 'N/A')}")
+
+# Stream large result sets page-by-page (memory-efficient)
 for page in (client.query.builder("account")
              .select("accountid", "name")
              .where(col("statecode") == 0)
-             .page_size(500)           # optional: override Dataverse default page size
+             .order_by("name asc")
+             .page_size(500)
              .execute_pages()):
     for record in page:
         print(record["name"])
 
-# Query with navigation property expansion — use the query builder (records.list() has no expand)
-from PowerPlatform.Dataverse.models.query_builder import ExpandOption
-from PowerPlatform.Dataverse.models.filters import col
-for record in (client.query.builder("account")
-               .select("name")
-               .expand(ExpandOption("primarycontactid").select("fullname"))
-               .where(col("statecode") == 0)
-               .execute()):
-    contact = record.get("primarycontactid", {})
-    print(f"{record['name']} - {contact.get('fullname', 'N/A')}")
+# Convert query results to a DataFrame
+df = (client.query.builder("account")
+      .select("accountid", "name")
+      .where(col("statecode") == 0)
+      .execute()
+      .to_dataframe())
+
+# Limit total results
+result = client.query.builder("account").select("name").top(100).execute()
+
+# Simple streaming shortcut via records.list_pages() (string filter only, same params as records.list())
+for page in client.records.list_pages("account", filter="statecode eq 0", select=["name"], page_size=500):
+    for record in page:
+        print(record["name"])
 ```
 
 #### Create Records with Lookup Bindings (@odata.bind)
@@ -212,18 +246,21 @@ client.records.delete("account", [id1, id2, id3], use_bulk_delete=True)
 
 The SDK provides DataFrame wrappers for all CRUD operations via the `client.dataframe` namespace, using pandas DataFrames and Series as input/output.
 
-> **Note:** `client.dataframe.get()` is deprecated. Use `client.query.builder(table).select(...).where(...).to_dataframe()` instead.
+> **Note:** `client.dataframe.get()` is deprecated. Use `client.query.builder(table).select(...).where(...).execute().to_dataframe()` instead. `QueryBuilder.to_dataframe()` (without `.execute()`) is also deprecated — always call `.execute()` first.
 
 ```python
 import pandas as pd
 
-# Query records -- returns a single DataFrame (GA builder pattern)
+# Query records -- returns a single DataFrame (GA pattern: .execute().to_dataframe())
 from PowerPlatform.Dataverse.models.filters import col
-df = client.query.builder("account").where(col("statecode") == 0).select("name").to_dataframe()
+df = client.query.builder("account").where(col("statecode") == 0).select("name").execute().to_dataframe()
 print(f"Got {len(df)} rows")
 
 # Limit results with top
-df = client.query.builder("account").select("name").top(100).to_dataframe()
+df = client.query.builder("account").select("name").top(100).execute().to_dataframe()
+
+# Via records.list() (simpler for basic queries)
+df = client.records.list("account", filter="statecode eq 0", select=["name"]).to_dataframe()
 
 # Fetch single record as one-row DataFrame
 df = client.records.retrieve("account", account_id, select=["name"]).to_dataframe()
@@ -444,8 +481,8 @@ Use `client.batch` to send multiple operations in one HTTP request. All batch me
 batch = client.batch.new()
 batch.records.create("account", {"name": "Contoso"})
 batch.records.update("account", account_id, {"telephone1": "555-0100"})
-batch.records.retrieve("account", account_id, select=["name"])     # single record (GA)
-batch.records.list("account", filter="statecode eq 0", top=50)     # multi-record, single page
+batch.records.retrieve("account", account_id, select=["name"], include_annotations="OData.Community.Display.V1.FormattedValue")  # single record
+batch.records.list("account", filter="statecode eq 0", select=["name"], orderby=["name asc"], top=50, page_size=25, count=True)  # multi-record, single page
 batch.query.sql("SELECT TOP 5 name FROM account")
 
 result = batch.execute()
@@ -530,16 +567,17 @@ except ValidationError as e:
 
 ### Performance Optimization
 
-1. **Use bulk operations** - Pass lists to create/update/delete for automatic optimization
-2. **Specify select fields** - Limit returned columns to reduce payload size
-3. **Control page size** - Use `top` and `page_size` parameters appropriately
-4. **Reuse client instances** - Don't create new clients for each operation
-5. **Use production credentials** - ClientSecretCredential or CertificateCredential for unattended operations
-6. **Error handling** - Implement retry logic for transient errors (`e.is_transient`)
-7. **Always include customization prefix** for custom tables/columns
-8. **Use lowercase for column names, match `$metadata` for navigation properties** - Column names in `$select`/`$filter`/record payloads use lowercase LogicalNames. Navigation properties in `$expand` and `@odata.bind` keys are case-sensitive and must match the entity's `$metadata` (PascalCase for custom lookups like `new_CustomerId`, lowercase for system lookups like `parentaccountid`)
-9. **Test in non-production environments** first
-10. **Use named constants** - Import cascade behavior constants from `PowerPlatform.Dataverse.common.constants`
+1. **Prefer `client.query.builder()` for any non-trivial query** — use the builder for filtering, sorting, expansion, or formatted values; `records.list()` is a convenience shortcut for simple filter+select only
+2. **Use bulk operations** - Pass lists to create/update/delete for automatic optimization
+3. **Specify select fields** - Limit returned columns to reduce payload size
+4. **Control page size** - Use `top` and `page_size` parameters appropriately; use `execute_pages()` for large sets
+5. **Reuse client instances** - Don't create new clients for each operation
+6. **Use production credentials** - ClientSecretCredential or CertificateCredential for unattended operations
+7. **Error handling** - Implement retry logic for transient errors (`e.is_transient`)
+8. **Always include customization prefix** for custom tables/columns
+9. **Use lowercase for column names, match `$metadata` for navigation properties** - Column names in `$select`/`$filter`/record payloads use lowercase LogicalNames. Navigation properties in `$expand` and `@odata.bind` keys are case-sensitive and must match the entity's `$metadata` (PascalCase for custom lookups like `new_CustomerId`, lowercase for system lookups like `parentaccountid`)
+10. **Test in non-production environments** first
+11. **Use named constants** - Import cascade behavior constants from `PowerPlatform.Dataverse.common.constants`
 
 ## Additional Resources
 
@@ -552,9 +590,10 @@ Load these resources as needed during development:
 
 ## Key Reminders
 
-1. **Schema names are required** - Never use display names
-2. **Custom tables need prefixes** - Include customization prefix (e.g., "new_")
-3. **Filter is case-sensitive** - Use lowercase logical names
-4. **Bulk operations are encouraged** - Pass lists for optimization
-5. **No trailing slashes in URLs** - Format: `https://org.crm.dynamics.com`
-6. **Structured errors** - Check `is_transient` for retry logic
+1. **Use `client.query.builder()` for queries** — it's the primary query pattern; `records.list()` is a shortcut for trivial filter+select only
+2. **Schema names are required** - Never use display names
+3. **Custom tables need prefixes** - Include customization prefix (e.g., "new_")
+4. **Filter is case-sensitive** - Use lowercase logical names
+5. **Bulk operations are encouraged** - Pass lists for optimization
+6. **No trailing slashes in URLs** - Format: `https://org.crm.dynamics.com`
+7. **Structured errors** - Check `is_transient` for retry logic

CHANGELOG.md

@@ -8,9 +8,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Added
-- `client.records.retrieve(table, record_id)` — fetch a single record by GUID; returns `None` on 404 instead of raising (#175)
-- `client.records.list(table, filter, select, top)` — eager fetch returning a flat `QueryResult`; GA replacement for `records.get()` without a record ID (#175)
-- `client.records.list_pages(table, filter, select, top)` — lazy iterator yielding one `QueryResult` per HTTP page; streaming counterpart to `list()` (#175)
+- `client.records.retrieve(table, record_id, *, select, include_annotations)` — fetch a single record by GUID; returns `None` on 404 instead of raising; `include_annotations` maps to the `Prefer: odata.include-annotations` header for formatted values and lookup labels (#175)
+- `client.records.list(table, *, filter, select, top, orderby, expand, page_size, count, include_annotations)` — eager fetch returning a flat `QueryResult`; GA replacement for `records.get()` without a record ID; `page_size` controls `Prefer: odata.maxpagesize`, `count=True` adds `$count=true`, `include_annotations` requests formatted values (#175)
+- `client.records.list_pages(table, *, filter, select, top, orderby, expand, page_size, count, include_annotations)` — lazy iterator yielding one `QueryResult` per HTTP page; streaming counterpart to `list()`; same parameter set (#175)
+- `client.batch.records.retrieve()` and `client.batch.records.list()` now accept the same `include_annotations`, `orderby`, `expand`, `page_size`, and `count` parameters as their non-batch counterparts (#175)
 - `client.query.fetchxml(xml)` — FetchXML support returning an inert `FetchXmlQuery`; no HTTP request is made until `.execute()` or `.execute_pages()` is called (#175)
 - `FetchXmlQuery` implements the correct Dataverse paging cookie algorithm: annotation parsed as outer XML, `pagingcookie` attribute double URL-decoded, server-supplied `pagenumber` used for next page, `morerecords` handled as both `bool` and `"true"` string, `UserWarning` emitted on simple paging fallback, 32,768-character URL limit enforced (documented Dataverse GET cap), 10,000-page circuit breaker against runaway iteration (#175)
 - `QueryBuilder.execute_pages()` — lazy per-page streaming returning one `QueryResult` per HTTP page; replaces deprecated `execute(by_page=True)` (#175)
@@ -19,6 +20,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `DataverseModel` structural `Protocol` (`models/protocol.py`) — implement on any entity class to enable typed integration with CRUD operations without specifying table names or serializing manually (#175)
 - `col()`, `raw()`, `QueryResult`, and `DataverseModel` exported from the top-level `PowerPlatform.Dataverse` package (#175)
 - v0→v1 migration tool: `tools/migrate_v0_to_v1.py` rewrites v0 call sites to the v1 API with `--dry-run` support; covers `create`, `update`, `delete`, `get`, `list`, `fetchxml`, and query builder patterns (#175)
+- Migration tool now auto-rewrites `QueryBuilder.to_dataframe()` → `.execute().to_dataframe()` (inserts `.execute()` when receiver is a recognised builder chain); output improved with `[NEEDS-MANUAL]` label for files that have no auto-rewrites but require manual attention, and a trailing note on `[MIGRATED]` lines when manual items remain (#175)
 
 ### Changed
 - `QueryBuilder.execute()` now returns a flat `QueryResult` (all pages collected eagerly) instead of `Iterable[Record]` (#175)