[Power BI] Address ARG payload size errors

docs-mslearn/toolkit/changelog.md

@@ -3,7 +3,7 @@ title: FinOps toolkit changelog
 description: Review the latest features and enhancements in the FinOps toolkit, including updates to FinOps hubs, Power BI reports, and more.
 author: MSBrett
 ms.author: brettwil
-ms.date: 03/04/2026
+ms.date: 03/11/2026
 ms.topic: reference
 ms.service: finops
 ms.subservice: finops-toolkit
@@ -70,6 +70,11 @@ The following section lists features and enhancements that are currently in deve
   - Fixed inverted verbose logging in [Start-FinOpsCostExport](powershell/cost-management/Start-FinOpsCostExport.md) that showed blank dates when a date range was specified.
   - Addressed minor lint warnings across PowerShell commands.
 
+### [Power BI reports](power-bi/reports.md) v14
+
+- **Fixed**
+  - Paginate Azure Resource Graph queries by subscription to mitigate [payload size limit](help/errors.md#response-payload-size-is-and-has-exceeded-the-limit) errors in the [Governance](power-bi/governance.md) and [Workload optimization](power-bi/workload-optimization.md) reports ([#1768](https://github.com/microsoft/finops-toolkit/issues/1768)).
+
 <br>
 
 ## v13 Update 1

docs-mslearn/toolkit/help/errors.md

@@ -3,7 +3,7 @@ title: Troubleshoot common FinOps toolkit errors
 description: This article describes common FinOps toolkit errors and provides solutions to help you resolve issues you might encounter.
 author: flanakin
 ms.author: micflan
-ms.date: 02/24/2026
+ms.date: 03/11/2026
 ms.topic: troubleshooting
 ms.service: finops
 ms.subservice: finops-toolkit
@@ -747,13 +747,59 @@ Azure Resource Graph queries in the Governance and Workload optimization Power B
 
 > _OLE DB or ODBC error: [Expression.Error] Please provide below info when asking for support: timestamp = {timestamp}, correlationId = {guid}. Details: Response payload size is {number}, and has exceeded the limit of 16777216. Please consider querying less data at a time and make paginated call if needed._
 
-This error means that you have more resources than are supported in an unfiltered Resource Graph query. This happens because FinOps toolkit reports are designed to show resource-level details and are not aggregated. They are designed for small- and medium-sized environments and not designed to support organizations with millions of resources.
+Azure Resource Graph enforces a 16 MB response payload limit per query. FinOps toolkit reports automatically paginate queries in batches of subscriptions (default: 100 per batch) to stay within this limit, so most environments should not encounter this error. If you still see it, it means the resources in a single batch of subscriptions exceed the 16 MB limit.
 
-**Mitigation**: If you experience this error, there are several options:
+**Mitigation**: Try the following options in order:
 
-- Remove columns that are not necessary for your needs.
-- Filter the query to return fewer resources based on what's most important for you (e.g., subscriptions, tags).
-- Disable the query so it doesn't block other queries from running.
+### Option 1: Reduce the batch size
+
+Reduce the number of subscriptions queried in each batch:
+
+1. Open Power BI Desktop and select **Transform data** from the ribbon.
+2. In the **Queries** pane on the left, expand the **Functions** folder.
+3. Select the **ftk_ARGBatchSize** function.
+4. Change the return value from `100` to a smaller number (e.g., `20` or `10`).
+5. Select **Close & Apply** to save changes.
+
+### Option 2: Filter by resource group or tags
+
+Add a filter clause to the failing query to reduce the number of resources returned:
+
+1. Open Power BI Desktop and select **Transform data** from the ribbon.
+2. In the **Queries** pane on the left, expand the **Resource Graph** folder.
+3. Select the query that's failing (e.g., **NetworkSecurityGroups**, **Resources**).
+4. In the query editor, find the `query = "` section in the formula bar.
+5. Add a filter clause after the `| where type` line and before any `| extend` clauses. For example:
+
+   ```kusto
+   | where resourceGroup in~ ('rg-production', 'rg-staging')
+   ```
+
+   Or filter by tags:
+
+   ```kusto
+   | where tags.Environment =~ 'Production'
+   ```
+
+6. Select **Close & Apply** to save changes.
+
+### Option 3: Remove unnecessary columns
+
+Reduce the payload size by removing columns you don't need:
+
+1. Open the query in Power Query Editor (steps 1-3 from Option 2).
+2. In the query text, remove column names from the `extend` or `project` statements that you don't need for your analysis.
+3. Be careful not to remove columns that are used in report visuals or relationships.
+
+### Option 4: Disable the failing query
+
+If a specific query consistently fails and isn't critical to your needs:
+
+1. In Power Query Editor, right-click the failing query in the **Queries** pane.
+2. Uncheck **Enable load** to prevent the query from loading data.
+3. The query will remain in the report but won't execute during refresh.
+
+For more information about Azure Resource Graph limits, see [Working with large Azure resource data sets](/azure/governance/resource-graph/concepts/work-with-data).
 
 <br>
 

docs-mslearn/toolkit/power-bi/governance.md

@@ -3,7 +3,7 @@ title: FinOps toolkit Governance report
 description: Summarize cloud governance posture including areas like compliance, security, operations, and resource management in Power BI.
 author: flanakin
 ms.author: micflan
-ms.date: 02/24/2026
+ms.date: 03/11/2026
 ms.topic: concept-article
 ms.service: finops
 ms.subservice: finops-toolkit
@@ -119,6 +119,14 @@ The **Network security groups** page lists network security groups and network s
 
 <br>
 
+## Known limitations
+
+The Governance report uses Azure Resource Graph to query resource details. Azure Resource Graph has a response payload limit of 16 MB per query. The report automatically paginates queries in batches of subscriptions to stay within this limit, but may not work for large environments where a single batch exceeds the limit.
+
+If you experience a "Response payload size... exceeded the limit" error, open the report in Power BI Desktop, go to the Power Query editor, and reduce the value returned by the `ftk_ARGBatchSize` function (default: 100). For detailed steps, see [Response payload size exceeded the limit](../help/errors.md#response-payload-size-is-and-has-exceeded-the-limit) in the error reference guide.
+
+<br>
+
 ## Looking for more?
 
 We'd love to hear about any reports, charts, or general reporting questions you're looking to answer. Create a new issue with the details that you'd like to see either included in existing or new reports.

docs-mslearn/toolkit/power-bi/workload-optimization.md

@@ -3,7 +3,7 @@ title: FinOps toolkit Workload optimization report
 description: Learn about the Workload optimization report, which identifies opportunities for rightsizing and removing unused resources to enhance efficiency.
 author: flanakin
 ms.author: micflan
-ms.date: 02/24/2026
+ms.date: 03/11/2026
 ms.topic: concept-article
 ms.service: finops
 ms.subservice: finops-toolkit
@@ -79,10 +79,11 @@ The chart shows the cost of each disk over time. The table shows the disks with
 
 <br>
 
-## See also
+## Known limitations
 
-- [Common terms](../help/terms.md)
-- [Data dictionary](../help/data-dictionary.md)
+The Workload optimization report uses Azure Resource Graph to query resource details. Azure Resource Graph has a response payload limit of 16 MB per query. The report automatically paginates queries in batches of subscriptions to stay within this limit, but may not work for large environments where a single batch exceeds the limit.
+
+If you experience a "Response payload size... exceeded the limit" error, open the report in Power BI Desktop, go to the Power Query editor, and reduce the value returned by the `ftk_ARGBatchSize` function (default: 100). For detailed steps, see [Response payload size exceeded the limit](../help/errors.md#response-payload-size-is-and-has-exceeded-the-limit) in the error reference guide.
 
 <br>
 
@@ -99,6 +100,11 @@ We'd love to hear about any reports, charts, or general reporting questions you'
 
 ## Related content
 
+Related resources:
+
+- [Common terms](../help/terms.md)
+- [Data dictionary](../help/data-dictionary.md)
+
 Related FinOps capabilities:
 
 - [Reporting and analytics](../../framework/understand/reporting.md)

src/power-bi/kql/Shared.Dataset/definition/expressions.tmdl

@@ -89,6 +89,45 @@ expression ftk_DemoFilter =
 
 	annotation PBI_ResultType = Function
 
+/// Number of subscriptions to include in each Azure Resource Graph query batch. Decrease if you experience payload limit errors. Default: 100.
+expression ftk_ARGBatchSize = () => 100
+	lineageTag: f7a1b2c3-d4e5-6789-0abc-def123456789
+	queryGroup: Functions
+
+	annotation PBI_NavigationStepName = Navigation
+
+	annotation PBI_ResultType = Function
+
+/// Queries Azure Resource Graph in batches of subscriptions and combines results to avoid the 16 MB payload limit.
+expression ftk_QueryARG =
+		(query as text, optional batchMultiplier as number) =>
+		    let
+		      batchSize = ftk_ARGBatchSize() * (if batchMultiplier <> null then batchMultiplier else 1),
+		      subQuery = "resourcecontainers | where type == 'microsoft.resources/subscriptions' | project subscriptionId" & ftk_DemoFilter(),
+		      Subscriptions = AzureResourceGraph.Query(subQuery, "Tenant", null, null, [resultTruncated = false]),
+		      SubscriptionIds = Subscriptions[subscriptionId],
+		      BatchCount = Number.RoundUp(List.Count(SubscriptionIds) / batchSize),
+		      Batches = List.Transform({0..BatchCount-1}, each
+		        let
+		          batchStart = _ * batchSize,
+		          batchIds = List.Range(SubscriptionIds, batchStart, List.Min({batchSize, List.Count(SubscriptionIds) - batchStart})),
+		          subFilter = " | where subscriptionId in (" & Text.Combine(List.Transform(batchIds, each "'" & _ & "'"), ",") & ")",
+		          fullQuery = query & subFilter,
+		          Source = AzureResourceGraph.Query(fullQuery, "Tenant", null, null, [resultTruncated = false])
+		        in
+		          if Table.HasColumns(Source, "Results") and Table.RowCount(Source) = 1 then null else Source
+		      ),
+		      Filtered = List.RemoveNulls(Batches),
+		      Combined = if List.Count(Filtered) > 0 then Table.Combine(Filtered) else null
+		    in
+		      Combined
+	lineageTag: a1b2c3d4-e5f6-7890-abcd-ef1234567890
+	queryGroup: Functions
+
+	annotation PBI_NavigationStepName = Navigation
+
+	annotation PBI_ResultType = Function
+
 /// URI of the FinOps hub Azure Data Explorer cluster to pull data from. Copy from the "clusterUri" deployment output.
 /// 
 /// Pro tip: As a shortcut, you can alternatively use the "{name}.{region}" subset of the URI.
@@ -114,14 +153,15 @@ expression PolicyDefinitions =
 		  query = "
 		  policyResources
 		  | where type =='microsoft.authorization/policydefinitions'
-		  | extend displayName = properties.displayName,
+		  | extend
+		      subscriptionId = tostring(split(id, '/')[2]),
+		      displayName = properties.displayName,
 		      description = properties.description,
 		      version = properties.version
-		    | project subscriptionId, id, name, displayName, description, version"
-		  & ftk_DemoFilter(),
-		  Source = AzureResourceGraph.Query(query, "Tenant", null, null, [resultTruncated = false]),
+		    | project subscriptionId, id, name, displayName, description, version",
+		  Source = ftk_QueryARG(query, 5),
 		  NullHandling =
-		    if Table.HasColumns(Source, "Results") and Table.RowCount(Source) = 1 then
+		    if Source = null then
 					#table(
 						{ "subscriptionId", "id", "name", "displayName", "description", "version" },
 						{}

src/power-bi/kql/Shared.Dataset/definition/tables/AdvisorRecommendations.tmdl

@@ -475,10 +475,8 @@ table AdvisorRecommendations
 		queryGroup: 'Resource Graph'
 		source = ```
 				let
-				  Source = AzureResourceGraph.Query("
-
-
-				  advisorresources 
+				  query = "
+				  advisorresources
 				  | where type == 'microsoft.advisor/recommendations'
 				  | where properties.category == 'Cost'
 				  | extend
@@ -523,11 +521,10 @@ table AdvisorRecommendations
 				    ['properties.resourceMetadata.action'] = properties.resourceMetadata.action,
 				    ['properties.resourceMetadata.source'] = properties.resourceMetadata.source,
 				    SortOrder = case(properties.impact == 'High', 1, properties.impact == 'Medium', 2, properties.impact == 'Low', 3, 9)
-
-
-				  " & ftk_DemoFilter(), "Tenant", null, null, [resultTruncated = true]),
+				  ",
+				  Source = ftk_QueryARG(query, 5),
 				  NullHandling =
-				    if Table.HasColumns(Source, "Results") and Table.RowCount(Source) = 1 then
+				    if Source = null then
 							#table(
 								{ "id", "name", "type", "properties.recommendationTypeId", "properties.shortDescription", "properties.resourceMetadata", "properties.suppressionIds", "properties.impactedField", "properties.impactedValue", "properties.lastUpdated", "properties.category", "properties.metadata", "properties.impact", "properties.shortDescription.problem", "properties.shortDescription.solution", "properties.extendedProperties.ObservationPeriodStartDate", "properties.extendedProperties.ObservationPeriodEndDate", "properties.extendedProperties.annualSavingsAmount", "properties.extendedProperties.HasRecommendation", "properties.extendedProperties.savingsCurrency", "properties.extendedProperties.IsInReplication", "properties.extendedProperties.Recommended_DTU", "properties.extendedProperties.Recommended_SKU", "properties.extendedProperties.ResourceGroup", "properties.extendedProperties.savingsAmount", "properties.extendedProperties.DatabaseSize", "properties.extendedProperties.DatabaseName", "properties.extendedProperties.ServerName", "properties.extendedProperties.Region", "properties.extendedProperties.lookbackPeriod", "properties.extendedProperties.subId", "properties.extendedProperties.scope", "properties.extendedProperties.term", "properties.extendedProperties.sku", "properties.extendedProperties.commitment", "properties.extendedProperties.currentSku", "properties.extendedProperties.targetSku", "properties.extendedProperties.recommendationMessage", "properties.resourceMetadata.resourceId", "properties.resourceMetadata.singular", "properties.resourceMetadata.plural", "properties.resourceMetadata.action", "properties.resourceMetadata.source", "SortOrder" },
 								{}