AZtfexport - problem with big subscriptions

[rafall92]

Hi
I try to do full export of all azure subscriptions in my tenant. I prepared script which perform export, and commit HCL files on github repository. Each of subscription has its own repo, so we run this script by github action. I use powershell parrarelism, so i run export for all of RG at once.
Unfortunately, some of them are big, Not because a lot of RG, but because of lots of resources. In one subscription we have aprox. 10k of resources, so exporting them takes a lot of time(more than 6 hours), so we hit github action limit. We decide to move this export to separate machine, and export many subscriptions at once. I try to do that with 2 subscriptions, and got this errors.

Error: initializing provider for the import directories: 5 errors occurred:
	* task error: error running terraform init: exit status 0xffffffff
	* task error: error running terraform init: exit status 0xffffffff
	* task error: error running terraform init: exit status 0xffffffff
	* task error: error running terraform init: exit status 0xffffffff
	* task error: error running terraform init: exit status 0xffffffff

Each of export has its own workDir, i use default pararel value(not using parameter, so stay at default value of 10). Is there any tips how can i handle my problem, or how can i decrese execution time? I don't need state, only HCL

This is part of code responsible for export

$resourceGroups = Get-AzResourceGroup

$resourceGroups | ForEach-Object {
    $rgName = $_.ResourceGroupName

    $exportPath = Join-Path -Path $Using:backupARMPath -ChildPath $Using:rgName | Join-Path -ChildPath "$resName.json"

    aztfexport resource-group --non-interactive `
        --subscription-id $subscriptionId `
        --output-dir $exportPath `
        --hcl-only=true `
        --append=false `
        --plain-ui `
        --exclude-azure-resource "secrets" `
        --exclude-azure-resource "systemTopics" `
        --exclude-azure-resource "certificates" `
        --exclude-azure-resource "listClusterUserCredential" `
        --exclude-azure-resource "tables/WADDiagnosticInfrastructureLogsTable" `
        --exclude-azure-resource "tables/WADMetricsPT1MP10DV2S20241010" `
        --exclude-azure-resource "tables/WADPerformanceCountersTable" `
        --exclude-azure-resource "virtualMachines/azure-*" `
        $rgName 2>&1 | Tee-Object -Variable result
} -ThrottleLimit 4

And I run this script for 2 subscriptions at once

[magodo]

Hi @rafall92, thank you for reaching out!

The default behavior of export is that it will create N+1 workspaces under /tmp directory - where N is the parallelism that defaults to 10. These workspaces will run terraform init individually, which will mainly download the provider. There is one way to accelerate this part is to enable the plugin caching. While the export process is still in scale since your subscription is big. To accelerate the export process, you can increase the parallelism, exclude some uninterested resource types by the --exclude-xxx options.

Since you mentioned that you only need want the HCL. Then there is an advanced usage to avoid the N+1 workspaces and terraform init:

1. Install the target provider in your machine. This can be done by manually do a terraform init with the corresponding provider config.
2. Run aztfexport <subcommand> --tfclient-plugin-path=<path-to-provider> --hcl-only <other options>...

In this way, there will be no state file generated, and the total resource consumption will be much lower as there will be only one provider instance spinned up. However, again the export process is still propotional to your subscription scale.

[rafall92]

@magodo: Thanks. This was vey helpfull. I reduce time of one subscription error from 7h to 45 miutes. But Now i would like to export many subscriptions at once. When i try to do this, I get randomly this error:

Failed to call provider plugin: Provider returned RPC error Unavailable: connection error: desc = "transport: error while dialing: dial tcp 127.0.0.1:10000: connectex: No connection could be made because the target machine actively refused it.".

or

Error: configure provider: [{69 Failed to call provider plugin Provider returned RPC error Unavailable: error reading from server: read tcp 127.0.0.1:60497->127.0.0.1:10001: wsarecv: An existing connection was forcibly closed by the remote host.. []}]

[magodo]

This is most likely the provider process being terminated by the OS for memory pressure. Please try to export these big subs in sequence.