Task: Invocation of interface 'Configure' of role 'Cloud\Fabric\SQL' failed. Step 60.62.101

[jdm019000]

Any Suggestions here. Trying to deploy ASDK 2005. All of the roles seem to be up. Not sure what direction I should go.

`2020-11-04 14:22:55 Verbose Step:Skipping task 'Deployment-Phase3-DeployADFS' of role 'Cloud' because it was skipped or run successfully previously.
2020-11-04 14:22:55 Verbose 5> Creating action host for action Deployment-Phase3-DeploySQL.
2020-11-04 14:22:55 Verbose 5> Found the role 'Cloud' in the current configuration.
2020-11-04 14:22:55 Verbose 5> Using the existing expanded action definition for action Deployment-Phase3-DeploySQL.
2020-11-04 14:22:55 Verbose 5> Creating action host for action Deployment-Phase3-DeploySQL.
2020-11-04 14:22:55 Verbose 5> Action host for action Deployment-Phase3-DeploySQL has been created.
2020-11-04 14:22:55 Verbose 5> Update the actionTask definition with the new nested actionHost definition for Cloud - Deployment-Phase3-DeploySQL.
2020-11-04 14:22:55 Verbose 5> Task: Running action type 'Deployment-Phase3-DeploySQL' of role 'Cloud'.
2020-11-04 14:22:55 Verbose 5> Action: Running action Deployment-Phase3-DeploySQL.
2020-11-04 14:22:55 Verbose 5> Action: Running action plan 'Deployment-Phase3-DeploySQL'.
2020-11-04 14:22:55 Verbose 5> The following steps are currently ready: 60.62.101.
2020-11-04 14:22:55 Verbose 5> Processing step: 60.62.101.
2020-11-04 14:22:55 Verbose 5> Executing step: 60.62.101.
2020-11-04 14:22:55 Verbose 5> Checking if any of the in progress steps are complete. The following steps are currently in progress: '60.62.101'.
2020-11-04 14:22:55 Verbose 5> The following steps are currently ready: .
2020-11-04 14:22:55 Verbose 5> No steps are ready for execution. Waiting for any of the currently-running steps to complete. The following steps are currently in progress: '60.62.101'.
2020-11-04 14:22:55 Verbose 5> Step: Running step 60.62.101 - (FBI) Setup SQL VMs
2020-11-04 14:22:55 Verbose 5> Task: Running interface 'Configure' of role 'Cloud\Fabric\SQL'. Attempt #1 of 3
2020-11-04 14:22:55 Verbose 5> Loading CustomerConfiguration for interface 'Configure' of role 'Cloud\Fabric\SQL'.
2020-11-04 14:22:55 Verbose 5> Task: Interface 'Configure' is getting created.
2020-11-04 14:22:55 Verbose 5> Interface: Path to module: C:\CloudDeployment\Classes\SQL\SQL.psm1
2020-11-04 14:22:55 Verbose 5> Interface: Running interface Configure (Classes\SQL\SQL.psm1, SQL:Configure)
2020-11-04 14:22:55 Verbose 5> [PSTask Concurrency] [113>SQL:Configure] Entering throttling instance, enqueue for waiting. Priority '0', Weight '10'.
2020-11-04 14:22:55 Verbose 5> [PSTask Concurrency] [113>SQL:Configure] Validating availability to start task.
2020-11-04 14:22:55 Verbose 5> [PSTask Concurrency] [113>SQL:Configure] Task available to start.
2020-11-04 14:22:55 Verbose 5> [PSTask Concurrency] [113>SQL:Configure] Parallel task count increased by '10' to '10'.
2020-11-04 14:22:55 Verbose 5> [PSTask Concurrency] [113>SQL:Configure] Throttling instance dequeued from waiting. Priority '0', Weight '10'.
2020-11-04 14:22:55 Verbose 5> [PSTask Concurrency] [113>SQL:Configure] TotalVirtualMemoryMB: [131071.55078125], FreeVirtualMemoryMB: [58925.70703125], FreePhysicalMemoryMB: [59237.52734375].
2020-11-04 14:23:08 Warning 5> [SQL:Configure] The names of some imported commands from the module 'RoleHelpers' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2020-11-04 14:23:10 Warning 5> [SQL:Configure] The names of some imported commands from the module 'NetworkHelpers' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2020-11-04 14:23:10 Warning 5> [SQL:Configure] The names of some imported commands from the module 'RoleHelpers' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2020-11-04 14:23:15 Warning 5> [SQL:Configure] The names of some imported commands from the module 'RoleHelpers' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2020-11-04 14:23:16 Warning 5> [SQL:Configure] The names of some imported commands from the module 'RoleHelpers' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2020-11-04 14:23:16 Warning 5> [SQL:Configure] The names of some imported commands from the module 'RoleHelpers' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2020-11-04 14:23:17 Warning 5> [SQL:Configure] The names of some imported commands from the module 'RoleHelpers' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2020-11-04 14:23:22 Warning 5> [SQL:Configure] The names of some imported commands from the module 'NetworkHelpers' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2020-11-04 14:23:22 Warning 5> [SQL:Configure] The names of some imported commands from the module 'SQL' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2020-11-04 14:23:23 Warning 5> [SQL:Configure] The names of some imported commands from the module 'SQL' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2020-11-04 14:23:23 Verbose 5> [SQL:Configure] Execution start of New-Sql.
2020-11-04 14:23:30 Verbose 5> [SQL:Configure] Prepare nodes for clustering: 'AzS-Sql01.azurestack.local'
2020-11-04 14:23:33 Verbose 5> [SQL:Configure] Enabling client/diagnostic event log on AzS-Sql01.azurestack.local
2020-11-04 14:23:40 Verbose 5> [SQL:Configure] Clean up the AD computer object AzS-SqlCl and its DNS record
2020-11-04 14:23:40 Verbose 5> [SQL:Configure] Removing DNS resource record for name 'AzS-SqlCl' with IP 'True' under zone name 'azurestack.local' on the domain controller 'AzS-DC01'.
2020-11-04 14:23:45 Verbose 5> [SQL:Configure] Record for 'True' (A) not present.
2020-11-04 14:23:45 Verbose 5> [SQL:Configure] Removing DNS resource record for name 'AzS-SqlCl' with IP 'True' under zone name 'azurestack.local' on the domain controller 'AzS-DC01'.
2020-11-04 14:23:45 Verbose 5> [SQL:Configure] Record for 'True' (AAAA) not present.
2020-11-04 14:23:45 Verbose 5> [SQL:Configure] Creating the cluster 'AzS-SqlCl'.
2020-11-04 14:24:08 Verbose 5> [SQL:Configure] Collecting client/diagnostic event log on AzS-Sql01.azurestack.local
2020-11-04 14:24:32 Verbose 5> [SQL:Configure] ECE PowerShellHost Script End
2020-11-04 14:24:32 Verbose 5> Interface: PowerShell job terminal state is Failed for interface 'Configure'.
2020-11-04 14:24:33 Verbose 5> [PSTask Concurrency] [113>SQL:Configure] Exiting throttling instance. Priority '0', Weight '10'.
2020-11-04 14:24:33 Verbose 5> [PSTask Concurrency] [113>SQL:Configure] Parallel task count decreased by '10' to '0'.
2020-11-04 14:24:33 Warning 5> Task: Invocation of interface 'Configure' of role 'Cloud\Fabric\SQL' failed:

Type 'Configure' of Role 'SQL' raised an exception:

There was an error adding node 'AzS-Sql01.azurestack.local' to the cluster
The computer 'AzS-Sql01.azurestack.local' is joined to a cluster.
Command Arguments

---

Add-GuestCluster {Parameters=CloudEngine.Configurations.EceInterfaceParameters, RunAsUserID=Fabric}
New-Sql {Parameters=CloudEngine.Configurations.EceInterfaceParameters, ErrorAction=Stop, Verbose=True}
{}
{CloudEngine.Configurations.EceInterfaceParameters}
{C:\CloudDeployment\CloudDeployment.psd1, CloudEngine.Configurations.EceInterfaceParameters, 1111131...

at Trace-Error, C:\CloudDeployment\Common\Tracer.psm1: line 63
at Add-GuestCluster, C:\CloudDeployment\Roles\Common\RoleHelpers.psm1: line 390
at New-Sql, C:\CloudDeployment\Roles\SQL\SQL.psm1: line 55
at Configure, C:\CloudDeployment\Classes\SQL\SQL.psm1: line 19
at , : line 56
at , : line 54
2020-11-04 14:24:33 Warning 5> Task: Retrying interface 'Configure' of role 'Cloud\Fabric\SQL'...
2020-11-04 14:24:33 Verbose 5> Task: Running interface 'Configure' of role 'Cloud\Fabric\SQL'. Attempt #2 of 3
2020-11-04 14:24:33 Verbose 5> Wait for 60 seconds before attempting retry.
2020-11-04 14:25:33 Verbose 5> Loading CustomerConfiguration for interface 'Configure' of role 'Cloud\Fabric\SQL'.
2020-11-04 14:25:33 Verbose 5> Task: Interface 'Configure' is getting created.
2020-11-04 14:25:33 Verbose 5> Interface: Path to module: C:\CloudDeployment\Classes\SQL\SQL.psm1
2020-11-04 14:25:33 Verbose 5> Interface: Running interface Configure (Classes\SQL\SQL.psm1, SQL:Configure)
2020-11-04 14:25:33 Verbose 5> [PSTask Concurrency] [113>SQL:Configure] Entering throttling instance, enqueue for waiting. Priority '0', Weight '10'.
2020-11-04 14:25:33 Verbose 5> [PSTask Concurrency] [113>SQL:Configure] Validating availability to start task.
2020-11-04 14:25:33 Verbose 5> [PSTask Concurrency] [113>SQL:Configure] Task available to start.
2020-11-04 14:25:33 Verbose 5> [PSTask Concurrency] [113>SQL:Configure] Parallel task count increased by '10' to '10'.
2020-11-04 14:25:33 Verbose 5> [PSTask Concurrency] [113>SQL:Configure] Throttling instance dequeued from waiting. Priority '0', Weight '10'.
2020-11-04 14:25:33 Verbose 5> [PSTask Concurrency] [113>SQL:Configure] TotalVirtualMemoryMB: [131071.55078125], FreeVirtualMemoryMB: [58910.1015625], FreePhysicalMemoryMB: [59238.4453125].
2020-11-04 14:25:40 Warning 5> [SQL:Configure] The names of some imported commands from the module 'RoleHelpers' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2020-11-04 14:25:41 Warning 5> [SQL:Configure] The names of some imported commands from the module 'NetworkHelpers' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2020-11-04 14:25:41 Warning 5> [SQL:Configure] The names of some imported commands from the module 'RoleHelpers' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2020-11-04 14:25:47 Warning 5> [SQL:Configure] The names of some imported commands from the module 'RoleHelpers' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2020-11-04 14:25:48 Warning 5> [SQL:Configure] The names of some imported commands from the module 'RoleHelpers' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2020-11-04 14:25:48 Warning 5> [SQL:Configure] The names of some imported commands from the module 'RoleHelpers' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2020-11-04 14:25:49 Warning 5> [SQL:Configure] The names of some imported commands from the module 'RoleHelpers' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2020-11-04 14:25:53 Warning 5> [SQL:Configure] The names of some imported commands from the module 'NetworkHelpers' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2020-11-04 14:25:53 Warning 5> [SQL:Configure] The names of some imported commands from the module 'SQL' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2020-11-04 14:25:54 Warning 5> [SQL:Configure] The names of some imported commands from the module 'SQL' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2020-11-04 14:25:54 Verbose 5> [SQL:Configure] Execution start of New-Sql.
2020-11-04 14:25:59 Verbose 5> [SQL:Configure] Prepare nodes for clustering: 'AzS-Sql01.azurestack.local'
2020-11-04 14:26:02 Verbose 5> [SQL:Configure] Enabling client/diagnostic event log on AzS-Sql01.azurestack.local
2020-11-04 14:26:04 Verbose 5> [SQL:Configure] Clean up the AD computer object AzS-SqlCl and its DNS record
2020-11-04 14:26:04 Verbose 5> [SQL:Configure] Removing DNS resource record for name 'AzS-SqlCl' with IP 'True' under zone name 'azurestack.local' on the domain controller 'AzS-DC01'.
2020-11-04 14:26:08 Verbose 5> [SQL:Configure] Record for 'True' (A) not present.
2020-11-04 14:26:08 Verbose 5> [SQL:Configure] Removing DNS resource record for name 'AzS-SqlCl' with IP 'True' under zone name 'azurestack.local' on the domain controller 'AzS-DC01'.
2020-11-04 14:26:08 Verbose 5> [SQL:Configure] Record for 'True' (AAAA) not present.
2020-11-04 14:26:08 Verbose 5> [SQL:Configure] Creating the cluster 'AzS-SqlCl'.
2020-11-04 14:26:19 Verbose 5> [SQL:Configure] Collecting client/diagnostic event log on AzS-Sql01.azurestack.local
2020-11-04 14:26:39 Verbose 5> [SQL:Configure] ECE PowerShellHost Script End
2020-11-04 14:26:39 Verbose 5> Interface: PowerShell job terminal state is Failed for interface 'Configure'.
2020-11-04 14:26:39 Verbose 5> [PSTask Concurrency] [113>SQL:Configure] Exiting throttling instance. Priority '0', Weight '10'.
2020-11-04 14:26:39 Verbose 5> [PSTask Concurrency] [113>SQL:Configure] Parallel task count decreased by '10' to '0'.
2020-11-04 14:26:39 Warning 5> Task: Invocation of interface 'Configure' of role 'Cloud\Fabric\SQL' failed:

Type 'Configure' of Role 'SQL' raised an exception:

There was an error adding node 'AzS-Sql01.azurestack.local' to the cluster
The computer 'AzS-Sql01.azurestack.local' is joined to a cluster.
Command Arguments

---

Add-GuestCluster {Parameters=CloudEngine.Configurations.EceInterfaceParameters, RunAsUserID=Fabric}
New-Sql {Parameters=CloudEngine.Configurations.EceInterfaceParameters, ErrorAction=Stop, Verbose=True}
{}
{CloudEngine.Configurations.EceInterfaceParameters}
{C:\CloudDeployment\CloudDeployment.psd1, CloudEngine.Configurations.EceInterfaceParameters, 1211131...

at Trace-Error, C:\CloudDeployment\Common\Tracer.psm1: line 63
at Add-GuestCluster, C:\CloudDeployment\Roles\Common\RoleHelpers.psm1: line 390
at New-Sql, C:\CloudDeployment\Roles\SQL\SQL.psm1: line 55
at Configure, C:\CloudDeployment\Classes\SQL\SQL.psm1: line 19
at , : line 56
at , : line 54`

[jdm019000]

Update after debugging all day yesterday.

1. Yesterday we found that the ASDK baseline installer script did not add the SQL cluster name DNS A record and IP to the DNS. It also did not add the SQL cluster name computer object to the AD. The script was looking for both of those and will fail if they are not there. Once we added them, then the rerun of the script progressed.
2. Last night the script ran from 5pm to around 10p and is now in a hung state. It did not exit and return to c prompt but seems to be hung. It is having issue with the storage accounts. It seems to be hung for around 12 hours now.
   a. The failover cluster manager events shows a recurring error.
   Cluster resource "Virtual Machine AzS-ACS01" of type "Virual Machine" in clustered role failed.
   Based on the failure policies for the resource and role, the cluster service may try to bring the resource online on
   the this node or move the group to another node and then restart it.
   Check the resouce and group state using failover cluster manager or the Get-ClusterResource Windows PowerShell cmdlet.
   b. The last stage on the PS console shows the below. The create logs statement C:\MASLogs\StorageAccounts_CreateStorageAccounts_20201105-022653.log on the server AzS-Xrp01.azurestack.local is 0 bytes empty. So not able to find anything. I am not sure what to do at this stage.

STARTING: Task Cloud\Fabric\StorageAccounts - Configure
VERBOSE: 1> Interface: Path to module: C:\CloudDeployment\Classes\StorageAccounts\StorageAccounts.psm1 - 11/5/2020 2:25:53 AM
VERBOSE: 1> Interface: Running interface Configure (Classes\StorageAccounts\StorageAccounts.psm1, StorageAccounts:Configure) - 11/5/2020 2:25:53 AM
VERBOSE: 1> [PSTask Concurrency] [63>StorageAccounts:Configure] Entering throttling instance, enqueue for waiting. Priority '0', Weight '10'. - 11/5/2020 2:25:53 AM
VERBOSE: 1> [PSTask Concurrency] [63>StorageAccounts:Configure] Validating availability to start task. - 11/5/2020 2:25:53 AM
VERBOSE: 1> [PSTask Concurrency] [63>StorageAccounts:Configure] Task available to start. - 11/5/2020 2:25:53 AM
VERBOSE: 1> [PSTask Concurrency] [63>StorageAccounts:Configure] Parallel task count increased by '10' to '10'. - 11/5/2020 2:25:53 AM
VERBOSE: 1> [PSTask Concurrency] [63>StorageAccounts:Configure] Throttling instance dequeued from waiting. Priority '0', Weight '10'. - 11/5/2020 2:25:53 AM
VERBOSE: 1> [PSTask Concurrency] [63>StorageAccounts:Configure] TotalVirtualMemoryMB: [131071.55078125], FreeVirtualMemoryMB: [59127.4296875],FreePhysicalMemoryMB: [59655.9296875]. - 11/5/2020 2:25:53 AM
WARNING: 1> [StorageAccounts:Configure] The names of some imported commands from the module 'StorageAccounts' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the
Verbose parameter. For a list of approved verbs, type Get-Verb. - 11/5/2020 2:26:21 AM
VERBOSE: 1> [StorageAccounts:Configure] Initializing remote powershell session on AzS-Xrp01.azurestack.local with common functions. - 11/5/2020 2:26:51 AM
VERBOSE: 1> [StorageAccounts:Configure] Transcript started, output file is C:\MASLogs\StorageAccounts_CreateStorageAccounts_20201105-022653.log - 11/5/2020 2:26:54 AM
VERBOSE: 1> [StorageAccounts:Configure] Registering the admin subscriptions [abaf2eb8-71b9-4d50-9c30-0c32668f6703 6d8d45aa-3e27-4914-96bb-23706ad8a66b] with the 'Microsoft.Storage' namespace - 11/5/2020 2:26:54 AM

[jdm019000]

The installer that seemed to hang finally failed after 17 hours at step 60.140.148.

We restarted/reset the AzS-XRP01 Role Node (Infrastructure management controller for Microsoft Azure Stack, including the Compute, Network, and Storage resource providers). And started the Installer again with the -rerun flag. It is now progressing again, and has moved passed the 60.140.148 step. Will see if it finishes.

error_step_60-140-148.txt

[rtibi]

has it completed this time?