Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New-AzKubernetesExtension - installing Flux fails with error "Failed to perform resource identity operation" #22455

Closed
janegilring opened this issue Aug 1, 2023 · 7 comments · Fixed by #24900 or #24903
Assignees
Labels
bug This issue requires a change to an existing behavior in the product in order to be resolved. Kubernetes Configuration Service Attention This issue is responsible by Azure service team.

Comments

@janegilring
Copy link

janegilring commented Aug 1, 2023

Description

I am trying to install the Flux extension on an AKS cluster using the following command:

New-AzKubernetesExtension -ClusterName $resourceName -ClusterType ManagedCluster -Name flux -ResourceGroupName $Env:resourceGroup -ExtensionType microsoft.flux -IdentityType 'SystemAssigned'

This fails with an identity-related error as shown below.

Installing the extension using Azure CLI works fine:

 az k8s-extension create --name "flux" --extension-type "microsoft.flux" --cluster-name $resourceName --resource-group $Env:resourceGroup --cluster-type $ClusterType 

Issue script & Debug output

PS C:\Windows\system32> New-AzKubernetesExtension -ClusterName $resourceName -ClusterType ManagedCluster -Name flux -ResourceGroupName $Env:resourceGroup -ExtensionType microsoft.flux -IdentityType 'SystemAssigned'
Az.KubernetesConfiguration.internal\New-AzKubernetesExtension : Identity operation for resource 
'/subscriptions/16471a83-9151-456e-bbb1-463027bed604/resourceGroups/aks-rg/providers/Microsoft.ContainerService/managedClusters/Ag-AKS-Staging/providers/Microsoft.KubernetesConfiguration/extensions/flux' failed with 
error 'Failed to perform resource identity operation. Status: 'BadRequest'. Response: '{"error":{"code":"BadRequest","message":""}}'.'.
At C:\Program Files\WindowsPowerShell\Modules\Az.KubernetesConfiguration\0.7.0\custom\New-AzKubernetesExtension.ps1:223 char:9
+         Az.KubernetesConfiguration.internal\New-AzKubernetesExtension ...
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: ({ SubscriptionI...gned"
  }
} }:<>f__AnonymousType4`7) [New-AzKubernetesExtension_CreateExpanded], Exception
    + FullyQualifiedErrorId : FailedIdentityOperation,Microsoft.Azure.PowerShell.Cmdlets.KubernetesConfiguration.Cmdlets.NewAzKubernetesExtension_CreateExpanded 

# If trying without specifying -IdentityType, the following is returned:

 PS C:\Windows\system32> New-AzKubernetesExtension -ClusterName $resourceName -ClusterType $ClusterType -Name flux -ResourceGroupName $Env:resourceGroup -ExtensionType microsoft.flux
New-AzKubernetesExtension : The value of the parameter -IdentityType must be 'SystemAssigned'.
At line:1 char:1
+ New-AzKubernetesExtension -ClusterName $resourceName -ClusterType $Cl ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,New-AzKubernetesExtension

Environment data

$PSVersionTable

Name                           Value                                                                                                                                                                                                                                                                                                                                                                                                                                               
----                           -----                                                                                                                                                                                                                                                                                                                                                                                                                                               
PSVersion                      5.1.20348.1850                                                                                                                                                                                                                                                                                                                                                                                                                                      
PSEdition                      Desktop                                                                                                                                                                                                                                                                                                                                                                                                                                             
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}                                                                                                                                                                                                                                                                                                                                                                                                                             
BuildVersion                   10.0.20348.1850                                                                                                                                                                                                                                                                                                                                                                                                                                     
CLRVersion                     4.0.30319.42000                                                                                                                                                                                                                                                                                                                                                                                                                                     
WSManStackVersion              3.0                                                                                                                                                                                                                                                                                                                                                                                                                                                 
PSRemotingProtocolVersion      2.3                                                                                                                                                                                                                                                                                                                                                                                                                                                 
SerializationVersion           1.1.0.1

Module versions

Get-Module Az*

ModuleType Version    Name                                ExportedCommands                                                                                                                                                                                                                                                                                                                                                                                                         
---------- -------    ----                                ----------------                                                                                                                                                                                                                                                                                                                                                                                                         
Script     2.12.5     Az.Accounts                         {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear-AzDefault...}                                                                                                                                                                                                                                                                                                                                                 
Script     0.7.0      Az.KubernetesConfiguration          {Get-AzKubernetesConfigFluxOperationStatus, Get-AzKubernetesConfiguration, Get-AzKubernetesConfigurationFlux, Get-AzKubernetesExtension...}                                                                                                                                                                                                                                                                              
Script     6.8.0      Az.Resources                        {Export-AzResourceGroup, Export-AzTemplateSpec, Get-AzDenyAssignment, Get-AzDeployment...}

Error output

Resolve-AzError


   HistoryId: 32


Message        : [FailedIdentityOperation] : Identity operation for resource '/subscriptions/16471a83-9151-456e-bbb1-463027bed604/resourceGroups/aks-rg/providers/Microsoft.ContainerService/managedClusters/Ag-AKS-Staging/providers/Microsoft.KubernetesConfiguration/extensions/flux' failed with error 'Failed to perform resource identity operation. Status: 'BadRequest'. Response: '{"error":{"code":"BadRequest","message":""}}'.'.
StackTrace     : 
Exception      : System.Exception
InvocationInfo : {New-AzKubernetesExtension_CreateExpanded}
Line           :         Az.KubernetesConfiguration.internal\New-AzKubernetesExtension @PSBoundParameters
                 
Position       : At C:\Program Files\WindowsPowerShell\Modules\Az.KubernetesConfiguration\0.7.0\custom\New-AzKubernetesExtension.ps1:223 char:9
                 +         Az.KubernetesConfiguration.internal\New-AzKubernetesExtension ...
                 +         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : 32
@janegilring janegilring added bug This issue requires a change to an existing behavior in the product in order to be resolved. needs-triage This is a new issue that needs to be triaged to the appropriate team. labels Aug 1, 2023
@ghost ghost removed the needs-triage This is a new issue that needs to be triaged to the appropriate team. label Aug 1, 2023
@isra-fel isra-fel added Service Attention This issue is responsible by Azure service team. Kubernetes Configuration labels Aug 3, 2023
@ghost
Copy link

ghost commented Aug 3, 2023

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @NarayanThiru.

Issue Details

Description

I am trying to install the Flux extension on an AKS cluster using the following command:

New-AzKubernetesExtension -ClusterName $resourceName -ClusterType ManagedCluster -Name flux -ResourceGroupName $Env:resourceGroup -ExtensionType microsoft.flux -IdentityType 'SystemAssigned'

This fails with an identity-related error as shown below.

Installing the extension using Azure CLI works fine:

 az k8s-extension create --name "flux" --extension-type "microsoft.flux" --cluster-name $resourceName --resource-group $Env:resourceGroup --cluster-type $ClusterType 

Issue script & Debug output

PS C:\Windows\system32> New-AzKubernetesExtension -ClusterName $resourceName -ClusterType ManagedCluster -Name flux -ResourceGroupName $Env:resourceGroup -ExtensionType microsoft.flux -IdentityType 'SystemAssigned'
Az.KubernetesConfiguration.internal\New-AzKubernetesExtension : Identity operation for resource 
'/subscriptions/16471a83-9151-456e-bbb1-463027bed604/resourceGroups/aks-rg/providers/Microsoft.ContainerService/managedClusters/Ag-AKS-Staging/providers/Microsoft.KubernetesConfiguration/extensions/flux' failed with 
error 'Failed to perform resource identity operation. Status: 'BadRequest'. Response: '{"error":{"code":"BadRequest","message":""}}'.'.
At C:\Program Files\WindowsPowerShell\Modules\Az.KubernetesConfiguration\0.7.0\custom\New-AzKubernetesExtension.ps1:223 char:9
+         Az.KubernetesConfiguration.internal\New-AzKubernetesExtension ...
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: ({ SubscriptionI...gned"
  }
} }:<>f__AnonymousType4`7) [New-AzKubernetesExtension_CreateExpanded], Exception
    + FullyQualifiedErrorId : FailedIdentityOperation,Microsoft.Azure.PowerShell.Cmdlets.KubernetesConfiguration.Cmdlets.NewAzKubernetesExtension_CreateExpanded 

# If trying without specifying -IdentityType, the following is returned:

 PS C:\Windows\system32> New-AzKubernetesExtension -ClusterName $resourceName -ClusterType $ClusterType -Name flux -ResourceGroupName $Env:resourceGroup -ExtensionType microsoft.flux
New-AzKubernetesExtension : The value of the parameter -IdentityType must be 'SystemAssigned'.
At line:1 char:1
+ New-AzKubernetesExtension -ClusterName $resourceName -ClusterType $Cl ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,New-AzKubernetesExtension

Environment data

$PSVersionTable

Name                           Value                                                                                                                                                                                                                                                                                                                                                                                                                                               
----                           -----                                                                                                                                                                                                                                                                                                                                                                                                                                               
PSVersion                      5.1.20348.1850                                                                                                                                                                                                                                                                                                                                                                                                                                      
PSEdition                      Desktop                                                                                                                                                                                                                                                                                                                                                                                                                                             
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}                                                                                                                                                                                                                                                                                                                                                                                                                             
BuildVersion                   10.0.20348.1850                                                                                                                                                                                                                                                                                                                                                                                                                                     
CLRVersion                     4.0.30319.42000                                                                                                                                                                                                                                                                                                                                                                                                                                     
WSManStackVersion              3.0                                                                                                                                                                                                                                                                                                                                                                                                                                                 
PSRemotingProtocolVersion      2.3                                                                                                                                                                                                                                                                                                                                                                                                                                                 
SerializationVersion           1.1.0.1

Module versions

Get-Module Az*

ModuleType Version    Name                                ExportedCommands                                                                                                                                                                                                                                                                                                                                                                                                         
---------- -------    ----                                ----------------                                                                                                                                                                                                                                                                                                                                                                                                         
Script     2.12.5     Az.Accounts                         {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear-AzDefault...}                                                                                                                                                                                                                                                                                                                                                 
Script     0.7.0      Az.KubernetesConfiguration          {Get-AzKubernetesConfigFluxOperationStatus, Get-AzKubernetesConfiguration, Get-AzKubernetesConfigurationFlux, Get-AzKubernetesExtension...}                                                                                                                                                                                                                                                                              
Script     6.8.0      Az.Resources                        {Export-AzResourceGroup, Export-AzTemplateSpec, Get-AzDenyAssignment, Get-AzDeployment...}

Error output

Resolve-AzError


   HistoryId: 32


Message        : [FailedIdentityOperation] : Identity operation for resource '/subscriptions/16471a83-9151-456e-bbb1-463027bed604/resourceGroups/aks-rg/providers/Microsoft.ContainerService/managedClusters/Ag-AKS-Staging/providers/Microsoft.KubernetesConfiguration/extensions/flux' failed with error 'Failed to perform resource identity operation. Status: 'BadRequest'. Response: '{"error":{"code":"BadRequest","message":""}}'.'.
StackTrace     : 
Exception      : System.Exception
InvocationInfo : {New-AzKubernetesExtension_CreateExpanded}
Line           :         Az.KubernetesConfiguration.internal\New-AzKubernetesExtension @PSBoundParameters
                 
Position       : At C:\Program Files\WindowsPowerShell\Modules\Az.KubernetesConfiguration\0.7.0\custom\New-AzKubernetesExtension.ps1:223 char:9
                 +         Az.KubernetesConfiguration.internal\New-AzKubernetesExtension ...
                 +         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : 32
Author: janegilring
Assignees: -
Labels:

Service Attention, bug, Kubernetes Configuration

Milestone: -

@bavneetsingh16
Copy link

I am currently collaborating with the PowerShell team to resolve the issue. I will update this thread as soon as I have more information.

@Alex-wdy
Copy link
Contributor

@lijinpei2008 Can you also pay attention to this issue? Can it be fixed?

@lijinpei2008
Copy link
Contributor

Okay, I'm trying to fix it.

@lijinpei2008
Copy link
Contributor

First of all, this error is expected.
This is because if the IdentityType value SystemAssigned is not provided when creating the Flux resource, the New-AzKubernetesExtension command will run for a long time and eventually throw a timeout error.

Also, creating a Flux resource with the command New-AzKubernetesExtension will be mutually exclusive with the resource created with the command New-AzKubernetesConfiguration.

Finally, when creating a Flux resource, you need to provide the parameter -AutoUpgradeMinorVersion and the parameter -ReleaseNamespace.

Here is the example:

  • Create Kubernetes service
  • Create Kubernetes - Azure Arc
  • Run command: New-AzKubernetesExtension -ClusterName azpstest_cluster_arc -ClusterType ConnectedClusters -Name flux -ResourceGroupName azps_test_group -ExtensionType microsoft.flux -AutoUpgradeMinorVersion -ReleaseNamespace flux-system -IdentityType 'SystemAssigned'

Help docs link:
https://learn.microsoft.com/en-us/powershell/module/az.kubernetesconfiguration/new-azkubernetesextension?view=azps-11.5.0

@bavneetsingh16
Copy link

bavneetsingh16 commented May 11, 2024

@lijinpei2008, the ‘IdentityType’ is specifically applicable for ‘connectedClusters’. However, in this context, we are dealing with an AKS cluster, also known as ‘managedClusters’. For this cluster type, it is not required to provide the ‘identity’ parameter. This is because AKS autonomously generates the identity for its extensions. The 'IdentityType' param should be mandatory only for connectedClusters but not in the case of managedClusters.

@lijinpei2008
Copy link
Contributor

Thanks @bavneetsingh16 for the explanation, after several attempts I found what you described to be correct.
I will fix this issue in the near future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue requires a change to an existing behavior in the product in order to be resolved. Kubernetes Configuration Service Attention This issue is responsible by Azure service team.
Projects
None yet
5 participants