Conflict of Newtonsoft.Json between PSES and Azure PowerShell

[isra-fel]

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest version
• Search the existing issues.

Steps to reproduce

Original issues: PowerShell/PowerShell#19646 , Azure/azure-powershell#21811

To reproduce, install the latest "Az" module and run the following script in a Windows PowerShell session hosted by the PowerShell extension of vscode.

$AzureContext = (Connect-AzAccount).context
Select-AzSubscription -Context $AzureContext
$global:AzStorageContext = (Get-AzStorageAccount).Context #<--- error happens here in visual studio code

Expected behavior

Should be able to assign storage account context

Actual behavior

Get-AzStorageAccount : Method 'get_SerializationSettings' in type 'Microsoft.Azure.Management.Storage.StorageManagementClient' from assembly 'Microsoft.Azure.PowerShell.Storage.Management.Sdk, Version=5.6.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' does not have an implementation.

Error details

Get-AzStorageAccount : Method 'get_SerializationSettings' in type 'Microsoft.Azure.Management.Internal.Resources.ResourceManagementClient' from assembly 
'Microsoft.Azure.PowerShell.Clients.ResourceManager, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' does not have an implementation.
At line:1 char:1
+ Get-AzStorageAccount
+ ~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Get-AzStorageAccount], TypeLoadException
    + FullyQualifiedErrorId : System.TypeLoadException,Microsoft.Azure.Commands.Management.Storage.GetAzureStorageAccountCommand

Environment data

Name                           Value
----                           -----
PSVersion                      5.1.22621.963
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.22621.963
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Version

3.8.4, the one comes with vscode PowerShell extension 2023.5.0

Visuals

No response

[isra-fel]

The issue was due to a conflict of Newtonsoft.Json between the two modules. PSES ships with 11.0.2 while Azure PowerShell depends on 13.0.2.

FullName : Newtonsoft.Json, Version=11.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed
Location : C:\Users\.\.vscode\extensions\ms-vscode.powershell-2023.5.0\modules\PowerShellEditorServices\bin\Common\Newtonsoft.Json.dll

Please consider updating Newtonsoft.Json to the latest, not only so that it can resolve the conflict, but also because all versions under 13 are known to be vulnerable: https://www.nuget.org/packages/Newtonsoft.Json

[andyleejordan]

Well moreover, you're using Windows PowerShell, which does not have an assembly load context in .NET, and therefore cannot isolate dependencies. Unfortunately, this can only be "fixed" by using PowerShell 7 (which has .NET Core and therefore can isolate dependencies). Please try that, and let me know!

[isra-fel]

@andschwa thanks for the quick response 🫡 Yes, I did inform the customer that switching to PS7 would resolve the problem, but it would also be great if you could share your plans on upgrading outdated dependencies.

[andyleejordan]

We don't actually directly depend on NewtonSoft.Json (i.e. it's not in any of our .csproj files). It's inherited from the OmniSharp library that provides our LSP layer: https://github.com/OmniSharp/csharp-language-server-protocol/blob/42527e922d4ee59cadd028409680f00c7e0b4138/Directory.Packages.props#L38

[StevieLamb]

Well moreover, you're using Windows PowerShell, which does not have an assembly load context in .NET, and therefore cannot isolate dependencies. Unfortunately, this can only be "fixed" by using PowerShell 7 (which has .NET Core and therefore can isolate dependencies). Please try that, and let me know!

Sorry, but that is not a viable solution for the reported problems.

Azure Automation only supports Windows PowerShell for production - v7.x runtimes are in preview, v7.1 has bene there for roughly a year. And these issues are being reported by customers who are developing Azure Automation Runbooks, so this is a crucial factor.

Also, the Endpoint Manager (Intune MDM) - as well as others, no doubt - offer no option to target the PowerShell v7 runtime when deploying scripts using either base PowerShell scripts or Proactive Remediations.

[andyleejordan]

That is an issue to take up with those products unfortunately. They are targeting software which is in maintenance mode only (Windows PowerShell), mostly because it has unsolvable issues around assembly dependency conflicts. It is architecturally not possible to fix without moving from .NET Framework to .NET Core, which is what happened with PowerShell Core.

[isra-fel]

Update: omnisharp LSP has released v0.19.9 and updated Newtonsoft.Json to v13.0.3.

Could you add this (update dependency to omnisharp LSP) to your plan? Thanks in advance!

[andyleejordan]

It's in the works!

[andyleejordan]

See #2083

[andyleejordan]

@isra-fel have you had a chance to test the new pre-release, v2023.9.3-preview? I included this update and I hope resolved your issue!

[e-jr]

Solution: Enable-AzureRmAlias -Scope LocalMachine

The error was gone for me when I enabled AzureRmAlias which is officially the compatibility mode for migrating from AzureRM to AzurePowershell https://learn.microsoft.com/en-us/powershell/azure/migrate-from-azurerm-to-az?view=azps-11.3.0#option-2-use-compatibility-mode-with-enable-azurermalias. I restarted my application (in my case a windows service) and it worked.

Not sure if if this means that the newest AzurePowershell version has a dependency to AzureRM and needs this alias or why I need this enabled.