[BUG] Publishing APIs fails with errors 409 PessimisticConcurrencyConflict or 422 ManagementApiRequestFailed #497
Comments
|
|
Perhaps this is related to numer of objects being published - maybe we hit hardware limits, management API quotas, West EU resource shortage?
|
|
Since these look like transient errors, maybe there should be retry policy implemented somewhere here: https://github.com/Azure/apiops/blob/156314ff120d1ae7ccbc392699a0fd09270a611d/tools/code/publisher/Program.cs#L193C1-L194C1 and number of retries provided as configuration parameter. |
|
I feel like this topic has been discussed before. Can you please take a look at the closed issues and if you can't find anything, let me know and I will loop in the lead dev on this. |
|
It was discussed under 141 and was fixed. It seems we are facing similar problem with apiops v5. The last comment in 141 mentioned that adding versionsets may cause it - will check and let you know tomorrow. |
|
If that resolves it please remember to tag this as a duplicate before you close. |
|
We did couple of runs to find any pattern, unfortunately it is still failing on random APIs. Below call stacks of both types of errors: crit: Publisher[0]
build 04-Mar-2024 11:38:08 System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/XXXXXXXXXXXX/resourceGroups/XXXXXXXXX/providers/Microsoft.ApiManagement/service/xxxxxxxxxx/apis/ksef-invoice?api-version=2022-04-01-preview failed with status code 409. Content is '{"error":{"code":"PessimisticConcurrencyConflict","message":"Operation on the API is in progress","details":null}}'.
build 04-Mar-2024 11:38:08 at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri)
build 04-Mar-2024 11:38:08 at common.HttpPipelineExtensions.PutResource(HttpPipeline pipeline, Uri uri, JsonObject resource, CancellationToken cancellationToken)
build 04-Mar-2024 11:38:08 at publisher.Program.<>c__DisplayClass12_0.<<GetPutRestResource>b__0>d.MoveNext()
build 04-Mar-2024 11:38:08 --- End of stack trace from previous location ---
build 04-Mar-2024 11:38:08 at publisher.Api.PutApi(ApiName apiName, ApiInformationFile apiInformationFile, ApiSpecificationFile specificationFile, JsonObject configurationApiJson, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
build 04-Mar-2024 11:38:08 at publisher.Api.<>c__DisplayClass23_0.<<ProcessArtifactsToPut>b__4>d.MoveNext()
build 04-Mar-2024 11:38:08 --- End of stack trace from previous location ---
build 04-Mar-2024 11:38:08 at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()
build 04-Mar-2024 11:38:08 --- End of stack trace from previous location ---
build 04-Mar-2024 11:38:08 at common.IEnumerableExtensions.ForEachParallel[T](IEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken)
build 04-Mar-2024 11:38:08 at publisher.Api.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
build 04-Mar-2024 11:38:08 at publisher.Service.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken)
build 04-Mar-2024 11:38:08 at publisher.Publisher.RunWithoutCommitId(CancellationToken cancellationToken)
build 04-Mar-2024 11:38:08 at publisher.Publisher.Run(CancellationToken cancellationToken)
build 04-Mar-2024 11:38:08 at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken)
build 04-Mar-2024 11:38:08 info: Microsoft.Hosting.Lifetime[0]
build 04-Mar-2024 11:38:08 Application is shutting down...
build 04-Mar-2024 11:38:08 fail: Microsoft.Extensions.Hosting.Internal.Host[9]
build 04-Mar-2024 11:38:08 BackgroundService failed
build 04-Mar-2024 11:38:08 System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/XXXXXXXXXXXX/resourceGroups/XXXXXXXXX/providers/Microsoft.ApiManagement/service/xxxxxxxxxx/apis/ksef-invoice?api-version=2022-04-01-preview failed with status code 409. Content is '{"error":{"code":"PessimisticConcurrencyConflict","message":"Operation on the API is in progress","details":null}}'.
build 04-Mar-2024 11:38:08 at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri)
build 04-Mar-2024 11:38:08 at common.HttpPipelineExtensions.PutResource(HttpPipeline pipeline, Uri uri, JsonObject resource, CancellationToken cancellationToken)
build 04-Mar-2024 11:38:08 at publisher.Program.<>c__DisplayClass12_0.<<GetPutRestResource>b__0>d.MoveNext()
build 04-Mar-2024 11:38:08 --- End of stack trace from previous location ---
build 04-Mar-2024 11:38:08 at publisher.Api.PutApi(ApiName apiName, ApiInformationFile apiInformationFile, ApiSpecificationFile specificationFile, JsonObject configurationApiJson, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
build 04-Mar-2024 11:38:08 at publisher.Api.<>c__DisplayClass23_0.<<ProcessArtifactsToPut>b__4>d.MoveNext()
build 04-Mar-2024 11:38:08 --- End of stack trace from previous location ---
build 04-Mar-2024 11:38:08 at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()
build 04-Mar-2024 11:38:08 --- End of stack trace from previous location ---
build 04-Mar-2024 11:38:08 at common.IEnumerableExtensions.ForEachParallel[T](IEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken)
build 04-Mar-2024 11:38:08 at publisher.Api.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
build 04-Mar-2024 11:38:08 at publisher.Service.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken)
build 04-Mar-2024 11:38:08 at publisher.Publisher.RunWithoutCommitId(CancellationToken cancellationToken)
build 04-Mar-2024 11:38:08 at publisher.Publisher.Run(CancellationToken cancellationToken)
build 04-Mar-2024 11:38:08 at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken)
build 04-Mar-2024 11:38:08 at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
build 04-Mar-2024 11:38:08 dbug: Microsoft.Extensions.Hosting.Internal.Host[3]
build 04-Mar-2024 11:38:08 Hosting stopping
build 04-Mar-2024 11:38:08 crit: Microsoft.Extensions.Hosting.Internal.Host[10]
build 04-Mar-2024 11:38:08 The HostOptions.BackgroundServiceExceptionBehavior is configured to StopHost. A BackgroundService has thrown an unhandled exception, and the IHost instance is stopping. To avoid this behavior, configure this to Ignore; however the BackgroundService will not be restarted.
build 04-Mar-2024 11:38:08 System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/XXXXXXXXXXXX/resourceGroups/XXXXXXXXX/providers/Microsoft.ApiManagement/service/xxxxxxxxxx/apis/ksef-invoice?api-version=2022-04-01-previewfailed with status code 409. Content is '{"error":{"code":"PessimisticConcurrencyConflict","message":"Operation on the API is in progress","details":null}}'.
build 04-Mar-2024 11:38:08 at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri)
build 04-Mar-2024 11:38:08 at common.HttpPipelineExtensions.PutResource(HttpPipeline pipeline, Uri uri, JsonObject resource, CancellationToken cancellationToken)
build 04-Mar-2024 11:38:08 at publisher.Program.<>c__DisplayClass12_0.<<GetPutRestResource>b__0>d.MoveNext()
build 04-Mar-2024 11:38:08 --- End of stack trace from previous location ---
build 04-Mar-2024 11:38:08 at publisher.Api.PutApi(ApiName apiName, ApiInformationFile apiInformationFile, ApiSpecificationFile specificationFile, JsonObject configurationApiJson, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
build 04-Mar-2024 11:38:08 at publisher.Api.<>c__DisplayClass23_0.<<ProcessArtifactsToPut>b__4>d.MoveNext()
build 04-Mar-2024 11:38:08 --- End of stack trace from previous location ---
build 04-Mar-2024 11:38:08 at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()
build 04-Mar-2024 11:38:08 --- End of stack trace from previous location ---
build 04-Mar-2024 11:38:08 at common.IEnumerableExtensions.ForEachParallel[T](IEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken)
build 04-Mar-2024 11:38:08 at publisher.Api.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
build 04-Mar-2024 11:38:08 at publisher.Service.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken)
build 04-Mar-2024 11:38:08 at publisher.Publisher.RunWithoutCommitId(CancellationToken cancellationToken)
build 04-Mar-2024 11:38:08 at publisher.Publisher.Run(CancellationToken cancellationToken)
build 04-Mar-2024 11:38:08 at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken)
build 04-Mar-2024 11:38:08 at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
build 04-Mar-2024 11:38:08 dbug: Microsoft.Extensions.Hosting.Internal.Host[4]
build 04-Mar-2024 11:38:08 Hosting stoppedand 422: failed with status code 422. Content is '{"error":{"code":"ManagementApiRequestFailed","message":"Failed to connect to management endpoint xxxxx.management.azure-api.net:3443 for a service deployed in a Virtual Network. Make sure to follow guidance at https://aka.ms/apim-vnet-common-issues for Inbound connectivity to Management endpoint. Check 'ApiManagement Control Plane - inbound' connectivity at [https://aka.ms/apimnetworkstatus.","details":null,"innerError":null}](https://aka.ms/apimnetworkstatus.%22,%22details%22:null,%22innerError%22:null%7d)}'.
build 04-Mar-2024 09:28:40 at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri)
build 04-Mar-2024 09:28:40 at common.HttpPipelineExtensions.WaitForLongRunningOperation(HttpPipeline pipeline, Response response, CancellationToken cancellationToken)
build 04-Mar-2024 09:28:40 at common.HttpPipelineExtensions.PutResource(HttpPipeline pipeline, Uri uri, JsonObject resource, CancellationToken cancellationToken)
build 04-Mar-2024 09:28:40 at publisher.Program.<>c__DisplayClass12_0.<<GetPutRestResource>b__0>d.MoveNext()
build 04-Mar-2024 09:28:40 --- End of stack trace from previous location ---
build 04-Mar-2024 09:28:40 at publisher.Api.PutApi(ApiName apiName, ApiInformationFile apiInformationFile, ApiSpecificationFile specificationFile, JsonObject configurationApiJson, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
build 04-Mar-2024 09:28:40 at publisher.Api.<>c__DisplayClass23_0.<<ProcessArtifactsToPut>b__4>d.MoveNext()
build 04-Mar-2024 09:28:40 --- End of stack trace from previous location ---
build 04-Mar-2024 09:28:40 at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()
build 04-Mar-2024 09:28:40 --- End of stack trace from previous location ---
build 04-Mar-2024 09:28:40 at common.IEnumerableExtensions.ForEachParallel[T](IEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken)
build 04-Mar-2024 09:28:40 at publisher.Api.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
build 04-Mar-2024 09:28:40 at publisher.Service.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken)
build 04-Mar-2024 09:28:40 at publisher.Publisher.RunWithoutCommitId(CancellationToken cancellationToken)
build 04-Mar-2024 09:28:40 at publisher.Publisher.Run(CancellationToken cancellationToken)
build 04-Mar-2024 09:28:40 at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken)
build 04-Mar-2024 09:28:40 at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
build 04-Mar-2024 09:28:40 dbug: Microsoft.Extensions.Hosting.Internal.Host[3]
build 04-Mar-2024 09:28:40 Hosting stopping
build 04-Mar-2024 09:28:40 crit: Microsoft.Extensions.Hosting.Internal.Host[10]
build 04-Mar-2024 09:28:40 The HostOptions.BackgroundServiceExceptionBehavior is configured to StopHost. A BackgroundService has thrown an unhandled exception, and the IHost instance is stopping. To avoid this behavior, configure this to Ignore; however the BackgroundService will not be restarted. |
|
@mgrabarz - the code should already use these default retry settings. Your 422 error message indicates a connectivity failure with your APIM service that lives in a VNET. If you enable Trace logging, you should see the contents of each PUT request (including the ones that fail). Can you share the contents of one of those failing API calls? Please sanitize the logs before sharing. |
|
When sanitising the PUT logs I just noticed that API being put at References the following VersionSet in the body: "apiVersionSetId": "/subscriptions/subId/resourceGroups/xxx-resougroup-dev-1/providers/Microsoft.ApiManagement/service/xxx-APIM-dev-1/apiVersionSets/yyy Not sure if this is generating the errors and what is the root cause of these being part of the body (extractor maybe, or pull request error), but need to investigate. I will reach back to you with findings. |
|
The code is exported from the DEV environment, so a snapshot of what is set in DEV is pulling to bitbucket. When publishing, we indicate the configuration file with which we overwrite some values, e.g. backends, namedValues, diagnostics. We follow https://github.com/Azure/apiops/blob/main/configuration.prod.yaml |
|
I replaced the references in the code before publishing. The same errors appear, so these references do not matter, especially since the publication is successful in a different environment (SIT). |
|
Sample http 422 error with trace=true when publishing API: crit: Publisher[0] |
|
Sample http 409 error form publisher trace: info: Publisher[0] |
|
@adkasperski - The version set issue makes sense. You can't publish an API and reference a version set in another APIM instance. You should be able to override any property supported by the REST API in your environment's configuration file. For the version set ID, I expect it will look something like this: apis
- name: myapi
properties:
apiVersionSetId: yourversionsetidAs for last comment, this is not an ApiOps issue - has to do with the fact that you're using the Developer SKU (which has no SLA). Every now and then, the APIM team makes changes to the platform, and instances using that SKU are unavailable while the changes happen. You'll just have to wait and try again later, or use a different SKU. |
|
One other thing when overriding: I've noticed that many APIM resource ID properties also support a shorter format if it's in the same instance. Rather than specifying a full version set resource ID that includes the subscription, resource group, and instance name |
|
@adkasperski please let us know if the shortened format that @guythetechie is suggesting works so we can update our wiki. |
|
Thanks. I'll check the @guythetechie suggestion and get back the result. |
|
The following configuration works correctly: apis:
On the other hand if we shorten the API specification, the publisher will also work correctly: { But it does not resolve the problem. Errors still appear. I think this is not an APIOPs problem, but a problem with the availability of the APIM service. We've already created the ticket to Microsoft APIM Support. |
|
We have noticed that in UAT/PRE environments where these errors occur, the publishing time of individual, even small API components (named values, backends, version set, etc.) takes significantly longer than in the PRD environment where these errors do not occur. Up to several dozen API components are published per second on PRD, while on UAT/PRE it takes more than 10 seconds to publish a single component. We read the publication durations from the Activity Logs in the Management Plane of the APIM service. |
|
Just as a piece of information, but every time we receive a 409 in the publisher pipeline, the APIM Management endpoint seems also irresponsive, so much so I receive an error if I try to open the APIs definitions from the Azure Portal. It happens every time |
|
Please see this section in my response. The developer SKU often goes offline for maintenance. I would recommend using non-developer SKUs if you need more availability.
|
|
Just to be sure, only the management portal goes sometimes offline while the gateway still works? I'm asking because when happened to us, request where processed correctly even if the pipeline was failing for those errors. |
|
We use Developer SKU for non-production environments. So we can close this issue. |
Release version
APIOPS v5.0.0
Describe the bug
Publishing process fails on the random API (all objects before are deployed fine) with one of two errors:
System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/sub/resourceGroups/apim-name/providers/Microsoft.ApiManagement/service/apim-name/apis/api-name?api-version=2022-04-01-preview failed with status code 422. Content is '{"error":{"code":"ManagementApiRequestFailed","message":"Failed to connect to management endpoint apim-name.management.azure-api.net:3443 for a service deployed in a Virtual Network. Make sure to follow guidance at https://aka.ms/apim-vnet-common-issues for Inbound connectivity to Management endpoint. Check 'ApiManagement Control Plane - inbound' connectivity at https://aka.ms/apimnetworkstatus.","details":null,"innerError":null}}'.System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/sub/resourceGroups/apim-name/providers/Microsoft.ApiManagement/service/apim-name/apis/another-api-name?api-version=2022-04-01-preview failed with status code 409. Content is '{"error":{"code":"PessimisticConcurrencyConflict","message":"Operation on the API is in progress","details":null}}'. build 27-Feb-2024 10:57:33 at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri)Expected behavior
We would like to be able to run publishing without errors.
Actual behavior
We have a quite large list of objects (APIs, products, tags, NVs, backends) we publish with APIOPS. Unfortunately publishing process fails on the random API (all objects before are deployed fine) with one of two errors:
System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/sub/resourceGroups/apim-name/providers/Microsoft.ApiManagement/service/apim-name/apis/api-name?api-version=2022-04-01-preview failed with status code 422. Content is '{"error":{"code":"ManagementApiRequestFailed","message":"Failed to connect to management endpoint apim-name.management.azure-api.net:3443 for a service deployed in a Virtual Network. Make sure to follow guidance at https://aka.ms/apim-vnet-common-issues for Inbound connectivity to Management endpoint. Check 'ApiManagement Control Plane - inbound' connectivity at https://aka.ms/apimnetworkstatus.","details":null,"innerError":null}}'.System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/sub/resourceGroups/apim-name/providers/Microsoft.ApiManagement/service/apim-name/apis/another-api-name?api-version=2022-04-01-preview failed with status code 409. Content is '{"error":{"code":"PessimisticConcurrencyConflict","message":"Operation on the API is in progress","details":null}}'. build 27-Feb-2024 10:57:33 at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri)Interestingly this was happening rarely on all our environments, but recently (since a month or so) it almost always fails on our UAT instance of APIM, but other environments are still affected at 1/10 of deployments.
Reproduction Steps
We cannot share contents of our repo here, but can run it during the online meeting.
The text was updated successfully, but these errors were encountered: