Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Publisher is failing with "409. Content is '{"error":{"code":"PessimisticConcurrencyConflict","message":"Operation on the API is in progress","details":null}}'" When publishing on an already published APIM #141

Closed
Gabriel123N opened this issue Nov 14, 2022 · 15 comments
Closed

[BUG] Publisher is failing with "409. Content is '{"error":{"code":"PessimisticConcurrencyConflict","message":"Operation on the API is in progress","details":null}}'" When publishing on an already published APIM #141

Gabriel123N opened this issue Nov 14, 2022 · 15 comments
Labels
bug Something isn't working

Comments

@Gabriel123N
Copy link
Contributor

Gabriel123N commented Nov 14, 2022 •
edited
Loading

Release version

v2

Describe the bug

When deploying the publisher for a second time, the pipeline is failing with the following error at several points:

2022-11-14T12:31:58.0180514Z       System.InvalidOperationException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/rg/providers/Microsoft.ApiManagement/service/tapim/apis/api?api-version=2021-12-01-preview failed with status code 409. Content is '{"error":{"code":"PessimisticConcurrencyConflict","message":"Operation on the API is in progress","details":null}}'.
2022-11-14T12:31:58.0182453Z          at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/common/Http.cs:line 85
2022-11-14T12:31:58.0183502Z          at common.HttpPipelineExtensions.PutResource(HttpPipeline pipeline, Uri uri, JsonObject resource, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/common/Http.cs:line 71
2022-11-14T12:31:58.0184544Z          at publisher.Program.<>c__DisplayClass17_0.<<GetPutRestResource>b__0>d.MoveNext() in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Program.cs:line 229
2022-11-14T12:31:58.0185500Z       --- End of stack trace from previous location ---
2022-11-14T12:31:58.0186715Z          at publisher.Api.PutApi(ApiName apiName, ApiInformationFile apiInformationFile, ApiSpecificationFile specificationFile, JsonObject configurationApiJson, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Api.cs:line 273
2022-11-14T12:31:58.0187971Z          at publisher.Api.<>c__DisplayClass23_0.<<ProcessArtifactsToPut>b__4>d.MoveNext() in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Api.cs:line 371
2022-11-14T12:31:58.0188863Z       --- End of stack trace from previous location ---
2022-11-14T12:31:58.0189433Z          at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()
2022-11-14T12:31:58.0190158Z       --- End of stack trace from previous location ---
2022-11-14T12:31:58.0190982Z          at common.IEnumerableExtensions.ForEachParallel[T](IEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/common/Functional.cs:line 45
2022-11-14T12:31:58.0192306Z          at publisher.Api.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Api.cs:line 365
2022-11-14T12:31:58.0193954Z          at publisher.Service.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Service.cs:line 51
2022-11-14T12:31:58.0195329Z          at publisher.Publisher.RunWithoutCommitId(CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Publisher.cs:line 84
2022-11-14T12:31:58.0196260Z          at publisher.Publisher.Run(CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Publisher.cs:line 69
2022-11-14T12:31:58.0197437Z          at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Publisher.cs:line 47
2022-11-14T12:31:58.0202917Z info: Microsoft.Hosting.Lifetime[0]
2022-11-14T12:31:58.0203450Z       Application is shutting down...
2022-11-14T12:31:58.0221292Z fail: Microsoft.Extensions.Hosting.Internal.Host[9]
2022-11-14T12:31:58.0221867Z       BackgroundService failed
2022-11-14T12:31:58.0223869Z       System.InvalidOperationException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/rg/providers/Microsoft.ApiManagement/service/apim/apis/api?api-version=2021-12-01-preview failed with status code 409. Content is '{"error":{"code":"PessimisticConcurrencyConflict","message":"Operation on the API is in progress","details":null}}'.
2022-11-14T12:31:58.0225521Z          at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/common/Http.cs:line 85
2022-11-14T12:31:58.0226535Z          at common.HttpPipelineExtensions.PutResource(HttpPipeline pipeline, Uri uri, JsonObject resource, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/common/Http.cs:line 71
2022-11-14T12:31:58.0227580Z          at publisher.Program.<>c__DisplayClass17_0.<<GetPutRestResource>b__0>d.MoveNext() in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Program.cs:line 229
2022-11-14T12:31:58.0228506Z       --- End of stack trace from previous location ---
2022-11-14T12:31:58.0229689Z          at publisher.Api.PutApi(ApiName apiName, ApiInformationFile apiInformationFile, ApiSpecificationFile specificationFile, JsonObject configurationApiJson, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Api.cs:line 273
2022-11-14T12:31:58.0230946Z          at publisher.Api.<>c__DisplayClass23_0.<<ProcessArtifactsToPut>b__4>d.MoveNext() in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Api.cs:line 371
2022-11-14T12:31:58.0231842Z       --- End of stack trace from previous location ---
2022-11-14T12:31:58.0232425Z          at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()
2022-11-14T12:31:58.0233151Z       --- End of stack trace from previous location ---
2022-11-14T12:31:58.0233962Z          at common.IEnumerableExtensions.ForEachParallel[T](IEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/common/Functional.cs:line 45
2022-11-14T12:31:58.0235282Z          at publisher.Api.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Api.cs:line 365
2022-11-14T12:31:58.0238305Z          at publisher.Service.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Service.cs:line 51
2022-11-14T12:31:58.0239563Z          at publisher.Publisher.RunWithoutCommitId(CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Publisher.cs:line 84
2022-11-14T12:31:58.0240378Z          at publisher.Publisher.Run(CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Publisher.cs:line 69
2022-11-14T12:31:58.0241170Z          at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Publisher.cs:line 47
2022-11-14T12:31:58.0241899Z          at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
2022-11-14T12:31:58.0257828Z crit: Microsoft.Extensions.Hosting.Internal.Host[10]
2022-11-14T12:31:58.0258677Z       The HostOptions.BackgroundServiceExceptionBehavior is configured to StopHost. A BackgroundService has thrown an unhandled exception, and the IHost instance is stopping. To avoid this behavior, configure this to Ignore; however the BackgroundService will not be restarted.
2022-11-14T12:31:58.0261300Z       System.InvalidOperationException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/rg/providers/Microsoft.ApiManagement/service/apim/apisapi?api-version=2021-12-01-preview failed with status code 409. Content is '{"error":{"code":"PessimisticConcurrencyConflict","message":"Operation on the API is in progress","details":null}}'.
2022-11-14T12:31:58.0262571Z          at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/common/Http.cs:line 85
2022-11-14T12:31:58.0263459Z          at common.HttpPipelineExtensions.PutResource(HttpPipeline pipeline, Uri uri, JsonObject resource, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/common/Http.cs:line 71
2022-11-14T12:31:58.0264383Z          at publisher.Program.<>c__DisplayClass17_0.<<GetPutRestResource>b__0>d.MoveNext() in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Program.cs:line 229
2022-11-14T12:31:58.0265293Z       --- End of stack trace from previous location ---
2022-11-14T12:31:58.0266215Z          at publisher.Api.PutApi(ApiName apiName, ApiInformationFile apiInformationFile, ApiSpecificationFile specificationFile, JsonObject configurationApiJson, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Api.cs:line 273
2022-11-14T12:31:58.0267341Z          at publisher.Api.<>c__DisplayClass23_0.<<ProcessArtifactsToPut>b__4>d.MoveNext() in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Api.cs:line 371
2022-11-14T12:31:58.0268088Z       --- End of stack trace from previous location ---
2022-11-14T12:31:58.0268528Z          at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()
2022-11-14T12:31:58.0269128Z       --- End of stack trace from previous location ---
2022-11-14T12:31:58.0269819Z          at common.IEnumerableExtensions.ForEachParallel[T](IEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/common/Functional.cs:line 45
2022-11-14T12:31:58.0271002Z          at publisher.Api.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Api.cs:line 365
2022-11-14T12:31:58.0272518Z          at publisher.Service.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Service.cs:line 51
2022-11-14T12:31:58.0273767Z          at publisher.Publisher.RunWithoutCommitId(CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Publisher.cs:line 84
2022-11-14T12:31:58.0274572Z          at publisher.Publisher.Run(CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Publisher.cs:line 69
2022-11-14T12:31:58.0275361Z          at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Publisher.cs:line 47
2022-11-14T12:31:58.0276074Z          at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
2022-11-14T12:31:59.3119145Z Exit code is 255

The APIM also becomes unresponsive after the failed pipeline (Management API becomes unreachable)

After deleting all APIs and attempting publish the configuration again, the pipeline completes with no issue.

Expected behavior

No 409 error are triggered when the pipeline is running against an APIM that was configured by the pipeline in the past

Actual behavior

It is not possible to publish changes to some APIs due to error HTTP error 409

Reproduction Steps

None

A rough summary of the APIM instance can be found here #135 (comment)
@guythetechie
Copy link
Contributor

Thanks for raising this. Is this an intermittent issue, or are you able to reproduce it consistently? If so, can you please detail reproduction steps?

@Gabriel123N
Copy link
Contributor Author

Gabriel123N commented Nov 14, 2022 •
edited
Loading

It is a recurring issue. Here is what I did:

  1. Deploy an APIM with bicep (Developer tier, public)
  2. Run the publisher pipeline against that new APIM (using the files from our DEV APIM) => Changes are applied and pipeline succeed
  3. Run the publisher pipeline again after some time => Error 409 happens on every run
  4. Delete all APIs from the APIM
  5. Run the publisher => Changes are applied and the pipeline succeed
  6. Return to point 3 until point 4 is done

@guythetechie
Copy link
Contributor

Thanks. A couple of small checks:

  • Are the published files in step 2 extracted from dev using the v2 extractor?
  • Do you have a configuration.yaml in PROD that matches the v2 format?

If both are true, would you mind sharing your repo contents + the bicep file in a private repo for troubleshooting? I understand there might be some privacy concerns, so please obfuscate/delete anything you're uncomfortable sharing.

@Gabriel123N
Copy link
Contributor Author

Gabriel123N commented Nov 14, 2022 •
edited
Loading

Are the published files in step 2 extracted from dev using the v2 extractor?

Yes. I did a full cleanup of the repo files after upgrading to v2

Do you have a configuration.yaml in PROD that matches the v2 format?

Yes. The file has been modified and lint-ed to verify its validity.

I added you to this repo where you can find files that I hope can help you resolve the issue https://github.com/Gabriel123N/issue-apiops-141

@waelkdouh waelkdouh added the bug Something isn't working label Nov 14, 2022
@guythetechie
Copy link
Contributor

So, one thing jumps out at me. I see in the failure that it's doing a put request against https://management.azure.com/subscriptions/***/resourceGroups/rg/providers/Microsoft.ApiManagement/service/tapim/apis/api?api-version=2021-12-01-preview.

That URL indicates that it's doing a PUT request against an API called api. If it's trying to update multiple APIs with that name simultaneously, the error message makes sense. We can validate that by looking at the publisher logs and checking if you see several instances of Putting API api...

Would you mind uploading the full publisher logs in the private repo? Also, please upload your configuration.prod.yaml, as that is what will be used for publishing.

@Gabriel123N
Copy link
Contributor Author

I added the pipeline logs at the root of the repo. The configuration file is also present directly under the "apim" folder.

@guythetechie
Copy link
Contributor

I believe I've identified the issue. Working on a fix now.

@guythetechie guythetechie mentioned this issue Nov 15, 2022
Merged
@guythetechie
Copy link
Contributor

@Gabriel123N - pushed a fix in the main branch that relates to this issue. Can you please retest with the latest code in main?

@Gabriel123N
Copy link
Contributor Author

Gabriel123N commented Nov 16, 2022 •
edited
Loading

I ran the pipeline a few times in a row by reproducing the steps in 2nd comment and no errors were triggered.

Thank you a lot for your help once more!

@waelkdouh
Copy link
Contributor

Can we close this issue?

@Gabriel123N
Copy link
Contributor Author

Yes. I wanted to leave it open a bit more in case it would suddenly appears again. But so far so good.

@guythetechie
Copy link
Contributor

Glad it's working now. Please feel free to reopen the issue if it reappears or create a new one if there's anything else. Thanks for the extensive testing!

@hscheffert
Copy link

@guythetechie @waelkdouh
I am seeing this happen still with v4.1.3 with the publisher pipeline for an api versionset with 2 api versions ("aeg-new-business-api-partner-test" and "aeg-new-business-api" ).

I've done a similar setup with other API version sets w/ 2 versions without issue. For some reason this one refuses to publish..is there something I'm missing? Shouldn't it be able to do both API versions at once?

Here's the latest pipeline logs:

2023-05-08T14:48:32.6310812Z dbug: Microsoft.Extensions.Hosting.Internal.Host[1]
2023-05-08T14:48:32.6322371Z       Hosting starting
2023-05-08T14:48:32.7387885Z info: Publisher[0]
2023-05-08T14:48:32.7388064Z       Beginning execution...
2023-05-08T14:48:32.7388233Z info: Publisher[0]
2023-05-08T14:48:32.7388454Z       Commit ID was not specified, will put all artifact files...
2023-05-08T14:48:32.7754000Z info: Microsoft.Hosting.Lifetime[0]
2023-05-08T14:48:32.7754842Z       Application started. Press Ctrl+C to shut down.
2023-05-08T14:48:32.7860417Z info: Microsoft.Hosting.Lifetime[0]
2023-05-08T14:48:32.7861377Z       Hosting environment: Production
2023-05-08T14:48:32.7862686Z info: Microsoft.Hosting.Lifetime[0]
2023-05-08T14:48:32.7862885Z       Content root path: /home/vsts/work/1/s
2023-05-08T14:48:32.7863083Z dbug: Microsoft.Extensions.Hosting.Internal.Host[2]
2023-05-08T14:48:32.7863265Z       Hosting started
2023-05-08T14:48:32.8126699Z info: Publisher[0]
2023-05-08T14:48:32.8127941Z       Putting apiVersionSet vs-aeg-new-business-api...
2023-05-08T14:48:32.8195876Z dbug: PutRestResource[0]
2023-05-08T14:48:32.8206207Z       Beginning request to put REST resource URI https://management.azure.com/subscriptions/***/resourceGroups/rg-apiman-dev-001/providers/Microsoft.ApiManagement/service/APIMan-dev-001/apiVersionSets/vs-aeg-new-business-api?api-version=2022-04-01-preview...
2023-05-08T14:48:33.6547741Z dbug: PutRestResource[0]
2023-05-08T14:48:33.6560934Z       Successfully put REST resource at URI https://management.azure.com/subscriptions/***/resourceGroups/rg-apiman-dev-001/providers/Microsoft.ApiManagement/service/APIMan-dev-001/apiVersionSets/vs-aeg-new-business-api?api-version=2022-04-01-preview.
2023-05-08T14:48:33.6708342Z info: Publisher[0]
2023-05-08T14:48:33.6708968Z       Putting backend newbusinessapi...
2023-05-08T14:48:33.6711177Z info: Publisher[0]
2023-05-08T14:48:33.6711718Z       Putting backend func-signalR-newbusinessapi...
2023-05-08T14:48:33.6720579Z dbug: PutRestResource[0]
2023-05-08T14:48:33.6721715Z       Beginning request to put REST resource URI https://management.azure.com/subscriptions/***/resourceGroups/rg-apiman-dev-001/providers/Microsoft.ApiManagement/service/APIMan-dev-001/backends/newbusinessapi?api-version=2022-04-01-preview...
2023-05-08T14:48:33.6722384Z dbug: PutRestResource[0]
2023-05-08T14:48:33.6724392Z       Beginning request to put REST resource URI https://management.azure.com/subscriptions/***/resourceGroups/rg-apiman-dev-001/providers/Microsoft.ApiManagement/service/APIMan-dev-001/backends/func-signalR-newbusinessapi?api-version=2022-04-01-preview...
2023-05-08T14:48:36.6387112Z dbug: PutRestResource[0]
2023-05-08T14:48:36.6388277Z       Successfully put REST resource at URI https://management.azure.com/subscriptions/***/resourceGroups/rg-apiman-dev-001/providers/Microsoft.ApiManagement/service/APIMan-dev-001/backends/newbusinessapi?api-version=2022-04-01-preview.
2023-05-08T14:48:36.6398549Z info: Publisher[0]
2023-05-08T14:48:36.6399052Z       Putting backend newbusinessapi-partner-test...
2023-05-08T14:48:36.6401806Z dbug: PutRestResource[0]
2023-05-08T14:48:36.6402701Z       Beginning request to put REST resource URI https://management.azure.com/subscriptions/***/resourceGroups/rg-apiman-dev-001/providers/Microsoft.ApiManagement/service/APIMan-dev-001/backends/newbusinessapi-partner-test?api-version=2022-04-01-preview...
2023-05-08T14:48:36.6995860Z dbug: PutRestResource[0]
2023-05-08T14:48:36.6996637Z       Successfully put REST resource at URI https://management.azure.com/subscriptions/***/resourceGroups/rg-apiman-dev-001/providers/Microsoft.ApiManagement/service/APIMan-dev-001/backends/func-signalR-newbusinessapi?api-version=2022-04-01-preview.
2023-05-08T14:48:37.2031771Z dbug: PutRestResource[0]
2023-05-08T14:48:37.2033366Z       Successfully put REST resource at URI https://management.azure.com/subscriptions/***/resourceGroups/rg-apiman-dev-001/providers/Microsoft.ApiManagement/service/APIMan-dev-001/backends/newbusinessapi-partner-test?api-version=2022-04-01-preview.
2023-05-08T14:48:37.2427796Z info: Publisher[0]
2023-05-08T14:48:37.2428317Z       Putting policyFragment SetBackendServiceBasedOnApplicationSource...
2023-05-08T14:48:37.3089680Z dbug: PutRestResource[0]
2023-05-08T14:48:37.3091250Z       Beginning request to put REST resource URI https://management.azure.com/subscriptions/***/resourceGroups/rg-apiman-dev-001/providers/Microsoft.ApiManagement/service/APIMan-dev-001/policyFragments/SetBackendServiceBasedOnApplicationSource?api-version=2022-04-01-preview&format=rawxml...
2023-05-08T14:48:40.7422747Z dbug: PutRestResource[0]
2023-05-08T14:48:40.7424335Z       Successfully put REST resource at URI https://management.azure.com/subscriptions/***/resourceGroups/rg-apiman-dev-001/providers/Microsoft.ApiManagement/service/APIMan-dev-001/policyFragments/SetBackendServiceBasedOnApplicationSource?api-version=2022-04-01-preview&format=rawxml.
2023-05-08T14:48:42.0597164Z info: Publisher[0]
2023-05-08T14:48:42.0597967Z       Putting API aeg-new-business-api-partner-test...
2023-05-08T14:48:42.0598765Z info: Publisher[0]
2023-05-08T14:48:42.0599082Z       Putting API aeg-new-business-api...
2023-05-08T14:48:42.6488133Z dbug: PutRestResource[0]
2023-05-08T14:48:42.6496319Z       Beginning request to put REST resource URI https://management.azure.com/subscriptions/***/resourceGroups/rg-apiman-dev-001/providers/Microsoft.ApiManagement/service/APIMan-dev-001/apis/aeg-new-business-api?api-version=2022-04-01-preview...
2023-05-08T14:48:42.7003942Z dbug: PutRestResource[0]
2023-05-08T14:48:42.7005412Z       Beginning request to put REST resource URI https://management.azure.com/subscriptions/***/resourceGroups/rg-apiman-dev-001/providers/Microsoft.ApiManagement/service/APIMan-dev-001/apis/aeg-new-business-api-partner-test?api-version=2022-04-01-preview...
2023-05-08T14:49:45.2932265Z dbug: PutRestResource[0]
2023-05-08T14:49:45.2934145Z       Successfully put REST resource at URI https://management.azure.com/subscriptions/***/resourceGroups/rg-apiman-dev-001/providers/Microsoft.ApiManagement/service/APIMan-dev-001/apis/aeg-new-business-api-partner-test?api-version=2022-04-01-preview.
2023-05-08T14:49:45.3100556Z crit: Publisher[0]
2023-05-08T14:49:45.3102457Z       System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/rg-apiman-dev-001/providers/Microsoft.ApiManagement/service/APIMan-dev-001/apis/aeg-new-business-api?api-version=2022-04-01-preview&asyncId=64590bd20838f8198066e2cf&asyncCode=200 failed with status code 409. Content is '{"error":{"code":"Conflict","message":"Operation on the API is in progress","details":null}}'.
2023-05-08T14:49:45.3133967Z          at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri)
2023-05-08T14:49:45.3134918Z          at common.HttpPipelineExtensions.WaitForLongRunningOperation(HttpPipeline pipeline, Response response, CancellationToken cancellationToken)
2023-05-08T14:49:45.3135992Z          at common.HttpPipelineExtensions.PutResource(HttpPipeline pipeline, Uri uri, JsonObject resource, CancellationToken cancellationToken)
2023-05-08T14:49:45.3136330Z          at publisher.Program.<>c__DisplayClass12_0.<<GetPutRestResource>b__0>d.MoveNext()
2023-05-08T14:49:45.3136740Z       --- End of stack trace from previous location ---
2023-05-08T14:49:45.3137124Z          at publisher.Api.PutApi(ApiName apiName, ApiInformationFile apiInformationFile, ApiSpecificationFile specificationFile, JsonObject configurationApiJson, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
2023-05-08T14:49:45.3137674Z          at publisher.Api.<>c__DisplayClass23_0.<<ProcessArtifactsToPut>b__4>d.MoveNext()
2023-05-08T14:49:45.3137974Z       --- End of stack trace from previous location ---
2023-05-08T14:49:45.3138220Z          at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()
2023-05-08T14:49:45.3138525Z       --- End of stack trace from previous location ---
2023-05-08T14:49:45.3138810Z          at common.IEnumerableExtensions.ForEachParallel[T](IEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken)
2023-05-08T14:49:45.3139230Z          at publisher.Api.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
2023-05-08T14:49:45.3139809Z          at publisher.Service.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken)
2023-05-08T14:49:45.3140258Z          at publisher.Publisher.RunWithoutCommitId(CancellationToken cancellationToken)
2023-05-08T14:49:45.3140509Z          at publisher.Publisher.Run(CancellationToken cancellationToken)
2023-05-08T14:49:45.3141155Z          at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken)
2023-05-08T14:49:45.3141357Z info: Microsoft.Hosting.Lifetime[0]
2023-05-08T14:49:45.3141544Z       Application is shutting down...
2023-05-08T14:49:45.3141742Z fail: Microsoft.Extensions.Hosting.Internal.Host[9]
2023-05-08T14:49:45.3142058Z       BackgroundService failed
2023-05-08T14:49:45.3143166Z       System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/rg-apiman-dev-001/providers/Microsoft.ApiManagement/service/APIMan-dev-001/apis/aeg-new-business-api?api-version=2022-04-01-preview&asyncId=64590bd20838f8198066e2cf&asyncCode=200 failed with status code 409. Content is '{"error":{"code":"Conflict","message":"Operation on the API is in progress","details":null}}'.
2023-05-08T14:49:45.3143718Z          at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri)
2023-05-08T14:49:45.3144031Z          at common.HttpPipelineExtensions.WaitForLongRunningOperation(HttpPipeline pipeline, Response response, CancellationToken cancellationToken)
2023-05-08T14:49:45.3144383Z          at common.HttpPipelineExtensions.PutResource(HttpPipeline pipeline, Uri uri, JsonObject resource, CancellationToken cancellationToken)
2023-05-08T14:49:45.3144691Z          at publisher.Program.<>c__DisplayClass12_0.<<GetPutRestResource>b__0>d.MoveNext()
2023-05-08T14:49:45.3144999Z       --- End of stack trace from previous location ---
2023-05-08T14:49:45.3145384Z          at publisher.Api.PutApi(ApiName apiName, ApiInformationFile apiInformationFile, ApiSpecificationFile specificationFile, JsonObject configurationApiJson, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
2023-05-08T14:49:45.3145784Z          at publisher.Api.<>c__DisplayClass23_0.<<ProcessArtifactsToPut>b__4>d.MoveNext()
2023-05-08T14:49:45.3146072Z       --- End of stack trace from previous location ---
2023-05-08T14:49:45.3146313Z          at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()
2023-05-08T14:49:45.3146613Z       --- End of stack trace from previous location ---
2023-05-08T14:49:45.3146891Z          at common.IEnumerableExtensions.ForEachParallel[T](IEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken)
2023-05-08T14:49:45.3147325Z          at publisher.Api.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
2023-05-08T14:49:45.3147891Z          at publisher.Service.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken)
2023-05-08T14:49:45.3148341Z          at publisher.Publisher.RunWithoutCommitId(CancellationToken cancellationToken)
2023-05-08T14:49:45.3148594Z          at publisher.Publisher.Run(CancellationToken cancellationToken)
2023-05-08T14:49:45.3148842Z          at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken)
2023-05-08T14:49:45.3149140Z          at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
2023-05-08T14:49:45.3161611Z crit: Microsoft.Extensions.Hosting.Internal.Host[10]
2023-05-08T14:49:45.3162014Z       The HostOptions.BackgroundServiceExceptionBehavior is configured to StopHost. A BackgroundService has thrown an unhandled exception, and the IHost instance is stopping. To avoid this behavior, configure this to Ignore; however the BackgroundService will not be restarted.
2023-05-08T14:49:45.3163369Z       System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/rg-apiman-dev-001/providers/Microsoft.ApiManagement/service/APIMan-dev-001/apis/aeg-new-business-api?api-version=2022-04-01-preview&asyncId=64590bd20838f8198066e2cf&asyncCode=200 failed with status code 409. Content is '{"error":{"code":"Conflict","message":"Operation on the API is in progress","details":null}}'.
2023-05-08T14:49:45.3164260Z          at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri)
2023-05-08T14:49:45.3164576Z          at common.HttpPipelineExtensions.WaitForLongRunningOperation(HttpPipeline pipeline, Response response, CancellationToken cancellationToken)
2023-05-08T14:49:45.3164935Z          at common.HttpPipelineExtensions.PutResource(HttpPipeline pipeline, Uri uri, JsonObject resource, CancellationToken cancellationToken)
2023-05-08T14:49:45.3165243Z          at publisher.Program.<>c__DisplayClass12_0.<<GetPutRestResource>b__0>d.MoveNext()
2023-05-08T14:49:45.3165574Z       --- End of stack trace from previous location ---
2023-05-08T14:49:45.3165960Z          at publisher.Api.PutApi(ApiName apiName, ApiInformationFile apiInformationFile, ApiSpecificationFile specificationFile, JsonObject configurationApiJson, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
2023-05-08T14:49:45.3166357Z          at publisher.Api.<>c__DisplayClass23_0.<<ProcessArtifactsToPut>b__4>d.MoveNext()
2023-05-08T14:49:45.3166674Z       --- End of stack trace from previous location ---
2023-05-08T14:49:45.3166915Z          at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()
2023-05-08T14:49:45.3167218Z       --- End of stack trace from previous location ---
2023-05-08T14:49:45.3167483Z          at common.IEnumerableExtensions.ForEachParallel[T](IEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken)
2023-05-08T14:49:45.3167914Z          at publisher.Api.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
2023-05-08T14:49:45.3168499Z          at publisher.Service.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken)
2023-05-08T14:49:45.3168948Z          at publisher.Publisher.RunWithoutCommitId(CancellationToken cancellationToken)
2023-05-08T14:49:45.3169199Z          at publisher.Publisher.Run(CancellationToken cancellationToken)
2023-05-08T14:49:45.3169445Z          at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken)
2023-05-08T14:49:45.3169725Z          at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
2023-05-08T14:49:45.3170217Z dbug: Microsoft.Extensions.Hosting.Internal.Host[3]
2023-05-08T14:49:45.3170414Z       Hosting stopping
2023-05-08T14:49:45.3180129Z dbug: Microsoft.Extensions.Hosting.Internal.Host[4]
2023-05-08T14:49:45.3180316Z       Hosting stopped
2023-05-08T14:49:45.5376723Z �[31;1mException: �[0m/home/vsts/work/_temp/b1b717bb-e3d8-4683-9b0d-a676814835e9.ps1:9
2023-05-08T14:49:45.5377430Z �[36;1mLine |
2023-05-08T14:49:45.5378251Z �[36;1m   9 | �[0m if ($LASTEXITCODE -ne 0) { �[36;1mthrow "Running publisher failed."�[0m}
2023-05-08T14:49:45.5378708Z �[36;1m     | �[31;1m                            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-05-08T14:49:45.5379000Z �[31;1m�[36;1m     | �[31;1mRunning publisher failed.
2023-05-08T14:49:45.5379184Z �[0m
2023-05-08T14:49:45.5948912Z ##[debug]Exit code 1 received from tool '/usr/bin/pwsh'
2023-05-08T14:49:45.5955166Z ##[debug]STDIO streams have closed for tool '/usr/bin/pwsh'
2023-05-08T14:49:45.5967050Z ##[debug]task result: Failed
2023-05-08T14:49:45.6008204Z ##[error]PowerShell exited with code '1'.
2023-05-08T14:49:45.6032379Z ##[debug]Processed: ##vso[task.issue type=error;]PowerShell exited with code '1'.
2023-05-08T14:49:45.6033753Z ##[debug]Processed: ##vso[task.complete result=Failed;]PowerShell exited with code '1'.
2023-05-08T14:49:45.6034158Z ##[debug]task result: Failed
2023-05-08T14:49:45.6034729Z ##[error]PowerShell wrote one or more lines to the standard error stream.
2023-05-08T14:49:45.6035269Z ##[debug]Processed: ##vso[task.issue type=error;]PowerShell wrote one or more lines to the standard error stream.
2023-05-08T14:49:45.6035968Z ##[debug]Processed: ##vso[task.complete result=Failed;]PowerShell wrote one or more lines to the standard error stream.
2023-05-08T14:49:45.6036877Z ##[error]�[31;1mException: �[0m/home/vsts/work/_temp/b1b717bb-e3d8-4683-9b0d-a676814835e9.ps1:9
�[36;1mLine |
�[36;1m   9 | �[0m if ($LASTEXITCODE -ne 0) { �[36;1mthrow "Running publisher failed."�[0m}
�[36;1m     | �[31;1m                            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
�[31;1m�[36;1m     | �[31;1mRunning publisher failed.
�[0m

2023-05-08T14:49:45.6037754Z ##[debug]Processed: ##vso[task.issue type=error;]�[31;1mException: �[0m/home/vsts/work/_temp/b1b717bb-e3d8-4683-9b0d-a676814835e9.ps1:9
�[36;1mLine |
�[36;1m   9 | �[0m if ($LASTEXITCODE -ne 0) { �[36;1mthrow "Running publisher failed."�[0m}
�[36;1m     | �[31;1m                            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
�[31;1m�[36;1m     | �[31;1mRunning publisher failed.
�[0m

@erdelacruz
Copy link

Hi @guythetechie @waelkdouh ,

I'm facing this issue with v4.10.0 Publisher. Can we reopen this ticket ? This is happening whenever I run the publisher using all-artifacts, it always throw this error in PUT API operation.

info: Publisher[0] Putting API x-replaced-apiv1... crit: Publisher[0] System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/replacedResourceGroupName/providers/Microsoft.ApiManagement/service/replacedApimName/apis/x-replaced-apiv1?api-version=2022-04-01-preview failed with status code 409. Content is '{"error":{"code":"PessimisticConcurrencyConflict","message":"Operation on the API is in progress","details":null}}'. at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri) at common.HttpPipelineExtensions.PutResource(HttpPipeline pipeline, Uri uri, JsonObject resource, CancellationToken cancellationToken) at publisher.Program.<>c__DisplayClass12_0.<<GetPutRestResource>b__0>d.MoveNext() --- End of stack trace from previous location --- at publisher.Api.PutApi(ApiName apiName, ApiInformationFile apiInformationFile, ApiSpecificationFile specificationFile, JsonObject configurationApiJson, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken) at publisher.Api.<>c__DisplayClass23_0.<<ProcessArtifactsToPut>b__4>d.MoveNext() --- End of stack trace from previous location --- at System.Threading.Tasks.Parallel.<>c__501.<b__50_0>d.MoveNext()
--- End of stack trace from previous location ---
at common.IEnumerableExtensions.ForEachParallel[T](IEnumerable1 enumerable, Func2 action, CancellationToken cancellationToken)
at publisher.Api.ProcessArtifactsToPut(IReadOnlyCollection1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken) at publisher.Service.ProcessArtifactsToPut(IReadOnlyCollection1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken)
at publisher.Publisher.RunWithoutCommitId(CancellationToken cancellationToken)
at publisher.Publisher.Run(CancellationToken cancellationToken)
at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken)
`

@waelkdouh waelkdouh reopened this Sep 7, 2023
@waelkdouh waelkdouh mentioned this issue Sep 7, 2023
Closed
@waelkdouh
Copy link
Contributor

closing due to inactivity.

@mgrabarz mgrabarz mentioned this issue Feb 29, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@waelkdouh @erdelacruz @guythetechie @Gabriel123N @hscheffert