Extractor Pipeline for ADO has windows-latest for PR Job

[corganfuzz]

Release version

latest

Question Details

is there any reason why the extractor pipeline for ADO that creates a pull request runs in windows-latest ?. I've switched mine to ubuntu-latest and it worked fine. I was just wondering so I wont have to create a windows VM on my end.

The github extractor pipeline doesnt seem to have this. Both jobs are running on ubuntu-latest

ADO-Extractor:

 - stage: create_template_branch
    displayName: Create template branch
    jobs:
      - job: create_artifacts_pull_request
        displayName: Create artifacts pull request
        pool:
          vmImage: windows-latest
        steps:
          - task: DownloadPipelineArtifact@2

Github-Extractor:

  create-pull-request:
    needs: extract
    runs-on: [ubuntu-latest]
    steps:
      - uses: actions/checkout@v3
          
      - name: Download artifacts-from-portal

Expected behavior

both run fine in ubuntu

Actual behavior

one runs in windows and the other one in ubuntu

Reproduction Steps

run it in ADO

here's the line in question:

ADO EXTRACTOR YAML line

[github-actions]

  Thank you for opening this issue! Please be patient while we will look into it and get back to you as this is an open source project. In the meantime make sure you take a look at the [closed issues](https://github.com/Azure/apiops/issues?q=is%3Aissue+is%3Aclosed) in case your question has already been answered. Don't forget to provide any additional information if needed (e.g. scrubbed logs, detailed feature requests,etc.).
  Whenever it's feasible, please don't hesitate to send a Pull Request (PR) our way. We'd greatly appreciate it, and we'll gladly assess and incorporate your changes.

[guythetechie]

@corganfuzz - You can change it if you like, either OS should work. That step executes different commands depending the OS. I think we changed it to windows-latest to test that the pipeline worked on Windows agents, but never changed it back to ubuntu-latest.

Do you have a problem with it running on Windows agents, or were you mentioning it just because that job OS is inconsistent with all the other pipeline OSes?

[corganfuzz]

Not a problem but more of an annoyance in our side. For our current implementation, we were provisioning linux vmss with custom software but we havent done for windows vmss which was a whole big process to go thru. thanks to this switch we can just our old linux vmss we already configured.

from the tech point of view , I thought windows vmss were doing something special. Glad thats not the case.

thanks for addresing this.

[corganfuzz]

Hey @waelkdouh so we stress test this extractor job in linux and found a bug with some inconsistent results:

The way I recreate the issue is like this:

I had my pipeline defaulted as "configuration.extractor.yaml" not "Extract All"

1. First Run: configuration.extractor.yaml options extracts what is supposed to (from yaml file) PR creates what is supposed to create
2. Second Run: Extract All option extracts everything to PR and it creates what is supposed to create
3. Third Run and above: Now it Runs as if you will always got Extract all selected no matter what you select BUG

After hours of troubleshooting, I've switched back to windows-latest and everything worked as expected.

I dig more into this problem and I found this. I dont know if it's related or not:

DownloadPipelineArtifacts issue
DownloadPipelineArtifacts other issue

what gave it away for me was this error I found on that step:

2021-02-25T15:56:56.8540082Z Verbose, Could not initialize dataport: Dataport will not be used as it is only currently available on Windows.

idk if those are the reasons but it definitely pointed me out to the bug.

Once I switched back to windows everything worked as intended

let me know if you can replicate:

this is what im using:
apiops v5.0.0
multiple subscriptions per environment

sorry if I've caused this, I felt it was my duty to let you know

[waelkdouh]

Thanks for brining this to our attention. We don't have the bandwidth to test it for now but we will keep a close eye on this in case others raise the same issue as it sounds like ado being idiosyncratic. Let us know if you find anything else.

[corganfuzz]

@waelkdouh

I think we might've founded the root cause:

in the latest version of run-extractor.yaml

Write-Information "Synchronizing artifacts..."

$extractorArtifactsFolderPath = Join-Path "$(Pipeline.Workspace)" "artifacts-from-portal" ${{ parameters.API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH }}
                
if ("$(Agent.OS)" -like "*win*") {
     & robocopy "$extractorArtifactsFolderPath" "$artifactFolderPath" /zb /mir /mt
     if ($LASTEXITCODE -gt 7) { throw "Setting $artifactFolderPath to contents of $extractorArtifactsFolderPath failed." }         
 }
else {
     & rsync --verbose --archive --delete --force --recursive "$extractorArtifactsFolderPath/" "$artifactFolderPath/"
     if ($LASTEXITCODE -ne 0) { throw "Setting $artifactFolderPath to contents of $extractorArtifactsFolderPath failed." }
 }

which is the only part of the Creating PR job where the instruction differ from linux to windows.

looks to me like that rsync command might be outdated.

thoughts ?

[waelkdouh]

We will take a look and get back to you.

[corganfuzz]

@waelkdouh I've zeroed in on the problem and found out it only happens in our side. Turns out we are using self hosted vmss that are running ubuntu version 20.04.6 LTS. the rsync version we are using is 3.1.3 which is outdated.

The issue does not appear when I switch to the ubuntu-latest pool so It's just an issue only for ourselves.

Thanks