Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use sparse checkout for generate matrix job #1452

Merged
2 commits merged into from
Mar 9, 2021
Merged

Use sparse checkout for generate matrix job #1452

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6f53ea8
Use sparse checkout for generate matrix job
benbp Mar 1, 2021
8a6da6c
Add repository parameter
benbp Mar 6, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 26 additions & 2 deletions eng/common/pipelines/templates/jobs/archetype-sdk-tests-generate.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,40 @@ parameters:
default: []
- name: JobTemplatePath
type: string
# Set this to false to do a full checkout for private repositories with the azure pipelines service connection
- name: SparseCheckout
type: boolean
default: true
- name: SparseCheckoutPaths
type: object
default: []
- name: Pool
type: string
default: azsdk-pool-mms-ubuntu-1804-general
- name: OsVmImage
type: string
default: MMSUbuntu18.04

jobs:
- job: generate_matrix
variables:
displayNameFilter: $[ coalesce(variables.jobMatrixFilter, '.*') ]
pool:
name: Azure Pipelines
vmImage: ubuntu-18.04
name: ${{ parameters.Pool }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason to parameterize this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason I have it this way is I want the /eng/common/scripts/job-matrix/samples/matrix-test.yml to be a working sample (partly because non-working samples are bad, and because I use it for testing changes like these). Since I test this sample pipeline in the playground project, it doesn't have access to the 1es pool, hence why I need to override the pool/image.

vmImage: ${{ parameters.OsVmImage }}
displayName: Generate Job Matrix
steps:
# Skip sparse checkout for the `azure-sdk-for-<lang>-pr` private mirrored repositories
# as we require the github service connection to be loaded.
- ${{ if and(parameters.SparseCheckout, not(contains(variables['Build.DefinitionName'], '-pr - '))) }}:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if it is better to check against the pipeline or repo name? I have a slight preference for the repo name so we aren't as dependent on someone failing to name their pipeline correctly.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Repo name isn't available at template build time :(

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#1452 (comment)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

booo... I guess we will have to see how this works out. I don't like how loosely coupled this is.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect this is going to come back to bite us once we start using it in more places in the private repos, but for now I guess we can start here and then reconsider other options if we hit an issue.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I'm not a fan of this approach but can't find an alternative. My thinking is also that the pr repos are right now our only standardized name private repos, and people that need to do temporary feature/pre-release testing can just keep SparseCheckout: false in their branch until they need to go live.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if they didn't have to have a code commit that would need to be undone once the merge to public. We could consider a pipeline variable that someone could set to opt-in/out of this but lets see if any when this becomes an issue.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The variable would make the template harder to define since we'd have to use a condition. I was using a SparseCheckout top-level parameter myself for testing, but if I had to check-enable it every time for private repo testing I think I would end up making a commit anyway as part of my workflow.

- template: /eng/common/pipelines/templates/steps/sparse-checkout.yml
parameters:
${{ if ne(length(parameters.SparseCheckoutPaths), 0) }}:
Paths: ${{ parameters.SparseCheckoutPaths }}
${{ if and(eq(length(parameters.SparseCheckoutPaths), 0), ne(parameters.AdditionalParameters.ServiceDirectory, '')) }}:
Paths:
- ${{ parameters.AdditionalParameters.ServiceDirectory }}

- ${{ each config in parameters.MatrixConfigs }}:
- ${{ if eq(config.GenerateVMJobs, 'true') }}:
- task: Powershell@2
Expand Down
20 changes: 20 additions & 0 deletions eng/common/pipelines/templates/steps/sparse-checkout.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
parameters:
- name: Paths
type: object
default: []

steps:
- checkout: none

- pwsh: |
git clone --no-checkout --filter=tree:0 git://github.com/$(Build.Repository.Name) .
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is probably worth adding the repository as a template parameter as I believe this template will be interesting for pipelines we clone multiple pipelines.

I also suggest adding a template parameter for the working directory for these steps so folks can control where this clone happens.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a repositories object, since with multiple repos we need to pair the working directory with the repo name and commit-ish. Also, I set the default the way it is so that when no repo is specified, we just checkout into the default location, this way any calling template doesn't need to manually specify WorkingDirectory when there's only one repo.

git sparse-checkout init
git sparse-checkout set eng
displayName: Initialize sparse checkout repository

- ${{ each path in parameters.Paths }}:
- pwsh: git sparse-checkout add ${{ path }}
displayName: Add sparse checkout path ${{ path }}

- pwsh: git checkout $(Build.SourceVersion)
displayName: Sparse checkout repository paths
5 changes: 4 additions & 1 deletion eng/common/scripts/job-matrix/samples/matrix-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,10 @@ jobs:
- template: /eng/common/pipelines/templates/jobs/archetype-sdk-tests-generate.yml
parameters:
JobTemplatePath: /eng/common/scripts/job-matrix/samples/matrix-job-sample.yml
AdditionalParameters: []
AdditionalParameters: {}
Pool: Azure Pipelines
OsVmImage: ubuntu-18.04
SparseCheckout: ${{ parameters.SparseCheckout }}
CloudConfig:
SubscriptionConfiguration: $(sub-config-azure-cloud-test-resources)
Location: eastus2
Expand Down