Azure DevOps cannot clone repo in non-default structure

[gaom25]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

The structure of the repositories in AzureDevops in our organization is:
{owner}.visualstudio.com/{project}_git/{repo}

When setting the variables for Atlantis of owner and project with the format

{owner}.visualstudio.com/{project}/*
or
dev.azure.com/{owner}/{project}/*

Atlantis tries to clone the repo with the structure
{owner}.visualstudio.com/{owner}/{project}/*
or
dev.azure.com/{owner}/{owner}/{project}/*

Causing a 404 Error

Reproduction Steps

Use the latest version of Atlantis(0.21.0) and set the variables to connect to AzureDevops.

You can use the variables of ATLANTIS_REPO_ALLOWLIST in wildcard(*) and it will trigger the error

Logs

{"level":"error","ts":"2022-12-05T16:04:29.127Z","caller":"events/command_runner.go:427","msg":"PANIC: runtime error: invalid memory address or nil pointer dereference
runtime/panic.go:260 (0x44ebd5)
runtime/signal_unix.go:835 (0x44eba5)
github.com/runatlantis/atlantis/server/events/vcs/azuredevops_client.go:71 (0xbc4b9a)
github.com/runatlantis/atlantis/server/events/vcs/proxy.go:56 (0xbd18d4)
github.com/runatlantis/atlantis/server/events/project_command_builder.go:215 (0xdf2618)
github.com/runatlantis/atlantis/server/events/project_command_builder.go:166 (0xdf1f44)
github.com/runatlantis/atlantis/server/events/instrumented_project_command_builder.go:44 (0xde7eeb)
github.com/runatlantis/atlantis/server/events/plan_command_runner.go:77 (0xdec2c3)
github.com/runatlantis/atlantis/server/events/plan_command_runner.go:245 (0xdee704)
github.com/runatlantis/atlantis/server/events/command_runner.go:174 (0xdd7cea)
runtime/asm_amd64.s:1594 (0x46a100)
","json":{"repo":"{owner}/{project}/{repo}","pull":"34002"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).logPanics
	github.com/runatlantis/atlantis/server/events/command_runner.go:427
runtime.gopanic
	runtime/panic.go:884
runtime.panicmem
	runtime/panic.go:260
runtime.sigpanic
	runtime/signal_unix.go:835
github.com/runatlantis/atlantis/server/events/vcs.(*AzureDevopsClient).GetModifiedFiles
	github.com/runatlantis/atlantis/server/events/vcs/azuredevops_client.go:71
github.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).GetModifiedFiles
	github.com/runatlantis/atlantis/server/events/vcs/proxy.go:56
github.com/runatlantis/atlantis/server/events.(*DefaultProjectCommandBuilder).buildPlanAllCommands
	github.com/runatlantis/atlantis/server/events/project_command_builder.go:215
github.com/runatlantis/atlantis/server/events.(*DefaultProjectCommandBuilder).BuildAutoplanCommands
	github.com/runatlantis/atlantis/server/events/project_command_builder.go:166
github.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandBuilder).BuildAutoplanCommands
	github.com/runatlantis/atlantis/server/events/instrumented_project_command_builder.go:44
github.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).runAutoplan
	github.com/runatlantis/atlantis/server/events/plan_command_runner.go:77
github.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).Run
	github.com/runatlantis/atlantis/server/events/plan_command_runner.go:245
github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunAutoplanCommand
	github.com/runatlantis/atlantis/server/events/command_runner.go:174"}
{"level":"error","ts":"2022-12-05T16:04:29.246Z","caller":"events/command_runner.go:434","msg":"unable to comment: Request to https://dev.azure.com/{owner}/{owner}/{project}/_apis/git/repositories/{repo}/pullrequests/34002/threads?api-version=5.1-preview.1 responded with status 404","json":{"repo":"{owner}/{project}/{repo}","pull":"34002"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).logPanics
	github.com/runatlantis/atlantis/server/events/command_runner.go:434
runtime.gopanic
	runtime/panic.go:884
runtime.panicmem
	runtime/panic.go:260
runtime.sigpanic
	runtime/signal_unix.go:835
github.com/runatlantis/atlantis/server/events/vcs.(*AzureDevopsClient).GetModifiedFiles
	github.com/runatlantis/atlantis/server/events/vcs/azuredevops_client.go:71
github.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).GetModifiedFiles
	github.com/runatlantis/atlantis/server/events/vcs/proxy.go:56
github.com/runatlantis/atlantis/server/events.(*DefaultProjectCommandBuilder).buildPlanAllCommands
	github.com/runatlantis/atlantis/server/events/project_command_builder.go:215
github.com/runatlantis/atlantis/server/events.(*DefaultProjectCommandBuilder).BuildAutoplanCommands
	github.com/runatlantis/atlantis/server/events/project_command_builder.go:166
github.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandBuilder).BuildAutoplanCommands
	github.com/runatlantis/atlantis/server/events/instrumented_project_command_builder.go:44
github.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).runAutoplan
	github.com/runatlantis/atlantis/server/events/plan_command_runner.go:77
github.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).Run
	github.com/runatlantis/atlantis/server/events/plan_command_runner.go:245
github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunAutoplanCommand
	github.com/runatlantis/atlantis/server/events/command_runner.go:174"}

Environment details

• Atlantis version: 0.21.0
• ATLANTIS_REPO_ALLOWLIST=*

Additional Context

This is the data that AzureDevops send

{
    "subscriptionId": "...",
    "notificationId": 13,
    "id": "...",
    "eventType": "git.pullrequest.updated",
    "publisherId": "tfs",
    "message": {
        "text": "..",
        "html": "...",
        "markdown": "..."
    },
    "detailedMessage": {
        "text": "...",
        "html": "...",
        "markdown": "..."
    },
    "resource": {
        "repository": {
            "id": "xxx",
            "name": "xxx",
            "url": "https://{owner}.visualstudio.com/xxx/_apis/git/repositories/yyy",
            "project": {
                "id": "xxx",
                "name": "{project}",
                "description": "xxx",
                "url": "https://{owner}.visualstudio.com/_apis/projects/xxx",
                "state": "wellFormed",
                "revision": 1535,
                "visibility": "private",
                "lastUpdateTime": "2022-06-25T00:50:28.69Z"
            },
            "size": 26100,
            "remoteUrl": "https://{owner}.visualstudio.com/{project}/_git/{repo}",
            "sshUrl": "{owner}@vs-ssh.visualstudio.com:v3/{owner}/{project}/{repo}",
            "webUrl": "https://{owner}.visualstudio.com/{project}/_git/{repo}",
            "isDisabled": false,
            "isInMaintenance": false
        },
        "pullRequestId": 34002,
        "codeReviewId": 34002,
        "status": "active",
        "createdBy": {},
        "creationDate": "date",
        "title": "Change script to start instances",
        "sourceRefName": "refs/heads/feature/branch",
        "targetRefName": "refs/heads/main",
        "mergeStatus": "succeeded",
        "isDraft": false,
        "mergeId": "xxx",
        "lastMergeSourceCommit": {
            "commitId": "xxx",
            "url": "https://{owner}.visualstudio.com/xxx/_apis/git/repositories/yyy/commits/zzz"
        },
        "lastMergeTargetCommit": {
            "commitId": "5cdf71f9e7e78390d20d5db1ac2b43a549f48a5c",
            "url": "https://{owner}.visualstudio.com/xxx/_apis/git/repositories/yyy/commits/zzz"
        },
        "lastMergeCommit": {},
        "reviewers": [ ],
        "url": "https://{owner}.visualstudio.com/xxx/_apis/git/repositories/yyy/pullRequests/34002",
        "_links": {
            "web": {
                "href": "https://{owner}.visualstudio.com/{project}/_git/{repo}/pullrequest/34002"
            },
            "statuses": {
                "href": "https://{owner}.visualstudio.com/xxx/_apis/git/repositories/yyy/pullRequests/34002/statuses"
            }
        },
        "supportsIterations": true,
        "artifactId": "vstfs:///Git/PullRequestId/xxx%2fyyy%2f34002"
    },
    "resourceVersion": "1.0",
    "resourceContainers": {
        "collection": {
            "id": "xxx",
            "baseUrl": "https://{owner}.visualstudio.com/"
        },
        "account": {
            "id": "xxx",
            "baseUrl": "https://{owner}.visualstudio.com/"
        },
        "project": {
            "id": "xxx",
            "baseUrl": "https://{owner}.visualstudio.com/"
        }
    },
    "createdDate": "date"
}

[nitrocode]

Hi @gaom25 . Thank you for the issue. I re-titled it since Azure DevOps is working for other folks. It seems that you want to use something outside of the non-default structure.

Please (either you or someone else who wants to take this on) take a look at the Azure DevOps golang client code and see if you can find a way to allow customizing the clone structure.

https://github.com/runatlantis/atlantis/blob/main/server/events/vcs/azuredevops_client.go

Also this one maybe more relevant

atlantis/server/events/event_parser.go

Lines 910 to 946 in 1fbbcca

	func (e *EventParser) ParseAzureDevopsRepo(adRepo *azuredevops.GitRepository) (models.Repo, error) {
	teamProject := adRepo.GetProject()
	parent := adRepo.GetParentRepository()
	owner := ""
	uri, err := url.Parse(adRepo.GetWebURL())
	if err != nil {
	return models.Repo{}, err
	}
	if parent != nil {
	owner = parent.GetName()
	} else {
	if strings.Contains(uri.Host, "visualstudio.com") {
	owner = strings.Split(uri.Host, ".")[0]
	} else {
	owner = strings.Split(uri.Path, "/")[1]
	}
	}
	// Construct our own clone URL so we always get the new dev.azure.com
	// hostname for now.
	// https://docs.microsoft.com/en-us/azure/devops/release-notes/2018/sep-10-azure-devops-launch#switch-existing-organizations-to-use-the-new-domain-name-url
	project := teamProject.GetName()
	repo := adRepo.GetName()
	host := uri.Host
	if host == "" {
	host = "dev.azure.com"
	}
	cloneURL := fmt.Sprintf("https://%s/%s/%s/_git/%s", host, owner, project, repo)
	fmt.Println("%", cloneURL)
	fullName := fmt.Sprintf("%s/%s/%s", owner, project, repo)
	return models.NewRepo(models.AzureDevops, fullName, cloneURL, e.AzureDevopsUser, e.AzureDevopsToken)
	}

Perhaps passing in a new flag like --azuredevops-clone-template "https://%s/%s/%s/_git/%s" would be good. That way you can modify the default from

https://%s/%s/%s/_git/%s

to

https://%s/%s/%s_git/%s

But it could be more complicated because I do not know how owner/project/repo map.

[jsw1993]

Getting the same issue.

I don't know how you can call this a non-default structure. AFAIK you cannot customize this structure when using Azure DevOps Services. I'm not setting any of these vars and am getting a similar issue in that its building the URL its trying to use to clone incorrectly

EDIT:
Looked into this further. It's due to a difference in Azure DevOps legacy URLs {organization}.visualstudio.com and the new dev.azure.com/{organization} URLs. Have done a PR which fixes