argocd cannot resolve kustomize manifest.

[jetersen]

Describe the bug

We use argo-cd to deploy argo-cd using kustomize, though since v1.5.6 we are seeing the error below. So downgraded to v1.5.5 where it works.
Tried to upgrade to v1.5.7 was working fine for a day or so but than the argo-cd application hit the error again.

rpc error: code = Unknown desc = `kustomize build /tmp/git@repo/argocd/overlay` failed exit status 1: Error: accumulating resources: accumulateFile "accumulating resources from 'github.com/argoproj/argo-cd//manifests/cluster-install?ref=v1.5.7': evalsymlink failure on '/tmp/git@repo/argocd/overlay/github.com/argoproj/argo-cd/manifests/cluster-install?ref=v1.5.7' : lstat /tmp/git@repo/argocd/overlay/github.com: no such file or directory", accumulateDirector: "recursed accumulation of path '/tmp/kustomize-881686007/repo': accumulating resources: accumulateFile \"accumulating resources from '../namespace-install': evalsymlink failure on '/tmp/kustomize-881686007/namespace-install' : lstat /tmp/kustomize-881686007/namespace-install: no such file or directory\", loader.New \"Error loading ../namespace-install with git: url lacks host: ../namespace-install, dir: evalsymlink failure on '/tmp/kustomize-881686007/namespace-install' : lstat /tmp/kustomize-881686007/namespace-install: no such file or directory, get: invalid source string: ../namespace-install\""

To Reproduce

Setup a argo-cd application using kustomize

bases:
- github.com/argoproj/argo-cd//manifests/cluster-install?ref=v1.5.7
namespace: argocd

Expected behavior

Kustomize should be able to resolve the manifest.

Screenshots

Version

argocd: v1.6.0+c10ae24
  BuildDate: 2020-06-16T22:39:40Z
  GitCommit: c10ae246ab02f1356147118a1979fedcd1ceb704
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
argocd-server: v1.5.7+e7d1553
  BuildDate: 2020-06-09T18:14:59Z
  GitCommit: e7d1553cfc10e059acd39914a35ed3450eab7574
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: {Version:kustomize/v3.6.1 GitCommit:c97fa946d576eb6ed559f17f2ac43b3b5a8d5dbd BuildDate:2020-05-27T20:47:35Z GoOs:linux GoArch:amd64}
  Helm Version: version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}
  Kubectl Version: v1.14.0

Logs

Repo server log:

time="2020-06-19T23:40:21Z" level=error msg="`kustomize build /tmp/git@repo/argocd/overlay` failed exit status 1: Error: accumulating resources: accumulateFile \"accumulating resources from 'github.com/argoproj/argo-cd//manifests/cluster-install?ref=v1.5.7': evalsymlink failure on '/tmp/git@repo/argocd/overlay/github.com/argoproj/argo-cd/manifests/cluster-install?ref=v1.5.7' : lstat /tmp/git@repo/argocd/overlay/github.com: no such file or directory\", accumulateDirector: \"recursed accumulation of path '/tmp/kustomize-827653883/repo': accumulating resources: accumulateFile \\\"accumulating resources from '../namespace-install': evalsymlink failure on '/tmp/kustomize-827653883/namespace-install' : lstat /tmp/kustomize-827653883/namespace-install: no such file or directory\\\", loader.New \\\"Error loading ../namespace-install with git: url lacks host: ../namespace-install, dir: evalsymlink failure on '/tmp/kustomize-827653883/namespace-install' : lstat /tmp/kustomize-827653883/namespace-install: no such file or directory, get: invalid source string: ../namespace-install\\\"\"" execID=uXqty
time="2020-06-19T23:40:21Z" level=error msg="finished unary call with code Unknown" error="`kustomize build /tmp/git@repo/argocd/overlay` failed exit status 1: Error: accumulating resources: accumulateFile \"accumulating resources from 'github.com/argoproj/argo-cd//manifests/cluster-install?ref=v1.5.7': evalsymlink failure on '/tmp/git@repo/argocd/overlay/github.com/argoproj/argo-cd/manifests/cluster-install?ref=v1.5.7' : lstat /tmp/git@repo/argocd/overlay/github.com: no such file or directory\", accumulateDirector: \"recursed accumulation of path '/tmp/kustomize-827653883/repo': accumulating resources: accumulateFile \\\"accumulating resources from '../namespace-install': evalsymlink failure on '/tmp/kustomize-827653883/namespace-install' : lstat /tmp/kustomize-827653883/namespace-install: no such file or directory\\\", loader.New \\\"Error loading ../namespace-install with git: url lacks host: ../namespace-install, dir: evalsymlink failure on '/tmp/kustomize-827653883/namespace-install' : lstat /tmp/kustomize-827653883/namespace-install: no such file or directory, get: invalid source string: ../namespace-install\\\"\"" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2020-06-19T23:41:14Z" grpc.service=repository.RepoServerService grpc.start_time="2020-06-19T23:40:14Z" grpc.time_ms=7284.024 span.kind=server system=grpc

Application controller log:

time="2020-06-19T23:43:21Z" level=info msg="Refreshing app status (comparison expired. reconciledAt: 2020-06-19 23:40:14 +0000 UTC, expiry: 3m0s), level (2)" application=argocd
time="2020-06-19T23:43:21Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: argocd)" application=argocd
time="2020-06-19T23:43:28Z" level=info msg="Normalized app spec: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2020-06-19T23:43:21Z\",\"message\":\"rpc error: code = Unknown desc = `kustomize build /tmp/git@repo/argocd/overlay` failed exit status 1: Error: accumulating resources: accumulateFile \\\"accumulating resources from 'github.com/argoproj/argo-cd//manifests/cluster-install?ref=v1.5.7': evalsymlink failure on '/tmp/git@repo/argocd/overlay/github.com/argoproj/argo-cd/manifests/cluster-install?ref=v1.5.7' : lstat /tmp/git@repo/argocd/overlay/github.com: no such file or directory\\\", accumulateDirector: \\\"recursed accumulation of path '/tmp/kustomize-531440274/repo': accumulating resources: accumulateFile \\\\\\\"accumulating resources from '../namespace-install': evalsymlink failure on '/tmp/kustomize-531440274/namespace-install' : lstat /tmp/kustomize-531440274/namespace-install: no such file or directory\\\\\\\", loader.New \\\\\\\"Error loading ../namespace-install with git: url lacks host: ../namespace-install, dir: evalsymlink failure on '/tmp/kustomize-531440274/namespace-install' : lstat /tmp/kustomize-531440274/namespace-install: no such file or directory, get: invalid source string: ../namespace-install\\\\\\\"\\\"\",\"type\":\"ComparisonError\"}]}}" application=argocd
time="2020-06-19T23:43:28Z" level=info msg="Update successful" application=argocd
time="2020-06-19T23:43:28Z" level=info msg="Reconciliation completed" application=argocd dedup_ms=0 dest-namespace=argocd dest-server="https://kubernetes.default.svc" diff_ms=21 fields.level=2 git_ms=6585 health_ms=4 live_ms=0 settings_ms=0 sync_ms=0 time_ms=6689

[alexmt]

Hello @jetersen ,

I think this is due to this bug in Kustomize 3.5.5: kubernetes-sigs/kustomize#2538 . Try changing git URL as following: https://github.com/argoproj/argo-cd/manifests/ha/cluster-install?ref=v1.5.7

Thanks,
Alex

[jetersen]

Yes it fixes it @alexmt though as I understand it would break again once go-getter fixes the issue which is not ideal 😭

[RyanSquared]

@jetersen actually, Kustomize follows a similar pattern to go-getter but does not use go-getter itself (this confused me as well, as a small optimization in Kustomize's version lead to me having issues not reproducible with go-getter).

source: kubernetes-sigs/kustomize#923 (comment)

[chicco785]

the new suggested format of url should reported in the documentation: https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#manage-argo-cd-using-argo-cd

[TeamDman]

This for me was an issue with the command kustomize edit add resource base/**/*.yaml where kustomize used backslashes in the paths in kustomization.yaml.

Manually switching the paths to forward slashes fixed the issue for me.
Running kustomize build on my windows machine worked, but I guess it doesn't play nice when running under linux with argo.

[SnoozeFreddo]

What's the reason behind the docs not reporting the new url? Nobody opening a PR or is there a special reason why it has to stay double slashed and without https?

[joebowbeer]

@SnoozeFreddo The standard kustomize URL format for remote directories contains both https and double slashes:

https://github.com/kubernetes-sigs/kustomize//examples/multibases/dev/?ref=v1.0.6

The legacy URL format does not:

github.com/Liujingfang1/kustomize/examples/helloWorld?ref=repoUrl2

The ArgoCD docs example is missing the https://

github.com/argoproj/argo-cd//manifests/cluster-install?ref=v1.0.1

[xgt001]

anyone else running into a variant of this when you refer the ../base/foo-file.yaml instead of ../base/ from your overlay? This format works exactly fine on my laptop (darwin arm64 + kustomize 5.0.1 as well as kustomize 5.1.0)
This fails only in argo