Unable to run Tekton pipelines locally since move away from InitContainers

[a-roberts]

For full context see https://knative.slack.com/archives/CDCQ72D0R/p1551956671118100

Caveat: we are doing this locally and our pipeline involves resources. We're creating the PipelineRun and the resources through Go code - something we plan to contribute as an example, based on receiving a webhook from a repository. I don't know if this is a problem with our own code or with Knative itself, just that since the move away from InitContainers we have the problem. Am I missing something?

Expected Behavior

Follow the docs for local development here and you should be able to create PipelineRuns that run successfully locally, using images from your local Docker registry. Pipelines and all associated tasks should run perfectly.

Actual Behavior

PipelineRuns fail citing tasks are not found:

adams-mbp:build-pipeline aroberts$ k get pipelinerun -o json
{
    "apiVersion": "v1",
    "items": [
        {
            "apiVersion": "tekton.dev/v1alpha1",
            "kind": "PipelineRun",
            "metadata": {
                "creationTimestamp": "2019-03-07T10:56:32Z",
                "generation": 1,
                "labels": {
                    "app": "devops-knative"
                },
                "name": "devops-pipeline-run-1551956192",
                "namespace": "default",
                "resourceVersion": "215773",
                "selfLink": "/apis/tekton.dev/v1alpha1/namespaces/default/pipelineruns/devops-pipeline-run-1551956192",
                "uid": "ab1967c6-40c7-11e9-9495-025000000001"
            },
            "spec": {
                "Status": "",
                "params": null,
                "pipelineRef": {
                    "name": "simple-pipeline"
                },
                "resources": [
                    {
                        "name": "docker-image",
                        "resourceRef": {
                            "name": "docker-image-1551956192"
                        }
                    },
                    {
                        "name": "git-source",
                        "resourceRef": {
                            "name": "git-source-1551956192"
                        }
                    }
                ],
                "serviceAccount": "default",
                "timeout": "1h0m0s",
                "trigger": {
                    "type": "manual"
                }
            },
            "status": {
                "conditions": [
                    {
                        "lastTransitionTime": "2019-03-07T10:56:32Z",
                        "message": "TaskRun devops-pipeline-run-1551956192-build-simple-6wlst has failed",
                        "reason": "Failed",
                        "status": "False",
                        "type": "Succeeded"
                    }
                ],
                "startTime": "2019-03-07T10:56:32Z",
                "taskRuns": {
                    "devops-pipeline-run-1551956192-build-simple-6wlst": {
                        "pipelineTaskName": "build-simple",
                        "status": {
                            "conditions": [
                                {
                                    "lastTransitionTime": "2019-03-07T10:56:32Z",
                                    "message": "References a Task default/build-push that doesn't exist:  couldn't create redirected Build: failed to add entrypoint to steps of TaskRun devops-pipeline-run-1551956192-build-simple-6wlst: failed to fetch remote image localhost:5000/knative/git-init-afd2a379df7ac007f1e3a5fc75688a50@sha256:6bf4f67a86fa3e8fc9066835231b853bda76df92fd76b06e72227b7a1c23d39e: couldn't get container image info from registry localhost:5000/knative/git-init-afd2a379df7ac007f1e3a5fc75688a50@sha256:6bf4f67a86fa3e8fc9066835231b853bda76df92fd76b06e72227b7a1c23d39e: Get http://localhost:5000/v2/: dial tcp 127.0.0.1:5000: connect: connection refused",
                                    "reason": "CouldntGetTask",
                                    "status": "False",
                                    "type": "Succeeded"
                                }
                            ],
                            "podName": "",
                            "startTime": "2019-03-07T10:56:32Z"
                        }
                    },
                    "devops-pipeline-run-1551956192-deploy-simple-9t7n4": {
                        "pipelineTaskName": "deploy-simple",
                        "status": {
                            "conditions": [
                                {
                                    "lastTransitionTime": "2019-03-07T10:56:32Z",
                                    "message": "References a Task default/deploy-simple-kubectl-task that doesn't exist:  couldn't create redirected Build: failed to add entrypoint to steps of TaskRun devops-pipeline-run-1551956192-deploy-simple-9t7n4: failed to fetch remote image localhost:5000/knative/git-init-afd2a379df7ac007f1e3a5fc75688a50@sha256:6bf4f67a86fa3e8fc9066835231b853bda76df92fd76b06e72227b7a1c23d39e: couldn't get container image info from registrylocalhost:5000/knative/git-init-afd2a379df7ac007f1e3a5fc75688a50@sha256:6bf4f67a86fa3e8fc9066835231b853bda76df92fd76b06e72227b7a1c23d39e: Get http://localhost:5000/v2/: dial tcp 127.0.0.1:5000: connect: connection refused",
                                    "reason": "CouldntGetTask",
                                    "status": "False",
                                    "type": "Succeeded"
                                }
                            ],
                            "podName": "",
                            "startTime": "2019-03-07T10:56:32Z"
                        }
                    }
                }
            }
        }
    ],

Tasks exist in the Kube sense but not in the images sense (If I do kubectl get tasks, I do see my tasks).

Steps to Reproduce the Problem

1. Follow docs here so that knative is installed locally https://github.com/knative/build-pipeline/blob/master/docs/tutorial.md#local-development. Follow any example to create a PipelineRun with tasks.
2. Observe the error when you do kubectl get pipelinerun -o json and observe that none of your tasks are run successfully.

Todo: I'm gonna add the noddiest reproduce with tiny tasks and a tiny PipelineRun.

Additional Info

If you git checkout any commit prior to c0285b30e9720ea4f1a0666c97db424362b411b5, there are no problems. The PipelineRun progresses to completion.

I believe the tasks should now be containers/images that are pushed to the local registry and we should be able to pull from them. I think the images do exist but they can't be pulled.

I've never had to use pull secrets locally and we're using the default service account (there wouldn't be any credentials anyway for my insecure lil registry).

We have a couple theories involving changes in entrypoint.go regarding authorization and another involving the use of ko.

Hopefully we just need to add to the docs again or it's a minor config issue on my end.

@abayer @pivotal-nader-ziada FYI

Here are the task definitions:

build-task.yaml:

apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: build-push
spec:
  inputs:
    resources:
      - name: git-source
        type: git
    params:
    - name: pathToDockerFile
      description: The path to the dockerfile to build
      default: /workspace/git-source/Dockerfile
    - name: pathToContext
      description: The build context used by Kaniko (https://github.com/GoogleContainerTools/kaniko#kaniko-build-contexts)
      default: /workspace/git-source
  outputs:
    resources:
      - name: builtImage
        type: image
  steps:
  - name: build
    image: docker
    command:
    - docker 
    args:
    - 'build'
    - '-f'
    - ${inputs.params.pathToDockerFile}
    - '-t'
    - ${outputs.resources.builtImage.url}
    - ${inputs.params.pathToContext}
    volumeMounts:
    - name: docker-socket
      mountPath: /var/run/docker.sock
  - name: push 
    image: docker
    command:
    - docker 
    args:
    - 'push'
    - ${outputs.resources.builtImage.url}
    volumeMounts:
    - name: docker-socket
      mountPath: /var/run/docker.sock
  volumes:
    - name: docker-socket
      hostPath:
        path: /var/run/docker.sock
        type: Socket

deploy-task.yaml:

apiVersion: tekton.dev/v1alpha1
kind: Task
metadata: 
  name: deploy-simple-kubectl-task
spec:
  inputs: 
    resources: 
    - name: git-source
      type: git
    - name: image-out
      type: image
  steps:
  - name: modify-yaml
    image: ubuntu
    command: ['/bin/bash']
    args: ['-c', 'find /workspace/git-source/config -type f -exec sed -i -e s!"image: PIPELINE_REPLACE:latest"!"image: ${IMG}"!g {} \;']
    env:
      - name: IMG
        value: ${inputs.resources.image-out.url}
  - name: install-new
    image: lachlanevenson/k8s-kubectl
    command: ['kubectl']
    args: 
    - 'apply'
    - '-f'
    - '/workspace/git-source/config'

(we replace the needle here with the commit ID so you can build and push your latest code)

pipeline.yaml:

apiVersion: tekton.dev/v1alpha1
kind: Pipeline
metadata:
  name: simple-pipeline
spec:
  resources: 
  - name: git-source
    type: git
  - name: docker-image
    type: image
  tasks:
  - name: build-simple
    taskRef:
      name: build-push
    resources:
      inputs:
      - name: git-source
        resource: git-source
      outputs:
      - name: builtImage
        resource: docker-image
  - name: deploy-simple
    #runAfter: [build-simple]
    taskRef: 
      name: deploy-simple-kubectl-task
    resources: 
      inputs:
      - name: git-source
        resource: git-source 
      - name: image-out
        resource: docker-image

[bobcatfish]

Whoops, looks like a bug for sure, sorry for the hassle!! We'll get on this right away

btw awesome issue write up @a-roberts thanks so much 🙏

[abayer]

Blech, my internet connection is terrible and I've never actually used Docker for Mac in a meaningful way, so I don't think I can get this reproduced locally. If someone else can tackle it, that'd probably be for the best.

[vdemeester]

/assign

[vdemeester]

To complete a bit the bug decription, it's also the case on minikube with a simple case like that (because of git-init containers that do no specify command)

apiVersion: tekton.dev/v1alpha1
kind: PipelineResource
metadata:
  name: skaffold-git
spec:
  type: git
  params:
  - name: revision
    value: master
  - name: url
    value: https://github.com/GoogleContainerTools/skaffold
---
#Builds an image via kaniko and pushes it to registry.
apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: build-simple
spec:
  inputs:
    resources:
    - name: workspace
      type: git
    params:
    - name: pathToDockerFile
      description: The path to the dockerfile to build
      default: /workspace/workspace/Dockerfile
    - name: pathToContext
      description: The build context used by Kaniko (https://github.com/GoogleContainerTools/kaniko#kaniko-build-contexts)
      default: /workspace/workspace
  steps:
  - name: build-simple
    image: docker.io/library/busybox
    command:
    - /bin/sh
    args:
    - -c
    - "ls -l ${pathToDockerFile} ${pathToContext}"
---
apiVersion: tekton.dev/v1alpha1
kind: TaskRun
metadata:
  name: build-simple
spec:
  taskRef:
    name: build-simple
  trigger:
    type: manual
  inputs:
    resources:
    - name: workspace
      resourceRef:
        name: skaffold-git
    params:
    - name: pathToDockerFile
      value: Dockerfile
    - name: pathToContext
      value: /workspace/workspace/examples/microservices/leeroy-web

Thinking outloud of possible fixes and workaround :

1. As we do remote lookup only when the command is not specified, we can, for internal containers (copy, git-init, creds-init, …) provide command and thus make it work. This is the easiest mitigation step.
2. For actually steps provided by the user, we may try to handle some edge/special cases, like ko.local or localhost:* remote parts, where we should be able to look directly into the cri-* image store (using go-containerregistry. That said, it means we need to add support for those in go-containerregistry and we're gonna need to get access to those cri-* implementation from the controller… it feels like a lot to do… (and as soon as a new cri-* implementation pops up we may need to add support for it in go-containerregistry)

I'm gonna go ahead and fix 1., but we're gonna need to discuss if 2. is worth the work or if we should just document that images in steps have to reference to registries that are accessible from the cluster (docker.io, quay.io,gcr.io, …, ip that containers from the cluster can access to, …)