Invalid memory address or nil pointer dereference in metadata_store.go

[rileyjbauer]

metadata_store.go may need some additional nil checks. I started seeing this error after running a pipeline Ning gave me:

I0322 20:09:29.856673       9 interceptor.go:29] /api.ReportService/ReportWorkflow handler starting
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x131dd94]

goroutine 641 [running]:
github.com/kubeflow/pipelines/backend/src/apiserver/metadata.(*Store).RecordOutputArtifacts(0xc00000e478, 0xc000451350, 0x24, 0xc000f69500, 0x3071, 0xc000bc9000, 0x421e, 0x1a75900, 0x0)
        backend/src/apiserver/metadata/metadata_store.go:53 +0x634
github.com/kubeflow/pipelines/backend/src/apiserver/storage.(*RunStore).UpdateRun(0xc0001d1290, 0xc000451350, 0x24, 0xc000e51a70, 0x9, 0xc000bc9000, 0x421e, 0x1, 0xc0004b8560)
        backend/src/apiserver/storage/run_store.go:380 +0xa3c
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).ReportWorkflowResource(0xc00073a4d0, 0xc0004b8560, 0x0, 0x0)
        backend/src/apiserver/resource/resource_manager.go:408 +0x86d
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ReportServer).ReportWorkflow(0xc00000e4b0, 0x1cdf780, 0xc000f0a6f0, 0xc000f0a720, 0xc00000e4b0, 0x1, 0x1)
        backend/src/apiserver/server/report_server.go:39 +0x53
github.com/kubeflow/pipelines/backend/api/go_client._ReportService_ReportWorkflow_Handler.func1(0x1cdf780, 0xc000f0a6f0, 0x1a40e60, 0xc000f0a720, 0x1, 0x0, 0xc00049d180, 0x0)
        bazel-out/k8-opt/bin/backend/api/linux_amd64_stripped/api_go_proto%/github.com/kubeflow/pipelines/backend/api/go_client/report.pb.go:174 +0x86
main.apiServerInterceptor(0x1cdf780, 0xc000f0a6f0, 0x1a40e60, 0xc000f0a720, 0xc00094bde0, 0xc00094be00, 0x195d7a0, 0x2a3a9c8, 0x1ac7b80, 0xc000fd9d00)
        backend/src/apiserver/interceptor.go:30 +0xf4
github.com/kubeflow/pipelines/backend/api/go_client._ReportService_ReportWorkflow_Handler(0x19671c0, 0xc00000e4b0, 0x1cdf780, 0xc000f0a6f0, 0xc0002af6d0, 0x1bacab8, 0x0, 0x0, 0xc000e37000, 0x4222)
        bazel-out/k8-opt/bin/backend/api/linux_amd64_stripped/api_go_proto%/github.com/kubeflow/pipelines/backend/api/go_client/report.pb.go:176 +0x158
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000463e00, 0x1ceb4e0, 0xc000a80a80, 0xc000fd9d00, 0xc0001d17a0, 0x29a64e0, 0x0, 0x0, 0x0)
        external/org_golang_google_grpc/server.go:966 +0x4a2
google.golang.org/grpc.(*Server).handleStream(0xc000463e00, 0x1ceb4e0, 0xc000a80a80, 0xc000fd9d00, 0x0)
        external/org_golang_google_grpc/server.go:1245 +0xd61
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000cbc270, 0xc000463e00, 0x1ceb4e0, 0xc000a80a80, 0xc000fd9d00)
        external/org_golang_google_grpc/server.go:685 +0x9f
created by google.golang.org/grpc.(*Server).serveStreams.func1
        external/org_golang_google_grpc/server.go:683 +0xa1

[neuromage]

What's the pipeline? Is this error in the API server?

[rileyjbauer]

Yes, this error is in the API server. Here is the pipeline:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: pipeline-flip-coin-
spec:
  arguments:
    parameters: []
  entrypoint: pipeline-flip-coin
  serviceAccountName: pipeline-runner
  templates:
  - dag:
      tasks:
      - arguments:
          parameters:
          - name: flip-output
            value: '{{inputs.parameters.flip-2-output}}'
        name: graph-flip-component-1
        template: graph-flip-component-1
    inputs:
      parameters:
      - name: flip-2-output
    name: condition-2
  - container:
      args:
      - python -c "import random; result = 'heads' if random.randint(0,1) == 0 else
        'tails'; print(result)" | tee /tmp/output
      command:
      - sh
      - -c
      image: python:alpine3.6
    name: flip
    outputs:
      artifacts:
      - name: mlpipeline-ui-metadata
        path: /mlpipeline-ui-metadata.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-ui-metadata.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
      - name: mlpipeline-metrics
        path: /mlpipeline-metrics.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-metrics.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
      parameters:
      - name: flip-output
        valueFrom:
          path: /tmp/output
  - container:
      args:
      - python -c "import random; result = 'heads' if random.randint(0,1) == 0 else
        'tails'; print(result)" | tee /tmp/output
      command:
      - sh
      - -c
      image: python:alpine3.6
    name: flip-2
    outputs:
      artifacts:
      - name: mlpipeline-ui-metadata
        path: /mlpipeline-ui-metadata.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-ui-metadata.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
      - name: mlpipeline-metrics
        path: /mlpipeline-metrics.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-metrics.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
      parameters:
      - name: flip-2-output
        valueFrom:
          path: /tmp/output
  - dag:
      tasks:
      - arguments:
          parameters:
          - name: flip-2-output
            value: '{{tasks.flip-2.outputs.parameters.flip-2-output}}'
        dependencies:
        - flip-2
        name: condition-2
        template: condition-2
        when: '{{tasks.flip-2.outputs.parameters.flip-2-output}} == heads'
      - dependencies:
        - print
        name: flip-2
        template: flip-2
      - arguments:
          parameters:
          - name: flip-output
            value: '{{inputs.parameters.flip-output}}'
        name: print
        template: print
    inputs:
      parameters:
      - name: flip-output
    name: graph-flip-component-1
    outputs:
      parameters:
      - name: flip-2-output
        valueFrom:
          parameter: '{{tasks.flip-2.outputs.parameters.flip-2-output}}'
  - dag:
      tasks:
      - name: flip
        template: flip
      - arguments:
          parameters:
          - name: flip-output
            value: '{{tasks.flip.outputs.parameters.flip-output}}'
        dependencies:
        - flip
        name: graph-flip-component-1
        template: graph-flip-component-1
      - arguments:
          parameters:
          - name: flip-2-output
            value: '{{tasks.graph-flip-component-1.outputs.parameters.flip-2-output}}'
        dependencies:
        - graph-flip-component-1
        name: print-2
        template: print-2
    name: pipeline-flip-coin
  - container:
      command:
      - echo
      - '{{inputs.parameters.flip-output}}'
      image: alpine:3.6
    inputs:
      parameters:
      - name: flip-output
    name: print
    outputs:
      artifacts:
      - name: mlpipeline-ui-metadata
        path: /mlpipeline-ui-metadata.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-ui-metadata.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
      - name: mlpipeline-metrics
        path: /mlpipeline-metrics.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-metrics.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
  - container:
      command:
      - echo
      - cool, it is over. {{inputs.parameters.flip-2-output}}
      image: alpine:3.6
    inputs:
      parameters:
      - name: flip-2-output
    name: print-2
    outputs:
      artifacts:
      - name: mlpipeline-ui-metadata
        path: /mlpipeline-ui-metadata.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-ui-metadata.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
      - name: mlpipeline-metrics
        path: /mlpipeline-metrics.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-metrics.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact

[neuromage]

Thanks Riley. I figured out the problem. Will send out a fix with a corresponding test in a short while.

[neuromage]

@rileyjbauer I believe the problem is fixed. Would you mind trying this out again from master and verify that it works? Thanks!

[rileyjbauer]

That seems to have done the trick! Thanks!