Library does not download the extra stack files

[valaparthvi]

Which area this feature is related to?

/kind bug

Which area this bug is related to?

/area library
/area registry

What versions of software are you using?

github.com/devfile/registry-support/registry-library v0.0.0-20221220163850-cab341c8f8f8

Go project

Operating System and version:
Fedora 35

Go Pkg Version:
go version go1.18.5 linux/amd64

Bug Summary

Describe the bug:

registry-support does not download the extra stack files.

To Reproduce:

1. mkdir /tmp/demo && cd /tmp/demo
2. Copy this main.go. The logic has been taken from here.

main.go

package main

import (
	registryLibrary "github.com/devfile/registry-support/registry-library/library"
)

func main() {
	stack := "java-springboot:latest"
	destDir := "/tmp/demo"
	registryURL := "https://registry.devfile.io"
	options := registryLibrary.RegistryOptions{
		NewIndexSchema: true,
	}
	err := registryLibrary.PullStackFromRegistry(registryURL, stack, destDir, options)
	if err != nil {
		panic(err)
	}
}

3. Run the above go file that attempts to download java-springboot:2.0.0 which comes with extra stack files.
4. It only download the Devfile.

Expected behavior

It should download the extra stack files, in this case kubernetes and docker directory.

Looking at https://github.com/devfile/registry-support/blob/main/registry-library/library/library.go#L281, it should download 'archive.tar' in the destination directory, in this case /tmp/demo, but it does not.

Any logs, error output, screenshots etc? Provide the devfile that sees this bug, if applicable

Additional context

odo issue: redhat-developer/odo#6448

This bug is also reproducible when I used registry-library binary with the following command.

➜ ./registry-library pull https://registry.devfile.io go:latest --context /tmp/demo --new-index-schema --all      
➜ less /tmp/demo/devfile.yaml      
➜ ls /tmp/demo                   
devfile.yaml

Any workaround?

Suggestion on how to fix the bug

Target Date: 01-13-2023

[kim-tsao]

Looks like the index is not generated correctly. The resources property only lists the devfile.yaml file when it should also include the other files.

 {
    "name": "java-springboot",
    "displayName": "Spring Boot",
    "description": "Spring Boot using Java",
    "type": "stack",
    "tags": [
      "Java",
      "Spring Boot"
    ],
    "icon": "https://spring.io/images/projects/spring-edf462fec682b9d48cf628eaf9e19521.svg",
    "projectType": "springboot",
    "language": "Java",
    "versions": [
      {
        "version": "2.0.0",
        "schemaVersion": "2.2.0",
        "description": "Spring Boot® using Java",
        "tags": [
          "Java",
          "Spring Boot"
        ],
        "icon": "https://spring.io/images/projects/spring-edf462fec682b9d48cf628eaf9e19521.svg",
        "links": {
          "self": "devfile-catalog/java-springboot:2.0.0"
        },
        "resources": [
          "devfile.yaml"
        ],
        "starterProjects": [
          "springbootproject"
        ]
      },

[valaparthvi]

As a side note, this issue blocks odo from using the latest devfile/library(on main branch). devfile/library tries to convert URI-ed data to inline-d while parsing a Devfile, and since it does not find the necessary stack files, it throws the following error:

$ mkcd /tmp/demo
$ odo init --devfile go --name my-go-app --devfile-version 2.0.0
  __
 /  \__     Initializing a new component
 \__/  \    
 /  \__/    odo version: v3.5.0
 \__/

 ✓  Downloading devfile "go:2.0.0" [4s]
 ✗  unable to parse devfile: failed to parse kubernetes/openshift component from uri to inlined: failed to convert Kubernetes Uri to inlined for component 'deploy': failed to read kubernetes resources definition from path '/tmp/demo/kubernetes/deploy.yaml': open /tmp/demo/kubernetes/deploy.yaml: no such file or directory
the command failed, the devfile has been removed from current directory

Unless this issue is fixed, I think we'd be blocked on using the latest devfile/library(on main branch) which in turn blocks us from fixing another issue that requires using the recent container-overrides and pod-overrides changes.

[michael-valdron]

devfile/registry-support#156 looks to fix this issue discovered in the registry library.

[michael-valdron]

@valaparthvi The fix is now in staging and production. This issue should now be resolved.

[kim-tsao]

@kadel FYI, since some stacks may have an archive.tar file now, there will be an additional telemetry download event. Explanation given here: #933 (comment)

[valaparthvi]

@valaparthvi The fix is now in staging and production. This issue should now be resolved.

Great! Thank you for fixing it @michael-valdron @kim-tsao! I can confirm it works for odo.