Could not get models from the server #6183

ShangWeiKuo · 2023-05-22T10:27:44Z

My actions before raising this issue

Read/searched the docs
Searched past issues
Docker version: 20.10.19
CVAT version: v2.4.1

Hello. I followed the steps to deploy nuclio dashboard

However. I got the error messages Error: Request failed with status code 502. "502 Server Error: Bad Gateway for url: http://nuclio:8070/api/functions". when I entered into localhost:8080

Besides, When I entered into localhost:8070, I also got two error messages as the pictures below.

How should I do to deal with the problems ?

Thank you for your help in advance.

This is log of nuclio showed by docker logs nuclio

Running in parallel
Starting dashboard
Starting nginx
23.05.22 23:51:53.648 hboard.healthcheck.server (I) Listening {"listenAddress": ":8082"}
23.05.22 23:51:53.648 rd.platform.docker.runner (D) Executing {"command": "docker version --format \"{{json .}}\""}
23.05.22 23:51:53.701 rd.platform.docker.runner (D) Command executed successfully {"output": "{\"Client\":{\"Platform\":{\"Name\":\"Docker Engine - Community\"},\"Version\":\"19.03.14\",\"ApiVersion\":\"1.40\",\"DefaultAPIVersion\":\"1.40\",\"GitCommit\":\"5eb3275\",\"GoVersion\":\"go1.13.15\",\"Os\":\"linux\",\"Arch\":\"amd64\",\"BuildTime\":\"Tue Dec  1 19:14:24 2020\",\"Experimental\":false},\"Server\":{\"Platform\":{\"Name\":\"Docker Engine - Community\"},\"Components\":[{\"Name\":\"Engine\",\"Version\":\"20.10.19\",\"Details\":{\"ApiVersion\":\"1.41\",\"Arch\":\"amd64\",\"BuildTime\":\"Thu Oct 13 16:44:11 2022\",\"Experimental\":\"false\",\"GitCommit\":\"c964641\",\"GoVersion\":\"go1.18.7\",\"KernelVersion\":\"4.15.0-206-generic\",\"MinAPIVersion\":\"1.12\",\"Os\":\"linux\"}},{\"Name\":\"containerd\",\"Version\":\"1.6.8\",\"Details\":{\"GitCommit\":\"9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6\"}},{\"Name\":\"nvidia\",\"Version\":\"1.1.4\",\"Details\":{\"GitCommit\":\"v1.1.4-0-g5fd4c4d\"}},{\"Name\":\"docker-init\",\"Version\":\"0.19.0\",\"Details\":{\"GitCommit\":\"de40ad0\"}}],\"Version\":\"20.10.19\",\"ApiVersion\":\"1.41\",\"MinAPIVersion\":\"1.12\",\"GitCommit\":\"c964641\",\"GoVersion\":\"go1.18.7\",\"Os\":\"linux\",\"Arch\":\"amd64\",\"KernelVersion\":\"4.15.0-206-generic\",\"BuildTime\":\"2022-10-13T16:44:11.000000000+00:00\"}}\n", "stderr": "", "exitCode": 0}
23.05.22 23:51:53.701 rd.platform.docker.runner (D) Executing {"command": "docker version --format \"{{json .}}\""}
23.05.22 23:51:53.746 rd.platform.docker.runner (D) Command executed successfully {"output": "{\"Client\":{\"Platform\":{\"Name\":\"Docker Engine - Community\"},\"Version\":\"19.03.14\",\"ApiVersion\":\"1.40\",\"DefaultAPIVersion\":\"1.40\",\"GitCommit\":\"5eb3275\",\"GoVersion\":\"go1.13.15\",\"Os\":\"linux\",\"Arch\":\"amd64\",\"BuildTime\":\"Tue Dec  1 19:14:24 2020\",\"Experimental\":false},\"Server\":{\"Platform\":{\"Name\":\"Docker Engine - Community\"},\"Components\":[{\"Name\":\"Engine\",\"Version\":\"20.10.19\",\"Details\":{\"ApiVersion\":\"1.41\",\"Arch\":\"amd64\",\"BuildTime\":\"Thu Oct 13 16:44:11 2022\",\"Experimental\":\"false\",\"GitCommit\":\"c964641\",\"GoVersion\":\"go1.18.7\",\"KernelVersion\":\"4.15.0-206-generic\",\"MinAPIVersion\":\"1.12\",\"Os\":\"linux\"}},{\"Name\":\"containerd\",\"Version\":\"1.6.8\",\"Details\":{\"GitCommit\":\"9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6\"}},{\"Name\":\"nvidia\",\"Version\":\"1.1.4\",\"Details\":{\"GitCommit\":\"v1.1.4-0-g5fd4c4d\"}},{\"Name\":\"docker-init\",\"Version\":\"0.19.0\",\"Details\":{\"GitCommit\":\"de40ad0\"}}],\"Version\":\"20.10.19\",\"ApiVersion\":\"1.41\",\"MinAPIVersion\":\"1.12\",\"GitCommit\":\"c964641\",\"GoVersion\":\"go1.18.7\",\"Os\":\"linux\",\"Arch\":\"amd64\",\"KernelVersion\":\"4.15.0-206-generic\",\"BuildTime\":\"2022-10-13T16:44:11.000000000+00:00\"}}\n", "stderr": "", "exitCode": 0}
23.05.22 23:51:53.746        dashboard.platform (D) Igniting container healthiness validator
23.05.22 23:51:53.746                 dashboard (D) Initializing platform {"platformType": "local"}
23.05.22 23:51:53.747 dashboard.platform.docker (D) Executing in container {"containerID": "nuclio-local-storage-reader", "execOptions": {"Command":"/bin/sh -c \"/bin/cat /etc/nuclio/store/projects/nuclio/default.json\"","Stdout":"","Stderr":"","Env":null}}
23.05.22 23:51:53.747 rd.platform.docker.runner (D) Executing {"command": "docker exec  nuclio-local-storage-reader /bin/sh -c \"/bin/cat /etc/nuclio/store/projects/nuclio/default.json\""}
23.05.22 23:51:53.782 rd.platform.docker.runner (D) Failed to execute command {"output": "Error: No such container: nuclio-local-storage-reader\n", "stderr": "", "exitCode": 1, "err": "exit status 1"}
23.05.22 23:51:53.782 dashboard.platform.docker (D) Failed to execute command in container {"err": "stdout:\nError: No such container: nuclio-local-storage-reader\n\nstderr:\n", "errVerbose": "\nError - exit status 1\n    /nuclio/pkg/cmdrunner/shellrunner.go:96\n\nCall stack:\nstdout:\nError: No such container: nuclio-local-storage-reader\n\nstderr:\n\n    /nuclio/pkg/cmdrunner/shellrunner.go:96\nstdout:\nError: No such container: nuclio-local-storage-reader\n\nstderr:\n", "stdout": "Error: No such container: nuclio-local-storage-reader\n", "stderr": ""}
23.05.22 23:51:53.782 dashboard.platform.docker (D) Running container {"imageName": "gcr.io/iguazio/alpine:3.15", "runOptions": {"Ports":null,"ContainerName":"nuclio-local-storage-reader","Env":null,"Labels":null,"Volumes":{"nuclio-local-storage":"/etc/nuclio/store"},"Remove":true,"Command":"/bin/sh -c \"/bin/sleep 6h\"","Stdout":"","Stderr":null,"Attach":false,"ImageMayNotExist":true,"Network":"","RestartPolicy":null,"GPUs":"","MountPoints":null,"RunAsUser":null,"RunAsGroup":null,"FSGroup":null}}
23.05.22 23:51:53.784 rd.platform.docker.runner (D) Executing {"command": "docker run -d --rm --name nuclio-local-storage-reader --volume nuclio-local-storage:/etc/nuclio/store  gcr.io/iguazio/alpine:3.15 /bin/sh -c \"/bin/sleep 6h\""}
23.05.22 23:51:54.522 rd.platform.docker.runner (D) Command executed successfully {"output": "a230259d6bc3106cfdc831454073e330a4d99281e73c70a5fcbfbd353135cb26\n", "stderr": "", "exitCode": 0}
23.05.22 23:51:54.522 dashboard.platform.docker (D) Executing in container {"containerID": "nuclio-local-storage-reader", "execOptions": {"Command":"/bin/sh -c \"/bin/cat /etc/nuclio/store/projects/nuclio/default.json\"","Stdout":"","Stderr":"","Env":null}}
23.05.22 23:51:54.522 rd.platform.docker.runner (D) Executing {"command": "docker exec  nuclio-local-storage-reader /bin/sh -c \"/bin/cat /etc/nuclio/store/projects/nuclio/default.json\""}
23.05.22 23:51:54.629 rd.platform.docker.runner (D) Command executed successfully {"output": "eyJtZXRhIjp7Im5hbWUiOiJkZWZhdWx0IiwibmFtZXNwYWNlIjoibnVjbGlvIn0sInNwZWMiOnt9LCJzdGF0dXMiOnsidXBkYXRlZEF0IjoiMjAyMy0wNS0yMlQwNzowMTo0MS43NjU0MDI1NjhaIn19\n", "stderr": "", "exitCode": 0}
23.05.22 23:51:54.630                 dashboard (D) Fetching function templates from git repository {"templatesGitRepository": "https://github.com/nuclio/nuclio-templates.git", "templatesGitRef": "refs/heads/master"}
23.05.22 23:51:54.643 itFunctionTemplateFetcher (D) Fetching templates from git {"ref": "refs/heads/master"}

Error - Get "https://github.com/nuclio/nuclio-templates.git/info/refs?service=git-upload-pack": dial tcp: lookup github.com on 127.0.0.11:53: read udp 127.0.0.1:49724->127.0.0.11:53: i/o timeout
    .../gitfunctiontemplatefetcher.go:109

Call stack:
Failed to initialize git repository
    .../gitfunctiontemplatefetcher.go:109
Failed to clone repository
    .../gitfunctiontemplatefetcher.go:67
Failed to fetch one of given templateFetchers
    .../dashboard/functiontemplates/repository.go:35
Failed to create repository out of given fetchers
    /nuclio/cmd/dashboard/app/run.go:286
Failed to create new dashboard
    /nuclio/cmd/dashboard/app/run.go:137
parallel: This job failed:
/runners/dashboard.sh
Exiting

The text was updated successfully, but these errors were encountered:

notpiri · 2023-05-23T01:49:31Z

I'm experiencing the same problem.

bsekachev · 2023-05-23T14:45:37Z

I see an error:

Error - Get "https://github.com/nuclio/nuclio-templates.git/info/refs?service=git-upload-pack": dial tcp: lookup github.com on 127.0.0.11:53: read udp 127.0.0.1:49724->127.0.0.11:53: i/o timeout

Try to wait for some time and run dashboad again, maybe it is a temporary network issue
Address the issue to nuclio community

ShangWeiKuo · 2023-05-24T05:46:10Z

Excuse me @bsekachev

Is it possible that I'm in the CVAT branch v2.4.1 instead of master so I could not get "https://github.com/nuclio/nuclio-templates.git/info/refs?service=git-upload-pack" ?

bsekachev · 2023-05-24T06:17:08Z

@ShangWeiKuo

I hardly think it depends on your branch.

MuhammadSYahyaS · 2023-08-02T10:32:49Z

I got the same issue on CVAT version v2.5.2. Any pointer?

Issue #6183 still exists in CVAT v2.5.2. Following up the addressed issue nuclio/nuclio#2938 in the [nuclio community](https://github.com/nuclio/nuclio), they suggested to use new nuclio version (1.11.x). It seems that the templates fetching mechanism on the old nuclio versions is not supported anymore, causing the mentioned issue. ### How has this been tested? Following the [Semi-automatic and Automatic Annotation](https://opencv.github.io/cvat/docs/administration/advanced/installation_automatic_annotation/) guideline on the CVAT v2.5.2 release branch (for the sake of stability) in a RHEL 8 machine, but changed the nuclio version to 1.11.24 and disabling the `/tmp` mounting to the nuclio container. Additionally, removed the unused static IP address value in the `no_proxy` environment. Tested using [SAM model](https://www.cvat.ai/post/facebook-segment-anything-model-in-cvat) with GPU following that guideline on some random images from a remote client.

…#6610) Issue cvat-ai#6183 still exists in CVAT v2.5.2. Following up the addressed issue nuclio/nuclio#2938 in the [nuclio community](https://github.com/nuclio/nuclio), they suggested to use new nuclio version (1.11.x). It seems that the templates fetching mechanism on the old nuclio versions is not supported anymore, causing the mentioned issue. ### How has this been tested? Following the [Semi-automatic and Automatic Annotation](https://opencv.github.io/cvat/docs/administration/advanced/installation_automatic_annotation/) guideline on the CVAT v2.5.2 release branch (for the sake of stability) in a RHEL 8 machine, but changed the nuclio version to 1.11.24 and disabling the `/tmp` mounting to the nuclio container. Additionally, removed the unused static IP address value in the `no_proxy` environment. Tested using [SAM model](https://www.cvat.ai/post/facebook-segment-anything-model-in-cvat) with GPU following that guideline on some random images from a remote client.

bsekachev closed this as completed May 23, 2023

MuhammadSYahyaS mentioned this issue Aug 3, 2023

Bump nuclio version and fix issues from the outdated version #6610

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Could not get models from the server #6183

Could not get models from the server #6183

ShangWeiKuo commented May 22, 2023 •

edited

Loading

notpiri commented May 23, 2023

bsekachev commented May 23, 2023 •

edited

Loading

ShangWeiKuo commented May 24, 2023 •

edited

Loading

bsekachev commented May 24, 2023

MuhammadSYahyaS commented Aug 2, 2023

Could not get models from the server #6183

Could not get models from the server #6183

Comments

ShangWeiKuo commented May 22, 2023 • edited Loading

My actions before raising this issue

notpiri commented May 23, 2023

bsekachev commented May 23, 2023 • edited Loading

ShangWeiKuo commented May 24, 2023 • edited Loading

bsekachev commented May 24, 2023

MuhammadSYahyaS commented Aug 2, 2023

ShangWeiKuo commented May 22, 2023 •

edited

Loading

bsekachev commented May 23, 2023 •

edited

Loading

ShangWeiKuo commented May 24, 2023 •

edited

Loading