Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DEXTR - Semi-Automatic Annotation - 504 Gateway Timeout #3301

Closed
2 tasks
rathishkumar opened this issue Jun 8, 2021 · 19 comments
Closed
2 tasks

DEXTR - Semi-Automatic Annotation - 504 Gateway Timeout #3301

rathishkumar opened this issue Jun 8, 2021 · 19 comments
Assignees
Labels
bug Something isn't working nuclio

Comments

@rathishkumar
Copy link

My actions before raising this issue

I have searched the docs and past issues - could not figure it out, kindly help.

Expected Behaviour

Automatic instance segmentation annotation

Current Behaviour

Getting below error:

Gateway Timeout
The gateway did not receive a timely response from the upstream server or application.

Possible Solution

From logs, it looks like connection not reaching the docker instance - may be providing some advice on configuring proxy/firewall will help us to understand the issue better.

Steps to Reproduce (for bugs)

Followed the steps provided in the installation guide:

https://openvinotoolkit.github.io/cvat/docs/for-users/installation/

Note: Installation is successful, I will share the logs and details below

Context

When trying to use AI Tool getting timeout error

Your Environment

  • Git hash commit (git log -1):
commit 3afd7e827878112e6d6bf9a435e7b96d38baaa26 (HEAD -> develop, origin/develop, origin/HEAD)
Author: Dmitry Kruchinin <33020454+dvkruchinin@users.noreply.github.com>
Date:   Mon Jun 7 11:59:10 2021 +0300
    Cypress test. Create an object, save, undo, save, redo, save. (#3299)
  • Docker version docker version (e.g. Docker 17.0.05):
Client: Docker Engine - Community
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 12:00:45 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:58:56 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
  • Are you using Docker Swarm or Kubernetes? No

  • Operating System and version (e.g. Linux, Windows, MacOS):

NAME="Ubuntu"
VERSION="21.04 (Hirsute Hippo)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 21.04"
VERSION_ID="21.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=hirsute
UBUNTU_CODENAME=hirsute
  • Code example or link to GitHub repo or gist to reproduce problem: Same as official repo

  • Other logs:

docker ps:

CONTAINER ID   IMAGE                                   COMMAND                  CREATED        STATUS                  PORTS                                                                                
                             NAMES
3e43c3e7b553   alpine:3.11                             "/bin/sh -c '/bin/sl…"   5 hours ago    Up 5 hours                                                                                                   
                             nuclio-local-storage-reader
296d755cc856   cvat/openvino.dextr:latest              "processor"              11 hours ago   Up 11 hours (healthy)   0.0.0.0:49153->8080/tcp, :::49153->8080/tcp                                          
                             nuclio-nuclio-openvino-dextr
8e4e3f6f04ae   nginx:stable-alpine                     "/docker-entrypoint.…"   11 hours ago   Up 11 hours             0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:808
0->80/tcp, :::8080->80/tcp   cvat_proxy
f1ec02907177   openvino/cvat_ui                        "/docker-entrypoint.…"   11 hours ago   Up 11 hours             80/tcp                                                                               
                             cvat_ui
1b32376d2c98   openvino/cvat_server                    "/usr/bin/supervisord"   11 hours ago   Up 11 hours             8080/tcp                                                                             
                             cvat
19675add9961   postgres:10-alpine                      "docker-entrypoint.s…"   11 hours ago   Up 11 hours             5432/tcp                                                                             
                             cvat_db
f52bcf45003b   redis:4.0-alpine                        "docker-entrypoint.s…"   11 hours ago   Up 11 hours             6379/tcp                                                                             
                             cvat_redis
0090c6624c5d   quay.io/nuclio/dashboard:1.5.16-amd64   "/docker-entrypoint.…"   11 hours ago   Up 11 hours (healthy)   80/tcp, 0.0.0.0:8070->8070/tcp, :::8070->8070/tcp                                    
                             nuclio

docker logs cvat: (only error part)

[Mon Jun 07 18:55:17.009057 2021] [wsgi:error] [pid 255:tid 140652454614784] [remote 172.28.0.7:34580] [2021-06-07 18:55:17,008] INFO cvat.server.task_2: get repository request
2021-06-07 18:55:17,010 DEBG 'runserver' stderr output:
[Mon Jun 07 18:55:17.010097 2021] [wsgi:error] [pid 255:tid 140652454614784] [remote 172.28.0.7:34580] INFO - 2021-06-07 18:55:17,008 - views - get repository request
2021-06-07 18:55:20,822 DEBG 'runserver' stderr output:
[Mon Jun 07 18:55:20.822152 2021] [wsgi:error] [pid 255:tid 140652437829376] [remote 172.28.0.7:34608] INFO - 2021-06-07 18:55:20,822 - views - {"client_id":582112,"name":"Send user activity","time":"2021
-06-07T18:55:20.723000Z","payload":{"working_time":50033},"is_active":true,"username":"admin"}
2021-06-07 18:56:35,536 DEBG 'runserver' stderr output:
[Mon Jun 07 18:56:35.536331 2021] [wsgi:error] [pid 322:tid 140652568270592] [client 172.28.0.7:34694] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/a
pi, referer: https://my-server-ip/tasks/2/jobs/2
2021-06-07 18:56:35,637 DEBG 'runserver' stderr output:
[Mon Jun 07 18:56:35.637400 2021] [wsgi:error] [pid 255:tid 140652446222080] [remote 172.28.0.7:34694] [2021-06-07 18:56:35,637] ERROR django.request: Gateway Timeout: /api/v1/lambda/functions/openvino-de
xtr
2021-06-07 18:56:35,637 DEBG 'runserver' stderr output:
[Mon Jun 07 18:56:35.637517 2021] [wsgi:error] [pid 255:tid 140652446222080] [remote 172.28.0.7:34694] ERROR - 2021-06-07 18:56:35,637 - log - Gateway Timeout: /api/v1/lambda/functions/openvino-dextr
2021-06-07 18:56:51,575 DEBG 'runserver' stderr output:
[Mon Jun 07 18:56:51.575198 2021] [wsgi:error] [pid 255:tid 140652454614784] [remote 172.28.0.7:35000] INFO - 2021-06-07 18:56:51,575 - views - {"job_id":2,"task_id":2,"client_id":582112,"name":"Load job"
,"time":"2021-06-07T18:55:20.723000Z","payload":{"duration":1450,"frame count":7,"track count":0,"object count":0,"box count":0,"polygon count":0,"polyline count":0,"points count":0,"cuboids count":0,"tag
 count":0},"is_active":true,"username":"admin"}
2021-06-07 18:56:51,575 DEBG 'runserver' stderr output:
[Mon Jun 07 18:56:51.575359 2021] [wsgi:error] [pid 255:tid 140652454614784] [remote 172.28.0.7:35000] INFO - 2021-06-07 18:56:51,575 - views - {"job_id":2,"task_id":2,"client_id":582112,"name":"Fit image
","time":"2021-06-07T18:55:22.176000Z","payload":{},"is_active":true,"username":"admin"}
[Mon Jun 07 18:56:51.575465 2021] [wsgi:error] [pid 255:tid 140652454614784] [remote 172.28.0.7:35000] INFO - 2021-06-07 18:56:51,575 - views - {"client_id":582112,"name":"Send user activity","time":"2021
-06-07T18:56:51.481000Z","payload":{"working_time":14505},"is_active":true,"username":"admin"}
2021-06-07 18:57:30,995 DEBG 'runserver' stderr output:
[Mon Jun 07 18:57:30.995435 2021] [wsgi:error] [pid 255:tid 140652437829376] [remote 172.28.0.7:35178] [2021-06-07 18:57:30,995] INFO cvat.server.task_2: get repository request
2021-06-07 18:57:30,995 DEBG 'runserver' stderr output:
[Mon Jun 07 18:57:30.995579 2021] [wsgi:error] [pid 255:tid 140652437829376] [remote 172.28.0.7:35178] INFO - 2021-06-07 18:57:30,995 - views - get repository request
2021-06-07 18:58:41,613 DEBG 'rqworker_low' stderr output:
DEBUG - 2021-06-07 18:58:41,612 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.
2021-06-07 18:58:41,812 DEBG 'rqworker_default_1' stderr output:
DEBUG - 2021-06-07 18:58:41,812 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.
2021-06-07 18:59:59,108 DEBG 'runserver' stderr output:
[Mon Jun 07 18:59:59.108022 2021] [wsgi:error] [pid 255:tid 140652454614784] [remote 172.28.0.7:35766] INFO - 2021-06-07 18:59:59,107 - views - {"client_id":582112,"name":"Send user activity","time":"2021
-06-07T18:59:59.009000Z","payload":{"working_time":50882},"is_active":true,"username":"admin"}
2021-06-07 19:01:13,421 DEBG 'runserver' stderr output:
[Mon Jun 07 19:01:13.421313 2021] [wsgi:error] [pid 322:tid 140652568004352] [client 172.28.0.7:35852] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/a
pi, referer: https://my-server-ip/tasks/2/jobs/2
2021-06-07 19:01:13,590 DEBG 'runserver' stderr output:
[Mon Jun 07 19:01:13.590114 2021] [wsgi:error] [pid 255:tid 140652437829376] [remote 172.28.0.7:35852] [2021-06-07 19:01:13,590] ERROR django.request: Gateway Timeout: /api/v1/lambda/functions/openvino-de
xtr
2021-06-07 19:01:13,590 DEBG 'runserver' stderr output:
[Mon Jun 07 19:01:13.590248 2021] [wsgi:error] [pid 255:tid 140652437829376] [remote 172.28.0.7:35852] ERROR - 2021-06-07 19:01:13,590 - log - Gateway Timeout: /api/v1/lambda/functions/openvino-dextr
2021-06-07 19:01:33,901 DEBG 'runserver' stderr output:
[Mon Jun 07 19:01:33.901047 2021] [core:info] [pid 322:tid 140652568803072] [client 172.28.0.7:36202] AH00128: File does not exist: /home/django/static/rest_framework/css/bootstrap.min.css.map
2021-06-07 19:01:54,672 DEBG 'rqworker_default_0' stderr output:
DEBUG - 2021-06-07 19:01:54,671 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

nuctl get function:

  NAMESPACE |      NAME      | PROJECT | STATE | NODE PORT | REPLICAS  
  nuclio    | openvino-dextr | cvat    | ready |     49153 | 1/1  

nuctl version:

Client version:
"Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3"root@gc-ase-ml-data-annotate1:/data/cvat# 

systemctl show --property=Environment docker:

Environment=HTTP_PROXY=http://my-proxy-ip:8080 HTTPS_PROXY=http://my-proxy-ip:8080 NO_PROXY=localhost,127.0.0.1

cat ~/.docker/config.json:

{
 "proxies":
 {
   "default":
   {
     "httpProxy": "http://my-proxy-ip:8080",
     "httpsProxy": "http://my-proxy-ip:8080",
     "noProxy": "*.mydomain.com,localhost,127.0.0.0/8"
   }
 }
}

docker-compose.override.yml:

version: '3.3'
services:
  cvat_proxy:
    environment:
      CVAT_HOST: 'my-host-ip'
    ports:
      - '80:80'
      - '443:443'
    volumes:
      - /etc/ssl/certs:/etc/ssl/certs
  cvat:
    environment:
      ALLOWED_HOSTS: '*'

Screenshots:

image

image

API Page:

image

API Output:

image

Logs for this request:

image

Next steps

I have joined gitter and shared details - waiting for response.

@azhavoro
Copy link
Contributor

azhavoro commented Jun 9, 2021

@rathishkumar Hi, make sure the no_proxy environment variable of the cvat container contains nuclio: docker exec cvat env | grep no_proxy

@azhavoro azhavoro added the need info Need more information to investigate the issue label Jun 9, 2021
@rathishkumar
Copy link
Author

@azhavoro Output of docker exec cvat env | grep no_proxy

no_proxy=kibana,logstash,nuclio

Is there anything else, need to check? like need to add 172.28.0.0/24?

@azhavoro
Copy link
Contributor

azhavoro commented Jun 9, 2021

please attach logs from nuclio docker logs nuclio > nuclio.log

@rathishkumar
Copy link
Author

Sure, @azhavoro , its 190 MB, I am attaching the tail part of it - please let me know, is there any other way to attach complete/zipped log here, if required.

21.06.09 12:33:13.376 �[0;37mdashboard.platform.docker�[0m �[0;32m(D)�[0m Container is healthy {"containerID": "296d755cc856341fd4d244df57b567967157ada0272100abe65bb18f91031f7b"}
21.06.09 12:33:43.174 �[0;37mdashboard.platform.docker�[0m �[0;32m(D)�[0m Executing in container {"containerID": "nuclio-local-storage-reader", "execOptions": {"Command":"/bin/sh -c \"/bin/cat /etc/nuclio/store/functions/nuclio/*\"","Stdout":"","Stderr":"","Env":null}}
21.06.09 12:33:43.174 �[0;37mrd.platform.docker.runner�[0m �[0;32m(D)�[0m Executing {"command": "docker exec  nuclio-local-storage-reader /bin/sh -c \"/bin/cat /etc/nuclio/store/functions/nuclio/*\""}
21.06.09 12:33:43.270 �[0;37mrd.platform.docker.runner�[0m �[0;32m(D)�[0m Command executed successfully {"output": "eyJtZXRhZGF0YSI6eyJuYW1lIjoib3BlbnZpbm8tZGV4dHIiLCJuYW1lc3BhY2UiOiJudWNsaW8iLCJsYWJlbHMiOnsibnVjbGlvLmlvL3Byb2plY3QtbmFtZSI6ImN2YXQifSwiYW5ub3RhdGlvbnMiOnsiZnJhbWV3b3JrIjoib3BlbnZpbm8iLCJtaW5fcG9zX3BvaW50cyI6IjQiLCJuYW1lIjoiREVYVFIiLCJzcGVjIjoiIiwidHlwZSI6ImludGVyYWN0b3IifX0sInNwZWMiOnsiZGVzY3JpcHRpb24iOiJEZWVwIEV4dHJlbWUgQ3V0IiwiaGFuZGxlciI6Im1haW46aGFuZGxlciIsInJ1bnRpbWUiOiJweXRob246My42IiwiZW52IjpbeyJuYW1lIjoiTlVDTElPX1BZVEhPTl9FWEVfUEFUSCIsInZhbHVlIjoiL29wdC9udWNsaW8vY29tbW9uL29wZW52aW5vL3B5dGhvbjMifV0sInJlc291cmNlcyI6e30sImltYWdlIjoiY3ZhdC9vcGVudmluby5kZXh0cjpsYXRlc3QiLCJ0YXJnZXRDUFUiOjc1LCJ0cmlnZ2VycyI6eyJteUh0dHBUcmlnZ2VyIjp7ImNsYXNzIjoiIiwia2luZCI6Imh0dHAiLCJuYW1lIjoibXlIdHRwVHJpZ2dlciIsIm1heFdvcmtlcnMiOjIsIndvcmtlckF2YWlsYWJpbGl0eVRpbWVvdXRNaWxsaXNlY29uZHMiOjEwMDAwLCJhdHRyaWJ1dGVzIjp7Im1heFJlcXVlc3RCb2R5U2l6ZSI6MzM1NTQ0MzJ9fX0sInZvbHVtZXMiOlt7InZvbHVtZSI6eyJuYW1lIjoidm9sdW1lLTEiLCJob3N0UGF0aCI6eyJwYXRoIjoiL2RhdGEvY3ZhdC9zZXJ2ZXJsZXNzL2NvbW1vbiJ9fSwidm9sdW1lTW91bnQiOnsibmFtZSI6InZvbHVtZS0xIiwibW91bnRQYXRoIjoiL29wdC9udWNsaW8vY29tbW9uIn19XSwiYnVpbGQiOnsiaW1hZ2UiOiJjdmF0L29wZW52aW5vLmRleHRyIiwiYmFzZUltYWdlIjoib3BlbnZpbm8vdWJ1bnR1MThfcnVudGltZToyMDIwLjIiLCJkaXJlY3RpdmVzIjp7InBvc3RDb3B5IjpbeyJraW5kIjoiUlVOIiwidmFsdWUiOiJjdXJsIC1PIGh0dHBzOi8vZG93bmxvYWQuMDEub3JnL29wZW52aW5vdG9vbGtpdC9tb2RlbHNfY29udHJpYi9jdmF0L2RleHRyX21vZGVsX3YxLnppcCJ9LHsia2luZCI6IlJVTiIsInZhbHVlIjoidW56aXAgZGV4dHJfbW9kZWxfdjEuemlwIn0seyJraW5kIjoiUlVOIiwidmFsdWUiOiJwaXAzIGluc3RhbGwgUGlsbG93In1dLCJwcmVDb3B5IjpbeyJraW5kIjoiVVNFUiIsInZhbHVlIjoicm9vdCJ9LHsia2luZCI6IldPUktESVIiLCJ2YWx1ZSI6Ii9vcHQvbnVjbGlvIn0seyJraW5kIjoiUlVOIiwidmFsdWUiOiJsbiAtcyAvdXNyL2Jpbi9waXAzIC91c3IvYmluL3BpcCJ9XX0sImNvZGVFbnRyeVR5cGUiOiJpbWFnZSIsInRpbWVzdGFtcCI6MTYyMzA5MTUyMn0sInBsYXRmb3JtIjp7ImF0dHJpYnV0ZXMiOnsibW91bnRNb2RlIjoidm9sdW1lIiwicmVzdGFydFBvbGljeSI6eyJtYXhpbXVtUmV0cnlDb3VudCI6MywibmFtZSI6ImFsd2F5cyJ9fX0sInJlYWRpbmVzc1RpbWVvdXRTZWNvbmRzIjo2MCwic2VjdXJpdHlDb250ZXh0Ijp7fSwiZXZlbnRUaW1lb3V0IjoiMzBzIn0sInN0YXR1cyI6eyJzdGF0ZSI6InJlYWR5IiwiaHR0cFBvcnQiOjQ5MTUzfX0=\n", "stderr": "", "exitCode": 0}
21.06.09 12:33:43.271 �[0;37mdashboard.platform.docker�[0m �[0;32m(D)�[0m Getting containers {"options": {"Name":"nuclio-nuclio-openvino-dextr","Labels":null,"Stopped":false,"ID":""}}
21.06.09 12:33:43.271 �[0;37mrd.platform.docker.runner�[0m �[0;32m(D)�[0m Executing {"command": "docker ps --quiet   --filter \"name=^/nuclio-nuclio-openvino-dextr$\"  "}
21.06.09 12:33:43.298 �[0;37mrd.platform.docker.runner�[0m �[0;32m(D)�[0m Command executed successfully {"output": "296d755cc856\n", "stderr": "", "exitCode": 0}
21.06.09 12:33:43.299 �[0;37mrd.platform.docker.runner�[0m �[0;32m(D)�[0m Executing {"command": "docker inspect 296d755cc856 "}
21.06.09 12:33:43.324 �[0;37mrd.platform.docker.runner�[0m �[0;32m(D)�[0m Command executed successfully {"output": "[\n    {\n        \"Id\": \"296d755cc856341fd4d244df57b567967157ada0272100abe65bb18f91031f7b\",\n        \"Created\": \"2021-06-07T18:45:22.551691825Z\",\n        \"Path\": \"processor\",\n        \"Args\": [],\n        \"State\": {\n            \"Status\": \"running\",\n            \"Running\": true,\n            \"Paused\": false,\n            \"Restarting\": false,\n            \"OOMKilled\": false,\n            \"Dead\": false,\n            \"Pid\": 19159,\n            \"ExitCode\": 0,\n            \"Error\": \"\",\n            \"StartedAt\": \"2021-06-07T18:45:22.913839538Z\",\n            \"FinishedAt\": \"0001-01-01T00:00:00Z\",\n            \"Health\": {\n                \"Status\": \"healthy\",\n                \"FailingStreak\": 0,\n                \"Log\": [\n                    {\n                        \"Start\": \"2021-06-09T12:33:37.911508869Z\",\n                        \"End\": \"2021-06-09T12:33:37.986232806Z\",\n                        \"ExitCode\": 0,\n                        \"Output\": \"\"\n                    },\n                    {\n                        \"Start\": \"2021-06-09T12:33:38.997114607Z\",\n                        \"End\": \"2021-06-09T12:33:39.065291715Z\",\n                        \"ExitCode\": 0,\n                        \"Output\": \"\"\n                    },\n                    {\n                        \"Start\": \"2021-06-09T12:33:40.079181437Z\",\n                        \"End\": \"2021-06-09T12:33:40.160937513Z\",\n                        \"ExitCode\": 0,\n                        \"Output\": \"\"\n                    },\n                    {\n                        \"Start\": \"2021-06-09T12:33:41.173912775Z\",\n                        \"End\": \"2021-06-09T12:33:41.240469872Z\",\n                        \"ExitCode\": 0,\n                        \"Output\": \"\"\n                    },\n                    {\n                        \"Start\": \"2021-06-09T12:33:42.252176574Z\",\n                        \"End\": \"2021-06-09T12:33:42.323030651Z\",\n                        \"ExitCode\": 0,\n                        \"Output\": \"\"\n                    }\n                ]\n            }\n        },\n        \"Image\": \"sha256:78c90f6198911eab39ee9eea164e651d12fed02264b3b0da53700f37de13f2f7\",\n        \"ResolvConfPath\": \"/var/lib/docker/containers/296d755cc856341fd4d244df57b567967157ada0272100abe65bb18f91031f7b/resolv.conf\",\n        \"HostnamePath\": \"/var/lib/docker/containers/296d755cc856341fd4d244df57b567967157ada0272100abe65bb18f91031f7b/hostname\",\n        \"HostsPath\": \"/var/lib/docker/containers/296d755cc856341fd4d244df57b567967157ada0272100abe65bb18f91031f7b/hosts\",\n        \"LogPath\": \"/var/lib/docker/containers/296d755cc856341fd4d244df57b567967157ada0272100abe65bb18f91031f7b/296d755cc856341fd4d244df57b567967157ada0272100abe65bb18f91031f7b-json.log\",\n        \"Name\": \"/nuclio-nuclio-openvino-dextr\",\n        \"RestartCount\": 0,\n        \"Driver\": \"overlay2\",\n        \"Platform\": \"linux\",\n        \"MountLabel\": \"\",\n        \"ProcessLabel\": \"\",\n        \"AppArmorProfile\": \"docker-default\",\n        \"ExecIDs\": null,\n        \"HostConfig\": {\n            \"Binds\": [\n                \"/data/cvat/serverless/common:/opt/nuclio/common\"\n            ],\n            \"ContainerIDFile\": \"\",\n            \"LogConfig\": {\n                \"Type\": \"json-file\",\n                \"Config\": {}\n            },\n            \"NetworkMode\": \"default\",\n            \"PortBindings\": {\n                \"8080/tcp\": [\n                    {\n                        \"HostIp\": \"\",\n                        \"HostPort\": \"\"\n                    }\n                ]\n            },\n            \"RestartPolicy\": {\n                \"Name\": \"always\",\n                \"MaximumRetryCount\": 0\n            },\n            \"AutoRemove\": false,\n            \"VolumeDriver\": \"\",\n            \"VolumesFrom\": null,\n            \"CapAdd\": null,\n            \"CapDrop\": null,\n            \"CgroupnsMode\": \"host\",\n            \"Dns\": [],\n            \"DnsOptions\": [],\n            \"DnsSearch\": [],\n            \"ExtraHosts\": null,\n            \"GroupAdd\": null,\n            \"IpcMode\": \"private\",\n            \"Cgroup\": \"\",\n            \"Links\": null,\n            \"OomScoreAdj\": 0,\n            \"PidMode\": \"\",\n            \"Privileged\": false,\n            \"PublishAllPorts\": false,\n            \"ReadonlyRootfs\": false,\n            \"SecurityOpt\": null,\n            \"UTSMode\": \"\",\n            \"UsernsMode\": \"\",\n            \"ShmSize\": 67108864,\n            \"Runtime\": \"runc\",\n            \"ConsoleSize\": [\n                0,\n                0\n            ],\n            \"Isolation\": \"\",\n            \"CpuShares\": 0,\n            \"Memory\": 0,\n            \"NanoCpus\": 0,\n            \"CgroupParent\": \"\",\n            \"BlkioWeight\": 0,\n            \"BlkioWeightDevice\": [],\n            \"BlkioDeviceReadBps\": null,\n            \"BlkioDeviceWriteBps\": null,\n            \"BlkioDeviceReadIOps\": null,\n            \"BlkioDeviceWriteIOps\": null,\n            \"CpuPeriod\": 0,\n            \"CpuQuota\": 0,\n            \"CpuRealtimePeriod\": 0,\n            \"CpuRealtimeRuntime\": 0,\n            \"CpusetCpus\": \"\",\n            \"CpusetMems\": \"\",\n            \"Devices\": [],\n            \"DeviceCgroupRules\": null,\n            \"DeviceRequests\": null,\n            \"KernelMemory\": 0,\n            \"KernelMemoryTCP\": 0,\n            \"MemoryReservation\": 0,\n            \"MemorySwap\": 0,\n            \"MemorySwappiness\": null,\n            \"OomKillDisable\": false,\n            \"PidsLimit\": null,\n            \"Ulimits\": null,\n            \"CpuCount\": 0,\n            \"CpuPercent\": 0,\n            \"IOMaximumIOps\": 0,\n            \"IOMaximumBandwidth\": 0,\n            \"Mounts\": [\n                {\n                    \"Type\": \"volume\",\n                    \"Source\": \"nuclio-nuclio-openvino-dextr\",\n                    \"Target\": \"/etc/nuclio/config/processor\",\n                    \"ReadOnly\": true\n                }\n            ],\n            \"MaskedPaths\": [\n                \"/proc/asound\",\n                \"/proc/acpi\",\n                \"/proc/kcore\",\n                \"/proc/keys\",\n                \"/proc/latency_stats\",\n                \"/proc/timer_list\",\n                \"/proc/timer_stats\",\n                \"/proc/sched_debug\",\n                \"/proc/scsi\",\n                \"/sys/firmware\"\n            ],\n            \"ReadonlyPaths\": [\n                \"/proc/bus\",\n                \"/proc/fs\",\n                \"/proc/irq\",\n                \"/proc/sys\",\n                \"/proc/sysrq-trigger\"\n            ]\n        },\n        \"GraphDriver\": {\n            \"Data\": {\n                \"LowerDir\": \"/var/lib/docker/overlay2/0fea8d13659259cdef53e30ba78a34b900aafe7ae15e83ac3ca054323bcf108b-init/diff:/var/lib/docker/overlay2/ef4a61cba20b238f8669d0a65469d0fbc30f23798f5ffa6dacc97631f1c3ab68/diff:/var/lib/docker/overlay2/7a9ede8471fc911f101315146d0be222c0183d844c08782ed72124e20b11a5e5/diff:/var/lib/docker/overlay2/c2d565d6196e2b9ab7756a1e758df0288270697c38d22bd20a96f2655d02deb2/diff:/var/lib/docker/overlay2/109f62d05963b11daa76bee1b253767a00033fe6165eb7639973ad6b5421a485/diff:/var/lib/docker/overlay2/1d373a4c3185437e3a907cd8fddbb2c9bf004c85a23089c2f2b358aad671f19a/diff:/var/lib/docker/overlay2/67b5e5e9ba2b053563ff7e7c4239fef80c7fb1591305da8079e26b8ed3fa9653/diff:/var/lib/docker/overlay2/190a89586fdca85c11a6fa888ce9ad3cb0989c73e6ae7f1c5754808b048ad326/diff:/var/lib/docker/overlay2/332cca9cde76ee0edf836e1ac0f786792f8bd79fa2537b824c3e04f1a0fbcccc/diff:/var/lib/docker/overlay2/c372dd7825885913f77d41259ed1d0d40720ad0699551afa04b69832cbba8cf2/diff:/var/lib/docker/overlay2/22110ca9967c02427818eaba056e1c8f6e25fd8e89a844a729197b369d926c67/diff:/var/lib/docker/overlay2/59a08bdea0770743e07771a19f5cc473d4eb64c7018bbb202a4f8deff31b0b76/diff:/var/lib/docker/overlay2/cd648b2f736e449672d75cd1212e8498eb45949e97bda2afb41ecda6c05df35b/diff:/var/lib/docker/overlay2/866973a30f3947709035d1dfa08ababfc13bbd310547a23da3f7eefe222ee32d/diff:/var/lib/docker/overlay2/d3dc8d69daa788b01da4cb89bc97cba433a79cb224eaaccb79dd437bc8e2e6bf/diff:/var/lib/docker/overlay2/ad196c192b809fc5c0cbf0ae433699d2d96231ab8636543c8104bdf3a839be41/diff:/var/lib/docker/overlay2/828cbeaae67416feede7e93defff45674dcc27a5f86d37bd28e457ef77a7d326/diff:/var/lib/docker/overlay2/67a5ea09a8e27f3b240d6251d58004fc2979b2e55fe8cb086044f9e8b8c3a643/diff:/var/lib/docker/overlay2/05c452421da3d4627bc515c888fe73e4ef8b3ed0a8db45438d5ce4ef88891a51/diff:/var/lib/docker/overlay2/ce7e973839c1995d4159485926d846c8eb84e78c90b2ffbab611796ad2f81a30/diff:/var/lib/docker/overlay2/3d4c90717a68d37617a3faa84916360e651e740bbd71fbcd514abd3b72a093cf/diff:/var/lib/docker/overlay2/7e52a86bc7e4f12e877a29a8f4f3f41d987d83b28d8106dab76ef108701d93a3/diff:/var/lib/docker/overlay2/1b59a6f044884674b1da6cc56629bb57b5fbf114d71cadd89eff4195fd8b8cc6/diff:/var/lib/docker/overlay2/926f4ad2214d3935f6081bf705834c53f5a4c02a243a0f2b3d841270b5c1b26b/diff:/var/lib/docker/overlay2/c17f7b54fbd172c384d02e198c4ef84a429650283f1ebed872da5a3d8841ba87/diff:/var/lib/docker/overlay2/c106e17795a5e6cc273554be97789e5d7eb679128ccd1d9b8438116709fecbff/diff:/var/lib/docker/overlay2/147eddd7bd977779c8612755d20c8a94387c3bd37a30ad40b5bd702ab05d2baa/diff:/var/lib/docker/overlay2/19523c5c558c5423b2533046dd44824a9286ebb73d1f3fa2a08534933bd2617b/diff:/var/lib/docker/overlay2/404ca9ad9993ddddf95ee1f456690bd9a81e2b3477735fc682accd797a72915b/diff:/var/lib/docker/overlay2/8a583ef75dc7e161de3a1c64fcf383b595b2f77d707d059fd9a6c252b9cf4e92/diff:/var/lib/docker/overlay2/f424db246a5b7825affa1af8770dcfea3c676a10416311dc471002ed36e35746/diff:/var/lib/docker/overlay2/2a92426b9c10542fe4650cc2bf3a27c025974b7ef00cfbcd62ca47916cd5e4b1/diff:/var/lib/docker/overlay2/b296d9e3e1e31f84e069a90245d15134717ff1edf0b8b07a60ce348196c406e9/diff:/var/lib/docker/overlay2/d5fd2c628edb14fb1a47bc701e4598a380b1045a73c0d4d40761e9f905c17bc3/diff:/var/lib/docker/overlay2/f794f88003624ccf85b610deb02278600e15520ad6b70f111c737a0650f906ea/diff\",\n                \"MergedDir\": \"/var/lib/docker/overlay2/0fea8d13659259cdef53e30ba78a34b900aafe7ae15e83ac3ca054323bcf108b/merged\",\n                \"UpperDir\": \"/var/lib/docker/overlay2/0fea8d13659259cdef53e30ba78a34b900aafe7ae15e83ac3ca054323bcf108b/diff\",\n                \"WorkDir\": \"/var/lib/docker/overlay2/0fea8d13659259cdef53e30ba78a34b900aafe7ae15e83ac3ca054323bcf108b/work\"\n            },\n            \"Name\": \"overlay2\"\n        },\n        \"Mounts\": [\n            {\n                \"Type\": \"bind\",\n                \"Source\": \"/data/cvat/serverless/common\",\n                \"Destination\": \"/opt/nuclio/common\",\n                \"Mode\": \"\",\n                \"RW\": true,\n                \"Propagation\": \"rprivate\"\n            },\n            {\n                \"Type\": \"volume\",\n                \"Name\": \"nuclio-nuclio-openvino-dextr\",\n                \"Source\": \"/var/lib/docker/volumes/nuclio-nuclio-openvino-dextr/_data\",\n                \"Destination\": \"/etc/nuclio/config/processor\",\n                \"Driver\": \"local\",\n                \"Mode\": \"z\",\n                \"RW\": false,\n                \"Propagation\": \"\"\n            }\n        ],\n        \"Config\": {\n            \"Hostname\": \"296d755cc856\",\n            \"Domainname\": \"\",\n            \"User\": \"root\",\n            \"AttachStdin\": false,\n            \"AttachStdout\": false,\n            \"AttachStderr\": false,\n            \"ExposedPorts\": {\n                \"8080/tcp\": {}\n            },\n            \"Tty\": false,\n            \"OpenStdin\": false,\n            \"StdinOnce\": false,\n            \"Env\": [\n                \"http_proxy=http://my-proxy:8080\",\n                \"HTTPS_PROXY=http://my-proxy:8080\",\n                \"https_proxy=http://my-proxy:8080\",\n                \"NO_PROXY=*.247.ai,localhost,127.0.0.0/8\",\n                \"no_proxy=*.247.ai,localhost,127.0.0.0/8\",\n                \"NUCLIO_PYTHON_EXE_PATH=/opt/nuclio/common/openvino/python3\",\n                \"HTTP_PROXY=http://my-proxy:8080\",\n                \"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\n                \"PYTHON=python3.6\",\n                \"INTEL_OPENVINO_DIR=/opt/intel/openvino\"\n            ],\n            \"Cmd\": [\n                \"processor\"\n            ],\n            \"Healthcheck\": {\n                \"Test\": [\n                    \"CMD-SHELL\",\n                    \"/usr/local/bin/uhttpc --url http://127.0.0.1:8082/ready || exit 1\"\n                ],\n                \"Interval\": 1000000000,\n                \"Timeout\": 3000000000\n            },\n            \"Image\": \"cvat/openvino.dextr:latest\",\n            \"Volumes\": null,\n            \"WorkingDir\": \"/opt/nuclio\",\n            \"Entrypoint\": null,\n            \"OnBuild\": null,\n            \"Labels\": {\n                \"Description\": \"This is the runtime image for Intel(R) Distribution of OpenVINO(TM) toolkit on Ubuntu 18.04 LTS\",\n                \"Vendor\": \"Intel Corporation\",\n                \"nuclio.io/annotations\": \"{\\\"framework\\\":\\\"openvino\\\",\\\"min_pos_points\\\":\\\"4\\\",\\\"name\\\":\\\"DEXTR\\\",\\\"spec\\\":\\\"\\\",\\\"type\\\":\\\"interactor\\\"}\",\n                \"nuclio.io/function-name\": \"openvino-dextr\",\n                \"nuclio.io/function-spec\": \"{\\\"description\\\":\\\"Deep Extreme Cut\\\",\\\"handler\\\":\\\"main:handler\\\",\\\"runtime\\\":\\\"python:3.6\\\",\\\"env\\\":[{\\\"name\\\":\\\"NUCLIO_PYTHON_EXE_PATH\\\",\\\"value\\\":\\\"/opt/nuclio/common/openvino/python3\\\"}],\\\"resources\\\":{},\\\"image\\\":\\\"cvat/openvino.dextr:latest\\\",\\\"targetCPU\\\":75,\\\"triggers\\\":{\\\"myHttpTrigger\\\":{\\\"class\\\":\\\"\\\",\\\"kind\\\":\\\"http\\\",\\\"name\\\":\\\"myHttpTrigger\\\",\\\"maxWorkers\\\":2,\\\"workerAvailabilityTimeoutMilliseconds\\\":10000,\\\"attributes\\\":{\\\"maxRequestBodySize\\\":33554432}}},\\\"volumes\\\":[{\\\"volume\\\":{\\\"name\\\":\\\"volume-1\\\",\\\"hostPath\\\":{\\\"path\\\":\\\"/data/cvat/serverless/common\\\"}},\\\"volumeMount\\\":{\\\"name\\\":\\\"volume-1\\\",\\\"mountPath\\\":\\\"/opt/nuclio/common\\\"}}],\\\"build\\\":{\\\"image\\\":\\\"cvat/openvino.dextr\\\",\\\"baseImage\\\":\\\"openvino/ubuntu18_runtime:2020.2\\\",\\\"directives\\\":{\\\"postCopy\\\":[{\\\"kind\\\":\\\"RUN\\\",\\\"value\\\":\\\"curl -O https://download.01.org/openvinotoolkit/models_contrib/cvat/dextr_model_v1.zip\\\"},{\\\"kind\\\":\\\"RUN\\\",\\\"value\\\":\\\"unzip dextr_model_v1.zip\\\"},{\\\"kind\\\":\\\"RUN\\\",\\\"value\\\":\\\"pip3 install Pillow\\\"}],\\\"preCopy\\\":[{\\\"kind\\\":\\\"USER\\\",\\\"value\\\":\\\"root\\\"},{\\\"kind\\\":\\\"WORKDIR\\\",\\\"value\\\":\\\"/opt/nuclio\\\"},{\\\"kind\\\":\\\"RUN\\\",\\\"value\\\":\\\"ln -s /usr/bin/pip3 /usr/bin/pip\\\"}]},\\\"codeEntryType\\\":\\\"image\\\",\\\"timestamp\\\":1623091522},\\\"platform\\\":{\\\"attributes\\\":{\\\"mountMode\\\":\\\"volume\\\",\\\"restartPolicy\\\":{\\\"maximumRetryCount\\\":3,\\\"name\\\":\\\"always\\\"}}},\\\"readinessTimeoutSeconds\\\":60,\\\"securityContext\\\":{},\\\"eventTimeout\\\":\\\"30s\\\"}\",\n                \"nuclio.io/namespace\": \"nuclio\",\n                \"nuclio.io/platform\": \"local\",\n                \"nuclio.io/project-name\": \"cvat\"\n            }\n        },\n        \"NetworkSettings\": {\n            \"Bridge\": \"\",\n            \"SandboxID\": \"f2a29770e04c2ccbc6cf554bef44310c137cc32f7ef9ed7b04ca64be583725f2\",\n            \"HairpinMode\": false,\n            \"LinkLocalIPv6Address\": \"\",\n            \"LinkLocalIPv6PrefixLen\": 0,\n            \"Ports\": {\n                \"8080/tcp\": [\n                    {\n                        \"HostIp\": \"0.0.0.0\",\n                        \"HostPort\": \"49153\"\n                    },\n                    {\n                        \"HostIp\": \"::\",\n                        \"HostPort\": \"49153\"\n                    }\n                ]\n            },\n            \"SandboxKey\": \"/var/run/docker/netns/f2a29770e04c\",\n            \"SecondaryIPAddresses\": null,\n            \"SecondaryIPv6Addresses\": null,\n            \"EndpointID\": \"331557200fb48b5e4929c478ddb14605a22273fb21478198812982543d7edb5e\",\n            \"Gateway\": \"172.17.0.1\",\n            \"GlobalIPv6Address\": \"\",\n            \"GlobalIPv6PrefixLen\": 0,\n            \"IPAddress\": \"172.17.0.4\",\n            \"IPPrefixLen\": 16,\n            \"IPv6Gateway\": \"\",\n            \"MacAddress\": \"02:42:ac:11:00:04\",\n            \"Networks\": {\n                \"bridge\": {\n                    \"IPAMConfig\": null,\n                    \"Links\": null,\n                    \"Aliases\": null,\n                    \"NetworkID\": \"91a66fa558cbc58ff33e457a1412e09a2dc68842b6e7758e4f1c91f6d3a5dc95\",\n                    \"EndpointID\": \"331557200fb48b5e4929c478ddb14605a22273fb21478198812982543d7edb5e\",\n                    \"Gateway\": \"172.17.0.1\",\n                    \"IPAddress\": \"172.17.0.4\",\n                    \"IPPrefixLen\": 16,\n                    \"IPv6Gateway\": \"\",\n                    \"GlobalIPv6Address\": \"\",\n                    \"GlobalIPv6PrefixLen\": 0,\n                    \"MacAddress\": \"02:42:ac:11:00:04\",\n                    \"DriverOpts\": null\n                }\n            }\n        }\n    }\n]\n", "stderr": "", "exitCode": 0}
21.06.09 12:33:43.325 �[0;37mdashboard.platform.docker�[0m �[0;32m(D)�[0m Awaiting container health {"containerID": "296d755cc856341fd4d244df57b567967157ada0272100abe65bb18f91031f7b", "timeout": "5s"}
21.06.09 12:33:43.325 �[0;37mdashboard.platform.docker�[0m �[0;32m(D)�[0m Getting containers {"options": {"Name":"","Labels":null,"Stopped":true,"ID":"296d755cc856341fd4d244df57b567967157ada0272100abe65bb18f91031f7b"}}
21.06.09 12:33:43.325 �[0;37mrd.platform.docker.runner�[0m �[0;32m(D)�[0m Executing {"command": "docker ps --quiet --all  --filter \"id=296d755cc856341fd4d244df57b567967157ada0272100abe65bb18f91031f7b\"  "}
21.06.09 12:33:43.374 �[0;37mrd.platform.docker.runner�[0m �[0;32m(D)�[0m Command executed successfully {"output": "296d755cc856\n", "stderr": "", "exitCode": 0}
21.06.09 12:33:43.374 �[0;37mrd.platform.docker.runner�[0m �[0;32m(D)�[0m Executing {"command": "docker inspect 296d755cc856 "}
21.06.09 12:33:43.410 �[0;37mrd.platform.docker.runner�[0m �[0;32m(D)�[0m Command executed successfully {"output": "[\n    {\n        \"Id\": \"296d755cc856341fd4d244df57b567967157ada0272100abe65bb18f91031f7b\",\n        \"Created\": \"2021-06-07T18:45:22.551691825Z\",\n        \"Path\": \"processor\",\n        \"Args\": [],\n        \"State\": {\n            \"Status\": \"running\",\n            \"Running\": true,\n            \"Paused\": false,\n            \"Restarting\": false,\n            \"OOMKilled\": false,\n            \"Dead\": false,\n            \"Pid\": 19159,\n            \"ExitCode\": 0,\n            \"Error\": \"\",\n            \"StartedAt\": \"2021-06-07T18:45:22.913839538Z\",\n            \"FinishedAt\": \"0001-01-01T00:00:00Z\",\n            \"Health\": {\n                \"Status\": \"healthy\",\n                \"FailingStreak\": 0,\n                \"Log\": [\n                    {\n                        \"Start\": \"2021-06-09T12:33:37.911508869Z\",\n                        \"End\": \"2021-06-09T12:33:37.986232806Z\",\n                        \"ExitCode\": 0,\n                        \"Output\": \"\"\n                    },\n                    {\n                        \"Start\": \"2021-06-09T12:33:38.997114607Z\",\n                        \"End\": \"2021-06-09T12:33:39.065291715Z\",\n                        \"ExitCode\": 0,\n                        \"Output\": \"\"\n                    },\n                    {\n                        \"Start\": \"2021-06-09T12:33:40.079181437Z\",\n                        \"End\": \"2021-06-09T12:33:40.160937513Z\",\n                        \"ExitCode\": 0,\n                        \"Output\": \"\"\n                    },\n                    {\n                        \"Start\": \"2021-06-09T12:33:41.173912775Z\",\n                        \"End\": \"2021-06-09T12:33:41.240469872Z\",\n                        \"ExitCode\": 0,\n                        \"Output\": \"\"\n                    },\n                    {\n                        \"Start\": \"2021-06-09T12:33:42.252176574Z\",\n                        \"End\": \"2021-06-09T12:33:42.323030651Z\",\n                        \"ExitCode\": 0,\n                        \"Output\": \"\"\n                    }\n                ]\n            }\n        },\n        \"Image\": \"sha256:78c90f6198911eab39ee9eea164e651d12fed02264b3b0da53700f37de13f2f7\",\n        \"ResolvConfPath\": \"/var/lib/docker/containers/296d755cc856341fd4d244df57b567967157ada0272100abe65bb18f91031f7b/resolv.conf\",\n        \"HostnamePath\": \"/var/lib/docker/containers/296d755cc856341fd4d244df57b567967157ada0272100abe65bb18f91031f7b/hostname\",\n        \"HostsPath\": \"/var/lib/docker/containers/296d755cc856341fd4d244df57b567967157ada0272100abe65bb18f91031f7b/hosts\",\n        \"LogPath\": \"/var/lib/docker/containers/296d755cc856341fd4d244df57b567967157ada0272100abe65bb18f91031f7b/296d755cc856341fd4d244df57b567967157ada0272100abe65bb18f91031f7b-json.log\",\n        \"Name\": \"/nuclio-nuclio-openvino-dextr\",\n        \"RestartCount\": 0,\n        \"Driver\": \"overlay2\",\n        \"Platform\": \"linux\",\n        \"MountLabel\": \"\",\n        \"ProcessLabel\": \"\",\n        \"AppArmorProfile\": \"docker-default\",\n        \"ExecIDs\": [\n            \"f6dd309c2bb40b61ea788aefd69d9f4b9d4ea16c73274127864174bfc1f1aed4\"\n        ],\n        \"HostConfig\": {\n            \"Binds\": [\n                \"/data/cvat/serverless/common:/opt/nuclio/common\"\n            ],\n            \"ContainerIDFile\": \"\",\n            \"LogConfig\": {\n                \"Type\": \"json-file\",\n                \"Config\": {}\n            },\n            \"NetworkMode\": \"default\",\n            \"PortBindings\": {\n                \"8080/tcp\": [\n                    {\n                        \"HostIp\": \"\",\n                        \"HostPort\": \"\"\n                    }\n                ]\n            },\n            \"RestartPolicy\": {\n                \"Name\": \"always\",\n                \"MaximumRetryCount\": 0\n            },\n            \"AutoRemove\": false,\n            \"VolumeDriver\": \"\",\n            \"VolumesFrom\": null,\n            \"CapAdd\": null,\n            \"CapDrop\": null,\n            \"CgroupnsMode\": \"host\",\n            \"Dns\": [],\n            \"DnsOptions\": [],\n            \"DnsSearch\": [],\n            \"ExtraHosts\": null,\n            \"GroupAdd\": null,\n            \"IpcMode\": \"private\",\n            \"Cgroup\": \"\",\n            \"Links\": null,\n            \"OomScoreAdj\": 0,\n            \"PidMode\": \"\",\n            \"Privileged\": false,\n            \"PublishAllPorts\": false,\n            \"ReadonlyRootfs\": false,\n            \"SecurityOpt\": null,\n            \"UTSMode\": \"\",\n            \"UsernsMode\": \"\",\n            \"ShmSize\": 67108864,\n            \"Runtime\": \"runc\",\n            \"ConsoleSize\": [\n                0,\n                0\n            ],\n            \"Isolation\": \"\",\n            \"CpuShares\": 0,\n            \"Memory\": 0,\n            \"NanoCpus\": 0,\n            \"CgroupParent\": \"\",\n            \"BlkioWeight\": 0,\n            \"BlkioWeightDevice\": [],\n            \"BlkioDeviceReadBps\": null,\n            \"BlkioDeviceWriteBps\": null,\n            \"BlkioDeviceReadIOps\": null,\n            \"BlkioDeviceWriteIOps\": null,\n            \"CpuPeriod\": 0,\n            \"CpuQuota\": 0,\n            \"CpuRealtimePeriod\": 0,\n            \"CpuRealtimeRuntime\": 0,\n            \"CpusetCpus\": \"\",\n            \"CpusetMems\": \"\",\n            \"Devices\": [],\n            \"DeviceCgroupRules\": null,\n            \"DeviceRequests\": null,\n            \"KernelMemory\": 0,\n            \"KernelMemoryTCP\": 0,\n            \"MemoryReservation\": 0,\n            \"MemorySwap\": 0,\n            \"MemorySwappiness\": null,\n            \"OomKillDisable\": false,\n            \"PidsLimit\": null,\n            \"Ulimits\": null,\n            \"CpuCount\": 0,\n            \"CpuPercent\": 0,\n            \"IOMaximumIOps\": 0,\n            \"IOMaximumBandwidth\": 0,\n            \"Mounts\": [\n                {\n                    \"Type\": \"volume\",\n                    \"Source\": \"nuclio-nuclio-openvino-dextr\",\n                    \"Target\": \"/etc/nuclio/config/processor\",\n                    \"ReadOnly\": true\n                }\n            ],\n            \"MaskedPaths\": [\n                \"/proc/asound\",\n                \"/proc/acpi\",\n                \"/proc/kcore\",\n                \"/proc/keys\",\n                \"/proc/latency_stats\",\n                \"/proc/timer_list\",\n                \"/proc/timer_stats\",\n                \"/proc/sched_debug\",\n                \"/proc/scsi\",\n                \"/sys/firmware\"\n            ],\n            \"ReadonlyPaths\": [\n                \"/proc/bus\",\n                \"/proc/fs\",\n                \"/proc/irq\",\n                \"/proc/sys\",\n                \"/proc/sysrq-trigger\"\n            ]\n        },\n        \"GraphDriver\": {\n            \"Data\": {\n                \"LowerDir\": \"/var/lib/docker/overlay2/0fea8d13659259cdef53e30ba78a34b900aafe7ae15e83ac3ca054323bcf108b-init/diff:/var/lib/docker/overlay2/ef4a61cba20b238f8669d0a65469d0fbc30f23798f5ffa6dacc97631f1c3ab68/diff:/var/lib/docker/overlay2/7a9ede8471fc911f101315146d0be222c0183d844c08782ed72124e20b11a5e5/diff:/var/lib/docker/overlay2/c2d565d6196e2b9ab7756a1e758df0288270697c38d22bd20a96f2655d02deb2/diff:/var/lib/docker/overlay2/109f62d05963b11daa76bee1b253767a00033fe6165eb7639973ad6b5421a485/diff:/var/lib/docker/overlay2/1d373a4c3185437e3a907cd8fddbb2c9bf004c85a23089c2f2b358aad671f19a/diff:/var/lib/docker/overlay2/67b5e5e9ba2b053563ff7e7c4239fef80c7fb1591305da8079e26b8ed3fa9653/diff:/var/lib/docker/overlay2/190a89586fdca85c11a6fa888ce9ad3cb0989c73e6ae7f1c5754808b048ad326/diff:/var/lib/docker/overlay2/332cca9cde76ee0edf836e1ac0f786792f8bd79fa2537b824c3e04f1a0fbcccc/diff:/var/lib/docker/overlay2/c372dd7825885913f77d41259ed1d0d40720ad0699551afa04b69832cbba8cf2/diff:/var/lib/docker/overlay2/22110ca9967c02427818eaba056e1c8f6e25fd8e89a844a729197b369d926c67/diff:/var/lib/docker/overlay2/59a08bdea0770743e07771a19f5cc473d4eb64c7018bbb202a4f8deff31b0b76/diff:/var/lib/docker/overlay2/cd648b2f736e449672d75cd1212e8498eb45949e97bda2afb41ecda6c05df35b/diff:/var/lib/docker/overlay2/866973a30f3947709035d1dfa08ababfc13bbd310547a23da3f7eefe222ee32d/diff:/var/lib/docker/overlay2/d3dc8d69daa788b01da4cb89bc97cba433a79cb224eaaccb79dd437bc8e2e6bf/diff:/var/lib/docker/overlay2/ad196c192b809fc5c0cbf0ae433699d2d96231ab8636543c8104bdf3a839be41/diff:/var/lib/docker/overlay2/828cbeaae67416feede7e93defff45674dcc27a5f86d37bd28e457ef77a7d326/diff:/var/lib/docker/overlay2/67a5ea09a8e27f3b240d6251d58004fc2979b2e55fe8cb086044f9e8b8c3a643/diff:/var/lib/docker/overlay2/05c452421da3d4627bc515c888fe73e4ef8b3ed0a8db45438d5ce4ef88891a51/diff:/var/lib/docker/overlay2/ce7e973839c1995d4159485926d846c8eb84e78c90b2ffbab611796ad2f81a30/diff:/var/lib/docker/overlay2/3d4c90717a68d37617a3faa84916360e651e740bbd71fbcd514abd3b72a093cf/diff:/var/lib/docker/overlay2/7e52a86bc7e4f12e877a29a8f4f3f41d987d83b28d8106dab76ef108701d93a3/diff:/var/lib/docker/overlay2/1b59a6f044884674b1da6cc56629bb57b5fbf114d71cadd89eff4195fd8b8cc6/diff:/var/lib/docker/overlay2/926f4ad2214d3935f6081bf705834c53f5a4c02a243a0f2b3d841270b5c1b26b/diff:/var/lib/docker/overlay2/c17f7b54fbd172c384d02e198c4ef84a429650283f1ebed872da5a3d8841ba87/diff:/var/lib/docker/overlay2/c106e17795a5e6cc273554be97789e5d7eb679128ccd1d9b8438116709fecbff/diff:/var/lib/docker/overlay2/147eddd7bd977779c8612755d20c8a94387c3bd37a30ad40b5bd702ab05d2baa/diff:/var/lib/docker/overlay2/19523c5c558c5423b2533046dd44824a9286ebb73d1f3fa2a08534933bd2617b/diff:/var/lib/docker/overlay2/404ca9ad9993ddddf95ee1f456690bd9a81e2b3477735fc682accd797a72915b/diff:/var/lib/docker/overlay2/8a583ef75dc7e161de3a1c64fcf383b595b2f77d707d059fd9a6c252b9cf4e92/diff:/var/lib/docker/overlay2/f424db246a5b7825affa1af8770dcfea3c676a10416311dc471002ed36e35746/diff:/var/lib/docker/overlay2/2a92426b9c10542fe4650cc2bf3a27c025974b7ef00cfbcd62ca47916cd5e4b1/diff:/var/lib/docker/overlay2/b296d9e3e1e31f84e069a90245d15134717ff1edf0b8b07a60ce348196c406e9/diff:/var/lib/docker/overlay2/d5fd2c628edb14fb1a47bc701e4598a380b1045a73c0d4d40761e9f905c17bc3/diff:/var/lib/docker/overlay2/f794f88003624ccf85b610deb02278600e15520ad6b70f111c737a0650f906ea/diff\",\n                \"MergedDir\": \"/var/lib/docker/overlay2/0fea8d13659259cdef53e30ba78a34b900aafe7ae15e83ac3ca054323bcf108b/merged\",\n                \"UpperDir\": \"/var/lib/docker/overlay2/0fea8d13659259cdef53e30ba78a34b900aafe7ae15e83ac3ca054323bcf108b/diff\",\n                \"WorkDir\": \"/var/lib/docker/overlay2/0fea8d13659259cdef53e30ba78a34b900aafe7ae15e83ac3ca054323bcf108b/work\"\n            },\n            \"Name\": \"overlay2\"\n        },\n        \"Mounts\": [\n            {\n                \"Type\": \"bind\",\n                \"Source\": \"/data/cvat/serverless/common\",\n                \"Destination\": \"/opt/nuclio/common\",\n                \"Mode\": \"\",\n                \"RW\": true,\n                \"Propagation\": \"rprivate\"\n            },\n            {\n                \"Type\": \"volume\",\n                \"Name\": \"nuclio-nuclio-openvino-dextr\",\n                \"Source\": \"/var/lib/docker/volumes/nuclio-nuclio-openvino-dextr/_data\",\n                \"Destination\": \"/etc/nuclio/config/processor\",\n                \"Driver\": \"local\",\n                \"Mode\": \"z\",\n                \"RW\": false,\n                \"Propagation\": \"\"\n            }\n        ],\n        \"Config\": {\n            \"Hostname\": \"296d755cc856\",\n            \"Domainname\": \"\",\n            \"User\": \"root\",\n            \"AttachStdin\": false,\n            \"AttachStdout\": false,\n            \"AttachStderr\": false,\n            \"ExposedPorts\": {\n                \"8080/tcp\": {}\n            },\n            \"Tty\": false,\n            \"OpenStdin\": false,\n            \"StdinOnce\": false,\n            \"Env\": [\n                \"http_proxy=http://my-proxy:8080\",\n                \"HTTPS_PROXY=http://my-proxy:8080\",\n                \"https_proxy=http://my-proxy:8080\",\n                \"NO_PROXY=*.247.ai,localhost,127.0.0.0/8\",\n                \"no_proxy=*.247.ai,localhost,127.0.0.0/8\",\n                \"NUCLIO_PYTHON_EXE_PATH=/opt/nuclio/common/openvino/python3\",\n                \"HTTP_PROXY=http://my-proxy:8080\",\n                \"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\n                \"PYTHON=python3.6\",\n                \"INTEL_OPENVINO_DIR=/opt/intel/openvino\"\n            ],\n            \"Cmd\": [\n                \"processor\"\n            ],\n            \"Healthcheck\": {\n                \"Test\": [\n                    \"CMD-SHELL\",\n                    \"/usr/local/bin/uhttpc --url http://127.0.0.1:8082/ready || exit 1\"\n                ],\n                \"Interval\": 1000000000,\n                \"Timeout\": 3000000000\n            },\n            \"Image\": \"cvat/openvino.dextr:latest\",\n            \"Volumes\": null,\n            \"WorkingDir\": \"/opt/nuclio\",\n            \"Entrypoint\": null,\n            \"OnBuild\": null,\n            \"Labels\": {\n                \"Description\": \"This is the runtime image for Intel(R) Distribution of OpenVINO(TM) toolkit on Ubuntu 18.04 LTS\",\n                \"Vendor\": \"Intel Corporation\",\n                \"nuclio.io/annotations\": \"{\\\"framework\\\":\\\"openvino\\\",\\\"min_pos_points\\\":\\\"4\\\",\\\"name\\\":\\\"DEXTR\\\",\\\"spec\\\":\\\"\\\",\\\"type\\\":\\\"interactor\\\"}\",\n                \"nuclio.io/function-name\": \"openvino-dextr\",\n                \"nuclio.io/function-spec\": \"{\\\"description\\\":\\\"Deep Extreme Cut\\\",\\\"handler\\\":\\\"main:handler\\\",\\\"runtime\\\":\\\"python:3.6\\\",\\\"env\\\":[{\\\"name\\\":\\\"NUCLIO_PYTHON_EXE_PATH\\\",\\\"value\\\":\\\"/opt/nuclio/common/openvino/python3\\\"}],\\\"resources\\\":{},\\\"image\\\":\\\"cvat/openvino.dextr:latest\\\",\\\"targetCPU\\\":75,\\\"triggers\\\":{\\\"myHttpTrigger\\\":{\\\"class\\\":\\\"\\\",\\\"kind\\\":\\\"http\\\",\\\"name\\\":\\\"myHttpTrigger\\\",\\\"maxWorkers\\\":2,\\\"workerAvailabilityTimeoutMilliseconds\\\":10000,\\\"attributes\\\":{\\\"maxRequestBodySize\\\":33554432}}},\\\"volumes\\\":[{\\\"volume\\\":{\\\"name\\\":\\\"volume-1\\\",\\\"hostPath\\\":{\\\"path\\\":\\\"/data/cvat/serverless/common\\\"}},\\\"volumeMount\\\":{\\\"name\\\":\\\"volume-1\\\",\\\"mountPath\\\":\\\"/opt/nuclio/common\\\"}}],\\\"build\\\":{\\\"image\\\":\\\"cvat/openvino.dextr\\\",\\\"baseImage\\\":\\\"openvino/ubuntu18_runtime:2020.2\\\",\\\"directives\\\":{\\\"postCopy\\\":[{\\\"kind\\\":\\\"RUN\\\",\\\"value\\\":\\\"curl -O https://download.01.org/openvinotoolkit/models_contrib/cvat/dextr_model_v1.zip\\\"},{\\\"kind\\\":\\\"RUN\\\",\\\"value\\\":\\\"unzip dextr_model_v1.zip\\\"},{\\\"kind\\\":\\\"RUN\\\",\\\"value\\\":\\\"pip3 install Pillow\\\"}],\\\"preCopy\\\":[{\\\"kind\\\":\\\"USER\\\",\\\"value\\\":\\\"root\\\"},{\\\"kind\\\":\\\"WORKDIR\\\",\\\"value\\\":\\\"/opt/nuclio\\\"},{\\\"kind\\\":\\\"RUN\\\",\\\"value\\\":\\\"ln -s /usr/bin/pip3 /usr/bin/pip\\\"}]},\\\"codeEntryType\\\":\\\"image\\\",\\\"timestamp\\\":1623091522},\\\"platform\\\":{\\\"attributes\\\":{\\\"mountMode\\\":\\\"volume\\\",\\\"restartPolicy\\\":{\\\"maximumRetryCount\\\":3,\\\"name\\\":\\\"always\\\"}}},\\\"readinessTimeoutSeconds\\\":60,\\\"securityContext\\\":{},\\\"eventTimeout\\\":\\\"30s\\\"}\",\n                \"nuclio.io/namespace\": \"nuclio\",\n                \"nuclio.io/platform\": \"local\",\n                \"nuclio.io/project-name\": \"cvat\"\n            }\n        },\n        \"NetworkSettings\": {\n            \"Bridge\": \"\",\n            \"SandboxID\": \"f2a29770e04c2ccbc6cf554bef44310c137cc32f7ef9ed7b04ca64be583725f2\",\n            \"HairpinMode\": false,\n            \"LinkLocalIPv6Address\": \"\",\n            \"LinkLocalIPv6PrefixLen\": 0,\n            \"Ports\": {\n                \"8080/tcp\": [\n                    {\n                        \"HostIp\": \"0.0.0.0\",\n                        \"HostPort\": \"49153\"\n                    },\n                    {\n                        \"HostIp\": \"::\",\n                        \"HostPort\": \"49153\"\n                    }\n                ]\n            },\n            \"SandboxKey\": \"/var/run/docker/netns/f2a29770e04c\",\n            \"SecondaryIPAddresses\": null,\n            \"SecondaryIPv6Addresses\": null,\n            \"EndpointID\": \"331557200fb48b5e4929c478ddb14605a22273fb21478198812982543d7edb5e\",\n            \"Gateway\": \"172.17.0.1\",\n            \"GlobalIPv6Address\": \"\",\n            \"GlobalIPv6PrefixLen\": 0,\n            \"IPAddress\": \"172.17.0.4\",\n            \"IPPrefixLen\": 16,\n            \"IPv6Gateway\": \"\",\n            \"MacAddress\": \"02:42:ac:11:00:04\",\n            \"Networks\": {\n                \"bridge\": {\n                    \"IPAMConfig\": null,\n                    \"Links\": null,\n                    \"Aliases\": null,\n                    \"NetworkID\": \"91a66fa558cbc58ff33e457a1412e09a2dc68842b6e7758e4f1c91f6d3a5dc95\",\n                    \"EndpointID\": \"331557200fb48b5e4929c478ddb14605a22273fb21478198812982543d7edb5e\",\n                    \"Gateway\": \"172.17.0.1\",\n                    \"IPAddress\": \"172.17.0.4\",\n                    \"IPPrefixLen\": 16,\n                    \"IPv6Gateway\": \"\",\n                    \"GlobalIPv6Address\": \"\",\n                    \"GlobalIPv6PrefixLen\": 0,\n                    \"MacAddress\": \"02:42:ac:11:00:04\",\n                    \"DriverOpts\": null\n                }\n            }\n        }\n    }\n]\n", "stderr": "", "exitCode": 0}
21.06.09 12:33:43.411 �[0;37mdashboard.platform.docker�[0m �[0;32m(D)�[0m Container is healthy {"containerID": "296d755cc856341fd4d244df57b567967157ada0272100abe65bb18f91031f7b"}

@azhavoro
Copy link
Contributor

azhavoro commented Jun 9, 2021

I can't find any errors in this part of the logs, please restart all containers with the compose down and up -d commands, try using the dextr model, then collect all the logs from the nuclio container and attach them as a zip archive. Thanks

@rathishkumar
Copy link
Author

@azhavoro I have tried multiple times, I could not see the errors in the nuclio log, however I can see the Gateway Timeout Error at the cvat log. I am not sure, what is causing this error, I could not curl and unable to install other packages to troubleshoot the issue.

@rathishkumar
Copy link
Author

@azhavoro I wanted to try adding 172.28.0.0/24 (subnet range mentioned in the error log) to no_proxy variable. How do I add this to no_proxy variable? there are two places, I am seeing: ~/.docker/config.json and docker service . I have tried adding to both the places and end up getting a different error model not loading something. I am not sure about the use cases of this two different files.

To set no_proxy value to image running inside the container, where do I need to modify? config.json or at the service level.

@rathishkumar
Copy link
Author

@nmanovic Could you please help on this issue?

@rathishkumar
Copy link
Author

Few more logs:

docker exec cvat env | grep no_proxy
no_proxy=kibana,logstash,nuclio,kibana,logstash,nuclio,172.28.0.0/24

cvat logs:

2021-06-10 06:02:35,354 DEBG 'runserver' stderr output:
[Thu Jun 10 06:02:35.354409 2021] [wsgi:error] [pid 243:tid 140625160574720] [remote 172.28.0.7:36292] INFO - 2021-
06-10 06:02:35,351 - views - get repository request
2021-06-10 06:02:37,048 DEBG 'runserver' stderr output:
[Thu Jun 10 06:02:37.048320 2021] [wsgi:error] [pid 243:tid 140625168967424] [remote 172.28.0.7:36318] INFO - 2021-
06-10 06:02:37,048 - views - {"client_id":942890,"name":"Send user activity","time":"2021-06-10T06:02:36.935000Z","
payload":{"working_time":12382},"is_active":true,"username":"admin"}
2021-06-10 06:03:52,475 DEBG 'runserver' stderr output:
[Thu Jun 10 06:03:52.474998 2021] [wsgi:error] [pid 275:tid 140625297118976] [client 172.28.0.7:36402] Timeout when
 reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: http://my-server/tasks/2/jobs/2
2021-06-10 06:03:52,539 DEBG 'runserver' stderr output:
[Thu Jun 10 06:03:52.539364 2021] [wsgi:error] [pid 243:tid 140625160574720] [remote 172.28.0.7:36402] [2021-06-10 
06:03:52,539] ERROR django.request: Gateway Timeout: /api/v1/lambda/functions/openvino-dextr
2021-06-10 06:03:52,539 DEBG 'runserver' stderr output:
[Thu Jun 10 06:03:52.539536 2021] [wsgi:error] [pid 243:tid 140625160574720] [remote 172.28.0.7:36402] ERROR - 2021
-06-10 06:03:52,539 - log - Gateway Timeout: /api/v1/lambda/functions/openvino-dextr
2021-06-10 06:05:32,721 DEBG 'runserver' stderr output:
[Thu Jun 10 06:05:32.721329 2021] [wsgi:error] [pid 275:tid 140625296578304] [client 172.28.0.7:36786] Timeout when
 reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: http://my-server/tasks/2/jobs/2
2021-06-10 06:05:32,883 DEBG 'runserver' stderr output:
[Thu Jun 10 06:05:32.883876 2021] [wsgi:error] [pid 243:tid 140625194145536] [remote 172.28.0.7:36786] [2021-06-10 
06:05:32,883] ERROR django.request: Gateway Timeout: /api/v1/lambda/functions/openvino-dextr
2021-06-10 06:05:32,884 DEBG 'runserver' stderr output:
[Thu Jun 10 06:05:32.884044 2021] [wsgi:error] [pid 243:tid 140625194145536] [remote 172.28.0.7:36786] ERROR - 2021
-06-10 06:05:32,883 - log - Gateway Timeout: /api/v1/lambda/functions/openvino-dextr

nuclio logs

 \"bind\",\n                \"Source\": \"/data/cvat/serverless/common\",\n                \"Destination\": \"/opt/nuclio/common\",\n                \"Mode\": \"\",\n                \"RW\": true,\n                \"Propagation\": \"rprivate\"\n            },\n            {\n                \"Type\": \"volume\",\n                \"Name\": \"nuclio-nuclio-openvino-dextr\",\n                \"Source\": \"/var/lib/docker/volumes/nuclio-nuclio-openvino-dextr/_data\",\n                \"Destination\": \"/etc/nuclio/config/processor\",\n                \"Driver\": \"local\",\n                \"Mode\": \"z\",\n                \"RW\": false,\n                \"Propagation\": \"\"\n            }\n        ],\n        \"Config\": {\n            \"Hostname\": \"0d8bee8ff72a\",\n            \"Domainname\": \"\",\n            \"User\": \"root\",\n            \"AttachStdin\": false,\n            \"AttachStdout\": false,\n            \"AttachStderr\": false,\n            \"ExposedPorts\": {\n                \"8080/tcp\": {}\n            },\n            \"Tty\": false,\n            \"OpenStdin\": false,\n            \"StdinOnce\": false,\n            \"Env\": [\n                \"NO_PROXY=*.247.ai,localhost,127.0.0.0/8,172.28.0.0/24\",\n                \"no_proxy=*.247.ai,localhost,127.0.0.0/8,172.28.0.0/24\",\n                \"NUCLIO_PYTHON_EXE_PATH=/opt/nuclio/common/openvino/python3\",\n                \"HTTP_PROXY=http://my-proxy:8080\",\n                \"http_proxy=http://my-proxy:8080\",\n                \"HTTPS_PROXY=http://my-proxy:8080\",\n                \"https_proxy=http://my-proxy8080\",\n                \"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\n                \"PYTHON=python3.6\",\n                \"INTEL_OPENVINO_DIR=/opt/intel/openvino\"\n            ],\n            \"Cmd\": [\n                \"processor\"\n            ],\n            \"Healthcheck\": {\n                \"Test\": [\n                    \"CMD-SHELL\",\n                    \"/usr/local/bin/uhttpc --url http://127.0.0.1:8082/ready || exit 1\"\n                ],\n                \"Interval\": 1000000000,\n                \"Timeout\": 3000000000\n            },\n            \"Image\": \"cvat/openvino.dextr:latest\",\n            \"Volumes\": null,\n            \"WorkingDir\": \"/opt/nuclio\",\n            \"Entrypoint\": null,\n            \"OnBuild\": null,\n            \"Labels\": {\n                \"Description\": \"This is the runtime image for Intel(R) Distribution of OpenVINO(TM) toolkit on Ubuntu 18.04 LTS\",\n                \"Vendor\": \"Intel Corporation\",\n                \"nuclio.io/annotations\": \"{\\\"framework\\\":\\\"openvino\\\",\\\"min_pos_points\\\":\\\"4\\\",\\\"name\\\":\\\"DEXTR\\\",\\\"spec\\\":\\\"\\\",\\\"type\\\":\\\"interactor\\\"}\",\n                \"nuclio.io/function-name\": \"openvino-dextr\",\n                \"nuclio.io/function-spec\": \"{\\\"description\\\":\\\"Deep Extreme Cut\\\",\\\"handler\\\":\\\"main:handler\\\",\\\"runtime\\\":\\\"python:3.6\\\",\\\"env\\\":[{\\\"name\\\":\\\"NUCLIO_PYTHON_EXE_PATH\\\",\\\"value\\\":\\\"/opt/nuclio/common/openvino/python3\\\"}],\\\"resources\\\":{},\\\"image\\\":\\\"cvat/openvino.dextr:latest\\\",\\\"targetCPU\\\":75,\\\"triggers\\\":{\\\"myHttpTrigger\\\":{\\\"class\\\":\\\"\\\",\\\"kind\\\":\\\"http\\\",\\\"name\\\":\\\"myHttpTrigger\\\",\\\"maxWorkers\\\":2,\\\"workerAvailabilityTimeoutMilliseconds\\\":10000,\\\"attributes\\\":{\\\"maxRequestBodySize\\\":33554432}}},\\\"volumes\\\":[{\\\"volume\\\":{\\\"name\\\":\\\"volume-1\\\",\\\"hostPath\\\":{\\\"path\\\":\\\"/data/cvat/serverless/common\\\"}},\\\"volumeMount\\\":{\\\"name\\\":\\\"volume-1\\\",\\\"mountPath\\\":\\\"/opt/nuclio/common\\\"}}],\\\"build\\\":{\\\"image\\\":\\\"cvat/openvino.dextr\\\",\\\"baseImage\\\":\\\"openvino/ubuntu18_runtime:2020.2\\\",\\\"directives\\\":{\\\"postCopy\\\":[{\\\"kind\\\":\\\"RUN\\\",\\\"value\\\":\\\"curl -O https://download.01.org/openvinotoolkit/models_contrib/cvat/dextr_model_v1.zip\\\"},{\\\"kind\\\":\\\"RUN\\\",\\\"value\\\":\\\"unzip dextr_model_v1.zip\\\"},{\\\"kind\\\":\\\"RUN\\\",\\\"value\\\":\\\"pip3 install Pillow\\\"}],\\\"preCopy\\\":[{\\\"kind\\\":\\\"USER\\\",\\\"value\\\":\\\"root\\\"},{\\\"kind\\\":\\\"WORKDIR\\\",\\\"value\\\":\\\"/opt/nuclio\\\"},{\\\"kind\\\":\\\"RUN\\\",\\\"value\\\":\\\"ln -s /usr/bin/pip3 /usr/bin/pip\\\"}]},\\\"codeEntryType\\\":\\\"image\\\",\\\"timestamp\\\":1623304083},\\\"platform\\\":{\\\"attributes\\\":{\\\"mountMode\\\":\\\"volume\\\",\\\"restartPolicy\\\":{\\\"maximumRetryCount\\\":3,\\\"name\\\":\\\"always\\\"}}},\\\"readinessTimeoutSeconds\\\":60,\\\"securityContext\\\":{},\\\"eventTimeout\\\":\\\"30s\\\"}\",\n                \"nuclio.io/namespace\": \"nuclio\",\n                \"nuclio.io/platform\": \"local\",\n                \"nuclio.io/project-name\": \"cvat\"\n            }\n        },\n        \"NetworkSettings\": {\n            \"Bridge\": \"\",\n            \"SandboxID\": \"60f7a866b3247f50e92188ee82235c8a8b356175f3ec5ed8b8be283b8a65d9ff\",\n            \"HairpinMode\": false,\n            \"LinkLocalIPv6Address\": \"\",\n            \"LinkLocalIPv6PrefixLen\": 0,\n            \"Ports\": {\n                \"8080/tcp\": [\n                    {\n                        \"HostIp\": \"0.0.0.0\",\n                        \"HostPort\": \"49153\"\n                    },\n                    {\n                        \"HostIp\": \"::\",\n                        \"HostPort\": \"49153\"\n                    }\n                ]\n            },\n            \"SandboxKey\": \"/var/run/docker/netns/60f7a866b324\",\n            \"SecondaryIPAddresses\": null,\n            \"SecondaryIPv6Addresses\": null,\n            \"EndpointID\": \"86b9592499e7cf75d62ba6b6863c870f582ecba4b5439ff02ce1f5619a3b4367\",\n            \"Gateway\": \"172.17.0.1\",\n            \"GlobalIPv6Address\": \"\",\n            \"GlobalIPv6PrefixLen\": 0,\n            \"IPAddress\": \"172.17.0.4\",\n            \"IPPrefixLen\": 16,\n            \"IPv6Gateway\": \"\",\n            \"MacAddress\": \"02:42:ac:11:00:04\",\n            \"Networks\": {\n                \"bridge\": {\n                    \"IPAMConfig\": null,\n                    \"Links\": null,\n                    \"Aliases\": null,\n                    \"NetworkID\": \"2c9f5d1439f0b89c9505c87733afd1400a12ca52c6e9832f39f76df37becfd5d\",\n                    \"EndpointID\": \"86b9592499e7cf75d62ba6b6863c870f582ecba4b5439ff02ce1f5619a3b4367\",\n                    \"Gateway\": \"172.17.0.1\",\n                    \"IPAddress\": \"172.17.0.4\",\n                    \"IPPrefixLen\": 16,\n                    \"IPv6Gateway\": \"\",\n                    \"GlobalIPv6Address\": \"\",\n                    \"GlobalIPv6PrefixLen\": 0,\n                    \"MacAddress\": \"02:42:ac:11:00:04\",\n                    \"DriverOpts\": null\n                }\n            }\n        }\n    }\n]\n", "stderr": "", "exitCode": 0}
21.06.10 06:30:53.875 dashboard.platform.docker (D) Container is healthy {"containerID": "0d8bee8ff72a6e1c83f936af5e60bd02904b910fe9f73a2759883096856b114f"}

dextr logs

21.06.10 05:48:04.293 cessor.healthcheck.server (I) Listening {"listenAddress": ":8082"}
21.06.10 05:48:04.293            processor.http (D) Creating worker pool {"num": 2}
21.06.10 05:48:04.314 sor.http.w1.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-c30qf503c5g3e8lkbtj0.sock"}
21.06.10 05:48:04.315 sor.http.w0.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-c30qf503c5g3e8lkbtjg.sock"}
21.06.10 05:48:04.315 sor.http.w1.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
21.06.10 05:48:04.315 sor.http.w0.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
21.06.10 05:48:04.315 sor.http.w1.python.logger (D) Using Python handler {"handler": "main:handler"}
21.06.10 05:48:04.315 sor.http.w0.python.logger (D) Using Python handler {"handler": "main:handler"}
21.06.10 05:48:04.315 sor.http.w1.python.logger (D) Using Python executable {"path": "/opt/nuclio/common/openvino/python3"}
21.06.10 05:48:04.315 sor.http.w0.python.logger (D) Using Python executable {"path": "/opt/nuclio/common/openvino/python3"}
21.06.10 05:48:04.315 sor.http.w0.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio"}
21.06.10 05:48:04.315 sor.http.w1.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio"}
21.06.10 05:48:04.315 sor.http.w1.python.logger (D) Running wrapper {"command": "/opt/nuclio/common/openvino/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-c30qf503c5g3e8lkbtj0.sock
 --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger"}
21.06.10 05:48:04.315 sor.http.w0.python.logger (D) Running wrapper {"command": "/opt/nuclio/common/openvino/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-c30qf503c5g3e8lkbtjg.sock
 --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger"}
[setupvars.sh] OpenVINO environment initialized
[setupvars.sh] OpenVINO environment initialized
21.06.10 05:48:04.769 sor.http.w1.python.logger (I) Wrapper connected {"wid": 1, "pid": 13}
21.06.10 05:48:04.770 sor.http.w1.python.logger (D) Waiting for start
l{"datetime": "2021-06-10 05:48:04,770", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "1"}}
21.06.10 05:48:04.779 sor.http.w0.python.logger (I) Wrapper connected {"wid": 0, "pid": 14}
21.06.10 05:48:04.780 sor.http.w0.python.logger (D) Waiting for start
l{"datetime": "2021-06-10 05:48:04,780", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "0"}}
l{"datetime": "2021-06-10 05:48:08,886", "level": "info", "message": "Init context...100%", "with": {"worker_id": "1"}}
l{"datetime": "2021-06-10 05:48:08,886", "level": "info", "message": "Replacing logger output", "with": {"handler_name": "default", "worker_id": "1"}}
21.06.10 05:48:08.886 sor.http.w1.python.logger (D) Started
l{"datetime": "2021-06-10 05:48:08,902", "level": "info", "message": "Init context...100%", "with": {"worker_id": "0"}}
l{"datetime": "2021-06-10 05:48:08,902", "level": "info", "message": "Replacing logger output", "with": {"handler_name": "default", "worker_id": "0"}}
21.06.10 05:48:08.903 sor.http.w0.python.logger (D) Started
21.06.10 05:48:08.903                 processor (I) Starting event timeout watcher {"timeout": "30s"}
21.06.10 05:48:08.903 .webadmin.server.triggers (D) Registered custom route {"pattern": "/{id}/stats", "method": "GET"}
21.06.10 05:48:08.903 processor.webadmin.server (D) Registered resource {"name": "triggers"}
21.06.10 05:48:08.903                 processor (W) No metric sinks configured, metrics will not be published
21.06.10 05:48:08.903                 processor (D) Starting triggers {"triggers": [{"ID":"myHttpTrigger","Logger":{},"WorkerAllocator":{},"Class":"sync","Kind":"http","Name":"myHttpTrigger","Statistics":{"EventsHandledSuccessTot
al":0,"EventsHandledFailureTotal":0,"WorkerAllocatorStatistics":{"WorkerAllocationCount":0,"WorkerAllocationSuccessImmediateTotal":0,"WorkerAllocationSuccessAfterWaitTotal":0,"WorkerAllocationTimeoutTotal":0,"WorkerAllocationWait
DurationMilliSecondsSum":0,"WorkerAllocationWorkersAvailablePercentage":0}},"Namespace":"nuclio","FunctionName":"openvino-dextr"}]}
21.06.10 05:48:08.903            processor.http (I) Starting {"listenAddress": ":8080", "readBufferSize": 16384, "maxRequestBodySize": 33554432, "cors": null}
21.06.10 05:48:08.903 processor.webadmin.server (I) Listening {"listenAddress": ":8081"}
21.06.10 05:48:08.903                 processor (D) Processor started
21.06.10 05:54:02.094 sor.http.w0.python.logger (I) call handler {"worker_id": "0"}
21.06.10 05:54:02.095 sor.http.w0.python.logger (E) Exception caught in handler {"traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 114, in serve_requests\n    self._handle_event(eve
nt)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 262, in _handle_event\n    entrypoint_output = self._entrypoint(self._context, event)\n  File \"/opt/nuclio/main.py\", line 18, in handler\n    points = data[\"pos_points\"]\nT
ypeError: 'NoneType' object is not subscriptable\n", "worker_id": "0", "exc": "'NoneType' object is not subscriptable"}
root@gc-ase-ml-data-annotate1:/data/cvat# 

@nmanovic nmanovic self-assigned this Jul 6, 2021
@nmanovic
Copy link
Contributor

nmanovic commented Jul 8, 2021

@rathishkumar , it is very difficult to help especially if you don't have access to the machine. Let's try to exclude CVAT from the equation. Please go to http://localhost:8070/projects/cvat/functions/openvino-dextr/code. In body select JSON and past the content from input.json (see below). Press the test button. What can you see? Does it work?

To generate input.json please do the following:

image=$(curl https://upload.wikimedia.org/wikipedia/en/7/7d/Lenna_%28test_image%29.png --output - | base64 | tr -d '\n')
cat << EOF > /tmp/input.json
{"image": "$image", "pos_points": [[10,10],[100,10],[100,100], [10,100]]}
EOF

image

@nikste
Copy link

nikste commented Jul 20, 2021

Hi @nmanovic, I have a custom nuclio detector that also times out (after 60s with the 504).
Just tried what you suggested above and in the nuclio ui it also times out.

@nmanovic
Copy link
Contributor

@nikste , could you please report the issue here https://github.com/nuclio/nuclio/issues?

@nikste
Copy link

nikste commented Jul 20, 2021

@nmanovic it seems to allready exist: nuclio/nuclio#1162
curiously if you send a request with curl, it does not send the timeout.

@nmanovic nmanovic added nuclio and removed need info Need more information to investigate the issue labels Jul 20, 2021
@nmanovic
Copy link
Contributor

@nikste , @rathishkumar ,

Do you have proxy on your machine? If so you need:

  1. Add information about your proxy into ~/.docker/config.json file: https://openvinotoolkit.github.io/cvat/docs/administration/basics/installation/#deploying-cvat-behind-a-proxy
  2. Also it is important to add your docker gateway into noProxy (e.g. 172.17.0.1). Nuclio dashboard and serverless functions are talking over the docker gateway and if it is protected by a proxy, they cannot talk.
  3. Down/Up nuclio container (Nuclio dashboard)

@basashi0215
Copy link

@nikste , @rathishkumar ,

Do you have proxy on your machine? If so you need:

  1. Add information about your proxy into ~/.docker/config.json file: https://openvinotoolkit.github.io/cvat/docs/administration/basics/installation/#deploying-cvat-behind-a-proxy
  2. Also it is important to add your docker gateway into noProxy (e.g. 172.17.0.1). Nuclio dashboard and serverless functions are talking over the docker gateway and if it is protected by a proxy, they cannot talk.
  3. Down/Up nuclio container (Nuclio dashboard)

I faced the same 504 Error problem as they did.
I was trying to implement CVAT in a proxied environment.
I tried the workaround you suggested and was able to avoid it.
Thanks

@nmanovic nmanovic added bug Something isn't working nuclio and removed nuclio labels Nov 23, 2021
@kingzcheung
Copy link

I also encountered the same problem.I feel it is a network problem between containers, but I don’t know how to solve it。

@kingzcheung
Copy link

docker container inspect cvat

...
"Networks": {
                "cvat_cvat": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "cvat",
                        "9cb15d0f7718"
                    ],
                    "NetworkID": "5f4854e28570b8fd1a8c5335c7321a52244a6a854d22cc7b85db99d2a1b2ca3c",
                    "EndpointID": "7606749e54d49a66dcef96ba30d950e5cb010d72c8d28b4e1208192e93f8c540",
                    "Gateway": "10.10.9.1",
                    "IPAddress": "10.10.9.6",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:0a:09:06",
                    "DriverOpts": null
                }
            }

docker container inspect nuclio

"Networks": {
                "cvat_cvat": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "6ec05751fa5b",
                        "nuclio"
                    ],
                    "NetworkID": "5f4854e28570b8fd1a8c5335c7321a52244a6a854d22cc7b85db99d2a1b2ca3c",
                    "EndpointID": "73da60d9e1347fe384c9a8214df588d54185fb6f98c5169e44e0875041b1dbd5",
                    "Gateway": "10.10.9.1",
                    "IPAddress": "10.10.9.2",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:0a:09:02",
                    "DriverOpts": null
                }
            }

I had set up my funtion so that its network and cvat are consistent:

platform:
    attributes:
      restartPolicy:
        name: always
        maximumRetryCount: 3
      mountMode: volume
      network: cvat_cvat

docker container inspect nuclio-nuclio-opencv-yolov4-detector

"Networks": {
                "cvat_cvat": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "3091481a95fb"
                    ],
                    "NetworkID": "5f4854e28570b8fd1a8c5335c7321a52244a6a854d22cc7b85db99d2a1b2ca3c",
                    "EndpointID": "eaac2aebcd89d963fab0ec8ec74a5bd8cb3b9abac6f9de60e73f7a83bcade265",
                    "Gateway": "10.10.9.1",
                    "IPAddress": "10.10.9.8",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:0a:09:08",
                    "DriverOpts": null
                }
            }

nuclio-nuclio-opencv-yolov4-detector is my function container name.

web error:

Error: Inference status for the task 53 is failed. requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: http://nuclio:8070/api/function_invocations

CVAT ERROR:

2021-12-16 02:08:33,206 DEBG 'rqworker_low' stderr output:
DEBUG - 2021-12-16 02:08:33,205 - worker - Handling failed execution of job 0341a7c1-ecf5-4f8d-9f01-303e5e9f3af7

2021-12-16 02:08:33,210 DEBG 'rqworker_low' stderr output:
ERROR - 2021-12-16 02:08:33,209 - worker - Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/rq/worker.py", line 936, in perform_job
    rv = job.perform()
  File "/opt/venv/lib/python3.8/site-packages/rq/job.py", line 684, in perform
    self._result = self._execute()
  File "/opt/venv/lib/python3.8/site-packages/rq/job.py", line 690, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/home/django/cvat/apps/lambda_manager/views.py", line 506, in __call__
    LambdaJob._call_detector(function, db_task, labels, quality,
  File "/home/django/cvat/apps/lambda_manager/views.py", line 374, in _call_detector
    annotations = function.invoke(db_task, data={
  File "/home/django/cvat/apps/lambda_manager/views.py", line 208, in invoke
    response = self.gateway.invoke(self, payload)
  File "/home/django/cvat/apps/lambda_manager/views.py", line 78, in invoke
    return self._http(method="post", url='/api/function_invocations',
  File "/home/django/cvat/apps/lambda_manager/views.py", line 56, in _http
    reply.raise_for_status()
  File "/opt/venv/lib/python3.8/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: http://nuclio:8070/api/function_invocations
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/rq/worker.py", line 936, in perform_job
    rv = job.perform()
  File "/opt/venv/lib/python3.8/site-packages/rq/job.py", line 684, in perform
    self._result = self._execute()
  File "/opt/venv/lib/python3.8/site-packages/rq/job.py", line 690, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/home/django/cvat/apps/lambda_manager/views.py", line 506, in __call__
    LambdaJob._call_detector(function, db_task, labels, quality,
  File "/home/django/cvat/apps/lambda_manager/views.py", line 374, in _call_detector
    annotations = function.invoke(db_task, data={
  File "/home/django/cvat/apps/lambda_manager/views.py", line 208, in invoke
    response = self.gateway.invoke(self, payload)
  File "/home/django/cvat/apps/lambda_manager/views.py", line 78, in invoke
    return self._http(method="post", url='/api/function_invocations',
  File "/home/django/cvat/apps/lambda_manager/views.py", line 56, in _http
    reply.raise_for_status()
  File "/opt/venv/lib/python3.8/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: http://nuclio:8070/api/function_invocations

Its OK by nuctl:

[root@localhost cvat]# nuctl get function
  NAMESPACE |          NAME          | PROJECT |  STATE   | REPLICAS |
  nuclio    | helloworld             | default | ready    | 1/1      | 49153
  nuclio    | opencv-yolov4-detector | cvat    | ready    | 1/1      | 49155
  nuclio    | openvino-dextr         | cvat    | building | 1/1      |
[root@localhost cvat]# nuctl invoke opencv-yolov4-detector
21.12.16 10:20:02.568    nuctl.platform.invoker (I) Executing function {"method": "GET", "url": "http://:49155", "headers": {"Content-Type":["text/plain"],"X-Nuclio-Log-Level":["info"],"X-Nuclio-Target":["opencv-yolov4-detector"]}}
21.12.16 10:20:02.572    nuctl.platform.invoker (I) Got response {"status": "500 Internal Server Error"}
21.12.16 10:20:02.572                     nuctl (I) >>> Start of function logs
21.12.16 10:20:02.572    opencv-yolov4-detector (E) Exception caught in handler {"traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 118, in serve_requests\n    await self._handle_event(event)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 312, in _handle_event\n    entrypoint_output = self._entrypoint(self._context, event)\n  File \"/opt/nuclio/main.py\", line 31, in handler\n    buf = base64.b64decode(data[\"image\"])\nTypeError: 'NoneType' object is not subscriptable\n", "worker_id": "0", "exc": "'NoneType' object is not subscriptable", "time": 1639621202572.2375}
21.12.16 10:20:02.573                     nuctl (I) <<< End of function logs

> Response headers:
Content-Length = 485
Server = nuclio
Date = Thu, 16 Dec 2021 02:20:02 GMT
Content-Type = text/plain

> Response body:
Exception caught in handler - "'NoneType' object is not subscriptable": Traceback (most recent call last):
  File "/opt/nuclio/_nuclio_wrapper.py", line 118, in serve_requests
    await self._handle_event(event)
  File "/opt/nuclio/_nuclio_wrapper.py", line 312, in _handle_event
    entrypoint_output = self._entrypoint(self._context, event)
  File "/opt/nuclio/main.py", line 31, in handler
    buf = base64.b64decode(data["image"])
TypeError: 'NoneType' object is not subscriptable

@nvqui97
Copy link

nvqui97 commented Oct 8, 2022

my server doesn't have proxy but i also met 504 Gateway Timeout.
I have run docker exec cvat_server env | grep no_proxy and get no_proxy=elasticsearch,kibana,logstash,nuclio,opa,

Can you help me solve this?!!!!!

@nvqui97
Copy link

nvqui97 commented Oct 14, 2022

Please help me to solve this. My server doesn't have proxy. I also see the result in model container but on interface isn't
image

nmanovic pushed a commit that referenced this issue Sep 11, 2023
See these several issues. They are connected to each other in some way. 
The thing is that nuclio has a default timeout of 1 minute. With this
change we can force nuclio dashboard not to terminate the connection.
nuclio/nuclio#3016
#3301
#6041
mikhail-treskin pushed a commit to retailnext/cvat that referenced this issue Oct 25, 2023
See these several issues. They are connected to each other in some way. 
The thing is that nuclio has a default timeout of 1 minute. With this
change we can force nuclio dashboard not to terminate the connection.
nuclio/nuclio#3016
cvat-ai#3301
cvat-ai#6041
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working nuclio
Projects
None yet
Development

No branches or pull requests

8 participants