Cant create task - CVAT served via cloudflare tunnels - Content Mismatch blocks uploads.

[baudneo]

My actions before raising this issue

• Read/searched the docs
• Searched past issues

I host CVAT using cloudflared. Cloudflare handles SSL termination and then proxies the request to CVAT over HTTP. I am unable to upload any images to tasks with this setup.

The workaround I am using for now is to setup CVAT locally and create tasks, then proxy it via cloudflared so people have access to annotate images.

Dev console output:

httpStack.js:96 Mixed Content: The page at 'https://cvat.<X>.com/tasks/create' was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint 'http://cvat.<X>.com/api/tasks/31/data/f64a6607-a93c-4f04-a483-b0b05c737a7d'. This request has been blocked; the content must be served over HTTPS.
(anonymous) @ httpStack.js:96

cvat-app.tsx:359 Error: tus: failed to upload chunk at offset 0, caused by [object ProgressEvent], originated from request (method: PATCH, url: http://cvat.<X>.com/api/tasks/31/data/f64a6607-a93c-4f04-a483-b0b05c737a7d, response code: n/a, response text: n/a, request id: n/a).

Steps to Reproduce (for bugs)

1. Clone CVAT repo
2. Export CVAT_HOST to FQDN: cvat.<mydomain>.com
3. Edit docker-compose.yml and add CVAT_HOST: ${CVAT_HOST:localhost-} to cvat_server environment section.
4. Setup cloudflared tunnel and host CVAT on your domain @
5. docker compose up -d
6. Create superuser
7. Wait for CVAT to come up fully, login with super-user, create a project
8. Create a task in the project and upload images.

Expected Behaviour

I expect that the Task creation is successful. Maybe a way to control the protocol in the URL via an ENV var like CVAT_BASE_URL (As far as I can tell CVAT_BASE_URL doesnt do anything? search CVAT_BASE_URL in cvat repo)

Current Behaviour

Task fails to be created with output from dev tools.

Possible Solution

Allow control of the protocol for the task creation url param. Possibly CVAT_BASE_URL ?

Context

I am trying to upload images and create a tasks. As of now the workaround is to deploy CVAT locally, create tasks then take it all down just to recreate the containers so cloudflared can proxy.

Your Environment

• Git hash commit (git log -1): ee1d421

• Docker version docker version (e.g. Docker 17.0.05): 24.0.5

• Are you using Docker Swarm or Kubernetes? docker

• Operating System and version (e.g. Linux, Windows, MacOS): Debian 11.7 (Proxmox) - Linux

• Code example or link to GitHub repo or gist to reproduce problem:

• Other diagnostic information / logs:

  Logs from `cvat` container

  2023-09-28 03:12:23,093 DEBG 'uvicorn-0' stdout output:
  INFO:     172.24.0.1:0 - "GET /api/projects/3 HTTP/1.0" 200 OK
  
  2023-09-28 03:12:23,240 DEBG 'uvicorn-1' stdout output:
  INFO:     172.24.0.1:0 - "GET /api/labels?project_id=3&org=CloudMule&page_size=500&page=1 HTTP/1.0" 200 OK
  
  2023-09-28 03:12:23,672 DEBG 'uvicorn-1' stdout output:
  INFO:     172.24.0.1:0 - "POST /api/tasks?org=CloudMule HTTP/1.0" 201 Created
  
  2023-09-28 03:12:23,902 DEBG 'uvicorn-0' stdout output:
  INFO:     172.24.0.1:0 - "POST /api/tasks/34/data?org=CloudMule HTTP/1.0" 202 Accepted
  
  2023-09-28 03:12:24,065 DEBG 'uvicorn-0' stdout output:
  INFO:     172.24.0.1:0 - "POST /api/tasks/34/data/ HTTP/1.0" 201 Created
  
  2023-09-28 03:12:24,359 DEBG 'uvicorn-0' stdout output:
  INFO:     172.24.0.1:0 - "DELETE /api/tasks/34?org=CloudMule HTTP/1.0" 204 No Content
  

[baudneo]

Hi, sorry to bug but, does anyone have any advice for this? I can't upload any images to CVAT!

I tried changing CVAT_HOST to use https://<fqdn> but when I do that and --force-recreate, the UI returns 404 page not found. So I will leave it as FQDN without protocol.

I set CVAT_BASE_URL: https://<fqdn> and I get the same mixed content blocked error when trying to create a task. I searched for CVAT_BASE_URL in CVAT repo and it doesnt seem to do anything?

╰─❯ docker exec -it cvat_server bash
django@d7a22a83b430:~$ env | grep CVAT
CVAT_HOST=cvat.xxxxxx.com
CVAT_REDIS_HOST=cvat_redis
CVAT_POSTGRES_HOST=cvat_db
CVAT_ANALYTICS=1
CVAT_BASE_URL=https://cvat.xxxxxx.com
django@d7a22a83b430:~$

[baudneo]

Hi, sorry to bug again but can anyone give me some advice? Is there some source code I can change to configure the protocol for these requests?

Any advice or guidance would be extremely appreciated.

Thank you.

[azhavoro]

@baudneo take a look at this #6322

[baudneo]

@baudneo take a look at this #6322

Hi, I'm active in that issue thread and have already applied that fix to my cvat setup. I'm the guy who mentioned CVAT_HOST isn't in the docker-compose cvat_server environment config (should be last comment in that issue).

This issue only happens on task image upload. Issue 6322 is in relation to the Django admin page and POST.

[azhavoro]

@baudneo Probably the reason is that all traffic coming to cvat is http, so the backend knows nothing about ssl terminating by cloudflare.
#4843

We'll check if perhaps this can be fixed on the backend side, without having to set ugly settings like trustedIPs

[baudneo]

The fix in #4843 works for me. I added CloudFlare IP ranges to traefik continer:

- '--entryPoints.web.forwardedHeaders.trustedIPs=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22'

[baudneo]

Should I leave this issue open? My issue is solved with the current solution but there is a mention of a newer fix proposal. I am unsure if I should close this issue or wait until a PR is proposed?

[baudneo]

This is very strange, I was able to create tasks and all of a sudden I am unable to create tasks. I get the tus error, but I don't see the Mismatched-Content error anymore:

Could not create the task

Error: tus: failed to upload chunk at offset 0, caused by [object ProgressEvent], originated from request (method: PATCH, url: http://cvat.<REDACTED>.com/api/tasks/5/data/63f88cfa-efe0-4f87-b823-7a903c706e2b, response code: n/a, response text: n/a, request id: n/a).

Logs from docker compose logs -f:

cvat_server                    | 2023-10-13 01:38:16,770 DEBG 'uvicorn-0' stdout output:
cvat_server                    | INFO:     172.22.0.1:0 - "GET /api/projects/2 HTTP/1.0" 200 OK
cvat_server                    |
cvat_server                    | 2023-10-13 01:38:16,873 DEBG 'uvicorn-0' stdout output:
cvat_server                    | INFO:     172.22.0.7:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified
cvat_server                    |
cvat_server                    | 2023-10-13 01:38:16,959 DEBG 'uvicorn-1' stdout output:
cvat_server                    | INFO:     172.22.0.1:0 - "GET /api/labels?project_id=2&org=&page_size=500&page=1 HTTP/1.0" 200 OK
cvat_server                    |
cvat_server                    | 2023-10-13 01:38:17,374 DEBG 'uvicorn-0' stdout output:
cvat_server                    | INFO:     172.22.0.1:0 - "POST /api/tasks?org= HTTP/1.0" 201 Created
cvat_server                    |
cvat_server                    | 2023-10-13 01:38:17,761 DEBG 'uvicorn-1' stdout output:
cvat_server                    | INFO:     172.22.0.1:0 - "POST /api/tasks/5/data?org= HTTP/1.0" 202 Accepted
cvat_server                    |
cvat_server                    | 2023-10-13 01:38:17,956 DEBG 'uvicorn-0' stdout output:
cvat_server                    | INFO:     172.22.0.1:0 - "POST /api/tasks/5/data/ HTTP/1.0" 201 Created
cvat_server                    |
cvat_server                    | 2023-10-13 01:38:18,297 DEBG 'uvicorn-0' stdout output:
cvat_server                    | INFO:     172.22.0.1:0 - "DELETE /api/tasks/5 HTTP/1.0" 204 No Content
cvat_server                    |

I get the same thing when trying to import dataset as well. I am perplexed.

Edit:
There is a weird warning in traefik DEBUG logs for cvat_server and cvat_ui:

traefik                        | time="2023-10-13T02:02:16Z" level=warning msg="Could not find network named 'cvat' for container '/cvat_ui'! Maybe you're missing the project's prefix in the label? Defaulting to first available network." providerName=docker container=cvat-ui-cvat-ffcdb0fe823bfd37e399ab2b68c2fcddd89e28447f87bb268f417b41e429ab2d serviceName=cvat-ui
-------------------------------------------------

traefik                        | time="2023-10-13T02:02:16Z" level=warning msg="Could not find network named 'cvat' for container '/cvat_server'! Maybe you're missing the project's prefix in the label? Defaulting to first available network." serviceName=cvat providerName=docker container=cvat-server-cvat-18b2d93dc8c774858312dc9ce0257f0e574dc0b483dda959b10872dc886e35ac

There is a cvat_cvat network.

╰─❯ docker network ls
NETWORK ID     NAME                  DRIVER    SCOPE
32463f9aaca3   bridge                bridge    local
e626d6bee2b5   cloudflared_default   bridge    local
e663802c6bf7   cvat_cvat             bridge    local
4105980eab99   host                  host      local
4ea78961b37d   none                  null      local

Both cvat_ui and cvat_server have

# cvat_server
    networks:
      cvat:
        aliases:
          - cvat-server
# cvat_ui
    networks:
      - cvat

[baudneo]

The only thing that has changed is a did a docker compose pull. So maybe something in new image is causing this?

[jpfleischer]

I did have an issue where i am using cvat over a cloudflare tunnel, where a task page was reloading infinitely. I thought it was some problem with CORS as I saw that error come up in the Console of the web browser, but it was a red herring.

Turns out the problem was that, I had grafted the task page into another organization when it was part of a separate project, and those tasks were bugged. I just needed to make a new task and then it works.