Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add quality modes and management docs #8732

Merged
merged 32 commits into from
Dec 4, 2024
Merged
Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
d1b933e
Add quality modes and management docs
zhiltsov-max Nov 21, 2024
94e1e65
add annotation management section
zhiltsov-max Nov 21, 2024
110bc8d
Partially update auto-qa section
zhiltsov-max Nov 22, 2024
9998749
Update docs
zhiltsov-max Nov 25, 2024
d8ea6ea
Apply suggestions from code review
zhiltsov-max Nov 29, 2024
85a8e11
Fix some typos and grammar
zhiltsov-max Nov 29, 2024
c521a9a
Resolve some comments and linter errors
zhiltsov-max Nov 29, 2024
3597da8
Describe parameters in modes more precisely
zhiltsov-max Nov 29, 2024
7754cf2
Improve parameter descriptions for honeypots
zhiltsov-max Nov 29, 2024
bff1094
Apply suggestions from code review
zhiltsov-max Dec 2, 2024
21a753f
Remove duplicating toc
zhiltsov-max Dec 2, 2024
8eefe52
Move gt management section
zhiltsov-max Dec 2, 2024
881a287
Fix list formatting
zhiltsov-max Dec 2, 2024
4becbb3
Add info about quality checks
zhiltsov-max Dec 2, 2024
f944e09
Update quality settings section
zhiltsov-max Dec 2, 2024
1b1cc6d
Update mode summary table
zhiltsov-max Dec 2, 2024
9198d7f
Add configuration link
zhiltsov-max Dec 2, 2024
4fce7ad
Fix some errors
zhiltsov-max Dec 2, 2024
61191a5
Improve conflict review section
zhiltsov-max Dec 2, 2024
5b9dbdf
Clear heading
zhiltsov-max Dec 2, 2024
a26addc
Add information on comparisons for annotations
zhiltsov-max Dec 2, 2024
a6a0ef6
Add info on quality reports
zhiltsov-max Dec 2, 2024
9696230
Improve quality analytics section
zhiltsov-max Dec 2, 2024
d5eb4da
Merge remote-tracking branch 'origin/zm/quality-docs' into zm/quality…
zhiltsov-max Dec 2, 2024
1787ab3
Fix linter warnings
zhiltsov-max Dec 2, 2024
fe3373c
Add some tabs
zhiltsov-max Dec 2, 2024
f872f18
Apply suggestions from code review
zhiltsov-max Dec 4, 2024
4dfcd99
Improve table layout
zhiltsov-max Dec 4, 2024
f41859d
Improve navigation in qa overview and immediate feedback sections
zhiltsov-max Dec 4, 2024
2f97d2d
Refactor immediate feedback section
zhiltsov-max Dec 4, 2024
54c57fc
Fix types, move tracks section
zhiltsov-max Dec 4, 2024
064a4c1
Compress some images
zhiltsov-max Dec 4, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 4 additions & 5 deletions site/content/en/docs/enterprise/immediate-feedback.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,10 @@ The annotator can either agree or disagree with the feedback.
If they disagree, they have the option to re-annotate the job and request feedback again.
However, feedback is only available a limited number of times, as specified in the task's quality settings.

To ensure transparency with the annotator, the immediate feedback shows the collected score and
To ensure transparency with the annotator, the immediate feedback shows the collected score and
the minimum required score.
Immediate feedback settings, such as `Target metric`, `Target metric threshold`,
Immediate feedback settings, such as `Target metric`, `Target metric threshold`,
`Max validations per job` and others, can be configured on the quality settings page:
<!--- TODO: Update quality page documentation and refer from this section -->

<img src="/images/immediate-feedback-quality-settings.png" style="max-width: 600px;">

Expand All @@ -40,11 +39,11 @@ In this case annotators do not see any feedback dialogs.
> The number of attempts resets when the job assignee is updated.

Requirements:
1. The task is configured with a Ground Truth job that has been annotated,
1. The task is configured with a Ground Truth job that has been annotated,
moved to the acceptance stage, and is in the completed state.
2. The current job is in the annotation stage.
3. The current job is a regular annotation job. Immediate feedback is not available for Ground Truth jobs
4. The `Max validations per job` setting has been configured on the quality settings page.
4. The `Max validations per job` setting has been configured on the quality settings page.



Large diffs are not rendered by default.

86 changes: 86 additions & 0 deletions site/content/en/docs/manual/basics/quality-control.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
---
title: 'Quality control'
linkTitle: 'Quality control'
weight: 21
description: 'Overview of quality control features'
---

CVAT has the following features for automated quality control of annotations:
- [Validation set configuration for a task](#how-to-enable-quality-control-for-a-new-task)
- Job validation on job finish ("{{< ilink "/docs/enterprise/immediate-feedback" "Immediate feedback" >}}")
- [Review mode for problems found](#how-to-review-problems-found)
- [Quality analytics](#how-to-check-task-quality-metrics)

In this section we only highlight the key steps in quality estimation.
Read the detailed guide on quality estimation in CVAT in the
{{< ilink "/docs/manual/advanced/analytics-and-monitoring/auto-qa" "Advanced section" >}}.

## How to enable quality control for a new task

1. Go to task creation
2. Select the source media, configure other task parameters
3. Scroll down to the **Quality Control** section
4. Select one of the
{{< ilink "/docs/manual/advanced/analytics-and-monitoring/auto-qa#validation-modes" "validation modes" >}} available

![Create task with validation mode](/images/honeypot09.jpg)

5. Create the task
6. Upload or create Ground Truth annotations in the Ground Truth job in the task
7. Switch the Ground Truth job into the `acceptance` stage and `completed` state

![Set job status](/images/honeypot10.jpg)

## How to enable quality control for an already existing task

> For already existing tasks only the Ground Truth validation mode is available. If you want
> to use Honeypots for your task, you will need to recreate the task.

1. Open the task page
2. Click the "+" button next to the job list

![Create job](/images/honeypot01.jpg)

3. Select Job Type **Ground truth** and configure the job parameters

![Configure job parameters](/images/honeypot02.jpg)

4. Upload or create Ground Truth annotations in the Ground Truth job in the task
5. Switch the Ground Truth job into the `acceptance`stage and `completed` state

![Set job status](/images/honeypot10.jpg)

## How to enable immediate job feedback

1. Open the task **Actions** menu > **Quality control** > **Settings**
2. Set **Max validations per job** to above zero. 3 is a good starting number.

![Configure job validations](/images/immediate-feedback-settings-validations.jpg)

3. Save the updated settings
4. Assign an annotator to an annotation job
5. Annotate the job
6. Mark the job finished using the corresponding button in the menu
7. Once the job is completed, you'll see the job validation dialog

![Job validation successful](/images/immediate-feedback-accept.png)

Each assignee gets no more than the specified number of validation attempts.

## How to check task quality metrics

1. Open the task **Actions** menu > **Quality control**
2. (optional) Request quality metrics computation, wait for completion
3. Review summaries or detailed reports

![Quality Analytics page](/images/honeypot05.jpg)

## How to review problems found

1. Open the task **Actions** menu > **Quality control**
2. Find an annotation job to be reviewed, it must have at least 1 validation frame
3. Click the job link
4. Switch to the **Review** mode
5. Enable display of Ground Truth annotations and conflicts

![GT conflict](/images/honeypot06.gif)
Binary file added site/content/en/images/honeypot09.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added site/content/en/images/honeypot10.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added site/content/en/images/honeypot11.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added site/content/en/images/honeypot12.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
80 changes: 80 additions & 0 deletions site/content/en/images/quality_comparison_bbox1.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading