Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add quality modes and management docs #8732

Merged
merged 32 commits into from
Dec 4, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
d1b933e
Add quality modes and management docs
zhiltsov-max Nov 21, 2024
94e1e65
add annotation management section
zhiltsov-max Nov 21, 2024
110bc8d
Partially update auto-qa section
zhiltsov-max Nov 22, 2024
9998749
Update docs
zhiltsov-max Nov 25, 2024
d8ea6ea
Apply suggestions from code review
zhiltsov-max Nov 29, 2024
85a8e11
Fix some typos and grammar
zhiltsov-max Nov 29, 2024
c521a9a
Resolve some comments and linter errors
zhiltsov-max Nov 29, 2024
3597da8
Describe parameters in modes more precisely
zhiltsov-max Nov 29, 2024
7754cf2
Improve parameter descriptions for honeypots
zhiltsov-max Nov 29, 2024
bff1094
Apply suggestions from code review
zhiltsov-max Dec 2, 2024
21a753f
Remove duplicating toc
zhiltsov-max Dec 2, 2024
8eefe52
Move gt management section
zhiltsov-max Dec 2, 2024
881a287
Fix list formatting
zhiltsov-max Dec 2, 2024
4becbb3
Add info about quality checks
zhiltsov-max Dec 2, 2024
f944e09
Update quality settings section
zhiltsov-max Dec 2, 2024
1b1cc6d
Update mode summary table
zhiltsov-max Dec 2, 2024
9198d7f
Add configuration link
zhiltsov-max Dec 2, 2024
4fce7ad
Fix some errors
zhiltsov-max Dec 2, 2024
61191a5
Improve conflict review section
zhiltsov-max Dec 2, 2024
5b9dbdf
Clear heading
zhiltsov-max Dec 2, 2024
a26addc
Add information on comparisons for annotations
zhiltsov-max Dec 2, 2024
a6a0ef6
Add info on quality reports
zhiltsov-max Dec 2, 2024
9696230
Improve quality analytics section
zhiltsov-max Dec 2, 2024
d5eb4da
Merge remote-tracking branch 'origin/zm/quality-docs' into zm/quality…
zhiltsov-max Dec 2, 2024
1787ab3
Fix linter warnings
zhiltsov-max Dec 2, 2024
fe3373c
Add some tabs
zhiltsov-max Dec 2, 2024
f872f18
Apply suggestions from code review
zhiltsov-max Dec 4, 2024
4dfcd99
Improve table layout
zhiltsov-max Dec 4, 2024
f41859d
Improve navigation in qa overview and immediate feedback sections
zhiltsov-max Dec 4, 2024
2f97d2d
Refactor immediate feedback section
zhiltsov-max Dec 4, 2024
54c57fc
Fix types, move tracks section
zhiltsov-max Dec 4, 2024
064a4c1
Compress some images
zhiltsov-max Dec 4, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 56 additions & 25 deletions site/content/en/docs/enterprise/immediate-feedback.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,49 +2,80 @@
title: 'Immediate job feedback'
linkTitle: 'Immediate job feedback'
weight: 5
description: 'This feature provides annotators with general feedback on their performance in a job.'
description: 'Quick responses about job annotation quality'
---

When an annotator finishes a job, a dialog is displayed showing the quality of their annotations.
The annotator can either agree or disagree with the feedback.
## Overview

The basic idea behind this feature is to provide annotators with quick feedback on their
performance in a job. When an annotator finishes a job, a dialog is displayed showing the
quality of their annotations. The annotator can either agree or disagree with the feedback.
If they disagree, they have the option to re-annotate the job and request feedback again.
However, feedback is only available a limited number of times, as specified in the task's quality settings.

To ensure transparency with the annotator, the immediate feedback shows the collected score and
the minimum required score.
Immediate feedback settings, such as `Target metric`, `Target metric threshold`,
`Max validations per job` and others, can be configured on the quality settings page:
<!--- TODO: Update quality page documentation and refer from this section -->
To ensure transparency with the annotator, the immediate feedback shows the computed score and
the minimum required score. Information about the specific errors or frames that have errors is
not available to annotators.

Feedback is only available a limited number of times for each assignment, to prevent
Ground Truth revealing by annotators. This is controlled by a configurable parameter, so
it can be adjusted to the requirements of each project.

## How to configure

Immediate feedback settings, such as `Target metric`, `Target metric threshold`,
`Max validations per job` and others, can be configured on the quality settings page.

This feature is considered enabled if the `Max validations per job` is above 0. You can change
the parameters any time.

> **Note**: This feature requires a configured validation set in the task. Read more
> in the
> {{< ilink "/docs/manual/basics/quality-control#how-to-enable-quality-control" "quality overview" >}}
> section or in the
{{< ilink "/docs/manual/advanced/analytics-and-monitoring/auto-qa#configuring-quality-estimation" "full guide" >}}.

1. Open the task **Actions** menu > **Quality control** > **Settings**

![Configure job validations](/images/immediate-feedback-quality-settings.png)

2. Set the `Target metric` and `Target metric threshold` values to what is required in your project.
3. Set **Max validations per job** to above zero. 3 is a good starting number.
4. Save the updated settings

<img src="/images/immediate-feedback-quality-settings.png" style="max-width: 600px;">
## How to receive a feedback

1. Assign an annotator to an annotation job
2. Annotate the job
3. Mark the job finished using the corresponding button in the menu
4. Once the job is completed, you'll see the job validation dialog

<img src="/images/immediate-feedback-accept.png" style="max-width: 500px;">

Each assignee gets no more than the specified number of validation attempts.

> **Note**: this functionality is only available in regular annotation jobs. For instance,
> it's not possible to use it in Ground Truth jobs.

### Available feedbacks

There are three types of feedbacks available for different cases:
- Accepted
- Rejected, but can be adjusted
- Rejected, with an option to fix mistakes
- Finally rejected when the number of attempts is exhausted

<img src="/images/immediate-feedback-accept.png" style="max-width: 300px;">
<img src="/images/immediate-feedback-reject.png" style="max-width: 300px;">
<img src="/images/immediate-feedback-final-reject.png" style="max-width: 300px;">

Notes:
## Additional details

> Immediate feedback has a default timeout of 20 seconds.
Feedback may be unavailable for large jobs or when there are too many immediate feedback requests.
In this case annotators do not see any feedback dialogs.
> Feedback may be unavailable for large jobs or when there are too many immediate feedback requests.
> In this case annotators do not see any feedback dialogs and annotate jobs as
> if the feature was disabled.

> The number of attempts does not decrease for staff members who have access to a job with ground truth annotations.
> The number of attempts does not decrease for staff members who have access to a job
> with ground truth annotations. For instance, if you're trying to test this feature as the task
> owner, you may be confused if you see the number of attempts doesn't decrease.

> The number of attempts resets when the job assignee is updated.

Requirements:
1. The task is configured with a Ground Truth job that has been annotated,
moved to the acceptance stage, and is in the completed state.
2. The current job is in the annotation stage.
3. The current job is a regular annotation job. Immediate feedback is not available for Ground Truth jobs
4. The `Max validations per job` setting has been configured on the quality settings page.



Loading
Loading