[CVAT] Adapt exchange/recording oracles for honeypots

[Marishka17]

Description

Adapted CVAT Exchange and Recording oracles to use the recently introduced CVAT honeypots.

Summary of changes

Exchange oracle changes:

• The logic to select validation frames for injection into frames to be annotated has been moved into CVAT. CVAT tasks are now created with a validation layout (gt_pool) and a specified number of validation frames per job, defined by manifest.validation.val_size or manifest.validation.val_size * job_size_mult for skeleton tasks. Random sorting is used on CVAT when creating a task.
• Waiting unitl task is created is moved partially into process_incoming_job_launcher_webhooks cron job to be able upload GT (CVAT_TASK_CREATION_CHECK_INTERVAL env is added (default 5s)).
• Oracle prepares GT annotations based on the GT dataset shared by the requester and uploads them to the CVAT GT job from the task. The DEFAULT_IMPORT_TIMEOUT environment is introduced to set the upper limit of waiting when GT annotations are uploaded. The default is 1h.
• Oracle configures quality control settings for each CVAT task (there are several new env variables):
  • CVAT_MAX_VALIDATION_CHECKS - defines the maximum number of validation checks to be performed when checking job immediate feedback on CVAT (default is 3). Keep this number small enough to avoid honeypot recognition by annotators
  • CVAT_IOU_THRESHOLD (default 0.8)
  • CVAT_OKS_SIGMA (default 0.1, for skeletons task)
• The annotation meta created by exchange oracle and shared with recording oracle now includes new fields for each job meta:
  • task_id (cvat task id)
  • start and stop frames (obtained from CVAT)
• Introduced a new setting CVAT_MAX_JOBS_PER_TASK (default 1000) that defines the maximum number of jobs that can be created in a cvat task. This allows us to avoid potential problems when working with unpaginated endpoints.
• Fixed an issue where GT frames might be missing from a job
• The Job model now includes new non-nullable fields: start_frame and stop_frame. That means that db must be reset. @Dzeranov
• Validation approach for the image_points task changed to compare user annotations with center points of GT boxes for compatibility with CVAT. This is going to be less accurate, so may lead to increased false positives and negatives during validation. Annotations are not changed.

Recording oracle updates:

• CVAT’s quality control functionality is now used to determine the quality of job annotations. The custom logic for comparing datasets across different task types has been removed.
• The oracle now interacts directly with CVAT, using the same environment variables as the exchange oracle (CVAT_URL, CVAT_ADMIN, CVAT_ADMIN_PASS, and CVAT_ORG_SLUG) (+ added cvat_sdk dependency).
• The oracle doesn't use job annotations from the internal Exchange Oracle bucket anymore
• Jobs accepted on the previous epochs are no longer assessed again
• It's not possible now to have only some jobs unverifiable. Now, it's an escrow property, so either the whole escrow is verifiable or not. This depends on the number of remaining valid honeypots.

How test the changes

Related issues

Keywords for linking issues

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
human-app	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 13, 2024 6:27pm
human-dashboard-frontend	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 13, 2024 6:27pm

2 Skipped Deployments

Name	Status	Preview	Comments	Updated (UTC)
faucet-frontend	⬜️ Ignored (Inspect)	Visit Preview		Nov 13, 2024 6:27pm
faucet-server	⬜️ Ignored (Inspect)	Visit Preview		Nov 13, 2024 6:27pm

[zhiltsov-max]

Need to implement the idea from the 16c67b0 description