Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Visual search update release #214

Merged
merged 16 commits into from
Dec 13, 2022
Merged

Visual search update release #214

merged 16 commits into from
Dec 13, 2022

Conversation

jn2clark
Copy link
Contributor

@jn2clark jn2clark commented Dec 9, 2022

  • What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)
    It introduces:
  1. additional heuristic methods for the image preprocessing via chunking (or patching) (see https://marqo.pages.dev/Preprocessing/Images/)
  2. additional models for the image preprocessing via chunking (or patching) (see https://marqo.pages.dev/Preprocessing/Images/)
  3. a text-image based reranker that allows reranking of images based on an open-vocabulary model (owl-vit)
  4. An example script and streamlit app to demonstrate the new functionality
  5. tests for the new functionality
  • What is the current behavior? (You can also link to an open issue here)
  1. Currently we have two methods for image based pre-processing. A simple grid based heuristic and a faster-rcnn based model from pytorch (trained on mscoco)
  2. There is no option for open-vocabulary localization or re-ranking based on images
  • What is the new behavior (if this is a feature change)?
  1. The new behavior is there are more options to select a model for pre-proessing. A yolox based onnx model trained on LVIS is added, models based on determining salient regions of images from attention maps is included (based on DINO)
  2. There is a new heuristic based model which uses overlapping boxes instead of a non-overlapping grid.
  3. Re-ranking of images from text based queries is possible

  • Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)
    No

  • Have unit tests been run against this PR? (Has there also been any additional testing?)
    Yes

  • Related Python client changes (link commit/PR here)
    None

  • Related documentation changes (link commit/PR here)
    To be included

  • Other information:
    api tests are here
    https://github.com/marqo-ai/marqo-api-tests/tree/visual_search_update
    Manual testing end-to-end via the demo has been performed as well

This is an update to the previous one. Mainline was remerged into the branch.
#205

  • Please check if the PR fulfills these requirements
  • The commit message follows our guidelines
  • Tests for the changes have been added (for bug fixes/features)
  • Docs have been added / updated (for bug fixes / features)

jn2clark and others added 14 commits December 8, 2022 11:18
@jn2clark
Visual search update (#210)
c90b57e 
* add parametrisation for chunking, overlapping boxes and combined model+boxes

* add more tests

* integrate yolox and simple grid changes

* helper functions for opencv and yolox

* add yolox patch class and helper functions

* update to use opencv

* update model cache and add logging

* fix model caching and device selection

* device conflicts

* add attention based bb generation

* update bboxes and test yolo

* refactor and add attention based ViT for bb determination

* create dino specific utils file for vit attention

* include dino files

* split the image file into seperate utils

* refactor, split into seperate files and use a proper base class

* add more tests

* update and add more tests

* refactor and clean up

* add more packages

* clean up and refactor

* update tests

* docker and cloud versions

* change

* fix device for owl

* update types and dco strings

* rename file

* rename file

* move tests

* pytorch utils test

* add another error for model loading

* update functions to handle some edge cases and update types

* add more tests

* update the yolox utils to download the proper model

* add yolox specific tests

* update reqs and setup to be on the latest

* update dockerfile to be same as new one

* clean up

* add the example and app

* update file locations

* update demo

* bump model version

* update demo

* minor text edits

* update demo

* update demo

* better error handling

* update tests

* change to PIL error

* add more error types

* small fixes for errors

* error handling

* update tests

* remove models

* clean up, doc strings and formatting

* update tests

* minor formatting

Co-authored-by: Jesse Clark <jesse@s2search.io>
@jn2clark
update some function names after merge
bd4da2a
@jn2clark
clean up and rename
debc93a
@jn2clark @pandu-k
Pinning tox ci (#211)
89e31cd 
* Branch aware ci tests (#209)

* set the MQ_API_TEST_BRANCH to the current branch

* setting github ref to just branch name

* Adding quotes around env var

* fixed syntax error

* parsing the github.ref string

* exporting just before running

* added image_to_test input

* fix: typo

* default image to test is now explicit

* updated documentation for image_to_test var

* Update unit_test_CI.yml

* tox pinned to 3.26

Co-authored-by: pandu-k <107458762+pandu-k@users.noreply.github.com>
Co-authored-by: Pandu Oliver Kerr <pandu@s2search.io>
@jn2clark
Update Dockerfile
6db50c1 
remove space
@jn2clark
clean up
f644d43
@jn2clark
move to headless opencv
f4d066a
@jn2clark
Merge branch 'visual_search_update_release' of https://github.com/mar…
11350f8 
…qo-ai/marqo into visual_search_update_release
tidying up
16debe2
update error
cb2c4bb
minor edits
1045e4a
fix PR feedback and add more descriptions in the docstrings
306c090
use literal type
42a31c4
clean up, make the names more descriptive
1f3c7d4
@jn2clark jn2clark requested a review from pandu-k December 9, 2022 05:02
@jn2clark
Copy link
Contributor Author

jn2clark commented Dec 9, 2022

Previous PR #205

@jn2clark jn2clark mentioned this pull request Dec 9, 2022
Closed
3 tasks
@jn2clark jn2clark temporarily deployed to marqo-test-suite December 9, 2022 05:09 — with GitHub Actions Inactive
@jn2clark jn2clark temporarily deployed to marqo-test-suite December 9, 2022 05:34 — with GitHub Actions Inactive
@jn2clark jn2clark temporarily deployed to marqo-test-suite December 9, 2022 05:34 — with GitHub Actions Inactive
@jn2clark jn2clark temporarily deployed to marqo-test-suite December 9, 2022 05:34 — with GitHub Actions Inactive
@jn2clark jn2clark temporarily deployed to marqo-test-suite December 9, 2022 05:34 — with GitHub Actions Inactive
@jn2clark jn2clark temporarily deployed to marqo-test-suite December 9, 2022 05:34 — with GitHub Actions Inactive
@jn2clark
Copy link
Contributor Author

jn2clark commented Dec 9, 2022

image

@jn2clark jn2clark marked this pull request as ready for review December 11, 2022 22:54
@jn2clark jn2clark mentioned this pull request Dec 13, 2022
Merged
3 tasks
@pandu-k
Copy link
Collaborator

pandu-k commented Dec 13, 2022

Strange git diff between this PR and mainline .github/workflows files. Manually checking diffs between files:

  • arm64_local_os: identical
  • cuda DIND: identical
  • cuda OS: identical
  • local OS: identical
  • s2search backend: identical
  • unit tests

@pandu-k pandu-k temporarily deployed to marqo-test-suite December 13, 2022 12:22 — with GitHub Actions Inactive
@pandu-k
Merge branch 'mainline' into visual_search_update_release
f858433 
# Conflicts:
#	src/marqo/s2_inference/errors.py
#	src/marqo/tensor_search/tensor_search.py
#	tests/s2_inference/test_encoding.py
#	tests/s2_inference/test_utils.py
@pandu-k pandu-k temporarily deployed to marqo-test-suite December 13, 2022 12:56 — with GitHub Actions Inactive
@pandu-k pandu-k temporarily deployed to marqo-test-suite December 13, 2022 12:57 — with GitHub Actions Inactive
@pandu-k pandu-k temporarily deployed to marqo-test-suite December 13, 2022 12:57 — with GitHub Actions Inactive
@pandu-k pandu-k temporarily deployed to marqo-test-suite December 13, 2022 12:57 — with GitHub Actions Inactive
@pandu-k pandu-k temporarily deployed to marqo-test-suite December 13, 2022 12:58 — with GitHub Actions Inactive
@pandu-k pandu-k temporarily deployed to marqo-test-suite December 13, 2022 12:59 — with GitHub Actions Inactive
@jn2clark
Copy link
Contributor Author

Sorry @pandu-k . I thought I had avoided those weird things be merging mainline back in.

@pandu-k pandu-k merged commit 590e8d8 into mainline Dec 13, 2022
@pandu-k pandu-k deleted the visual_search_update_release branch December 13, 2022 22:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pandu-k pandu-k pandu-k approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jn2clark @pandu-k