-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for BLIP-2 multimodal feature extraction #25474
base: main
Are you sure you want to change the base?
Add support for BLIP-2 multimodal feature extraction #25474
Conversation
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
cc @amyeroberts and @younesbelkada |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome work @youssefadr ! let us know when the PR is ready for review
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
Hey @youssefadr, feel free to re-open this if you still plan on adding this feature |
@ArthurZucker Thank you, I'll do it, I am sorry I couldn't find time lately for it. It is on my backlog! |
@NielsRogge Hi, should I open a new PR for this, or is there a way to re-open this one? It seems to be requested by a lot of users. |
Is this still being worked on? I am in need of this tool. |
it's stale, feel free to take it over |
What does this PR do?
This PR introduces the addition of
get_image_feature
andget_text_feature
methods to theBlip2ForConditionalGeneration
class. These changes align with the original Qformer implementation, which utilized both text and image inputs.The current implementation in HuggingFace lacks support for multimodal embeddings, especially the capacity to extract embeddings by passing both text and image to the QFormer. This PR addresses this shortcoming.
Fixes #25300 #25245
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.