MM-ReAct (Multimodal Reasoning and Action)

[slavakurilyak]

For GPT-4, image inputs are still in limited alpha.

For GPT-3.5, it would be great to see LangChain use the MM-ReAct agent.

• Repo: github.com/microsoft/MM-REACT
• Website: multimodal-react.github.io
• Demo: huggingface.co/spaces/microsoft-cognitive-service/mm-react
• Paper Abstract: arxiv.org/abs/2303.11381
• Paper PDF: arxiv.org/pdf/2303.11381.pdf

[aryansid]

Is it available on langchain now?

[slavakurilyak]

GPT-4(Vision) is now available which unlocks multi-modal reasoning and action.

According to OpenAI:

We’re excited to roll out these capabilities to other groups of users, including developers, soon after.

This model is currently available using the web UI but API access is inevitable.

[alifanov]

llava (multimodal llama) is open sourced now

[slavakurilyak]

Llava is now merged into Llama.cpp here

LangChain provides bindings to Llama.cpp here and here

I'm closing this issue since Llava is a great alternative to MM-ReAct

[rlancemartin]

It's true that LLaVA merged w/ llama.cpp.

But, we also need multi-modal integration w/ lama-cpp-python to run LLaVA directly in LangChain.

I put a ticket up for it:
abetlen/llama-cpp-python#813

Worth keeping this open until that is in place.

I merged a notebook that runs LLaVA (bash script) w/o the py integration:
#11582

[Se-Hun]

Now, we can use gpt-4-vision-preview model.

https://platform.openai.com/docs/guides/vision

[slavakurilyak]

According to OpenAI:

GPT-4 with vision is currently available to all developers who have access to GPT-4 via the gpt-4-vision-preview model and the Chat Completions API which has been updated to support image inputs

Since multimodal LLMs like GPT-4V are now generally available, I'm closing this issue