Add support for bedrock meta.llama3-3-70b-instruct-v1:0 tool calling #8512
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
26e7d70
into
BerriAI:litellm_bedrock_llama_3_3_tool_calling
2 checks passed
ishaan-jaff
added a commit
that referenced
this pull request
Feb 14, 2025
…support + cost tracking + base llm unit test for tool calling (#8545) * Add support for bedrock meta.llama3-3-70b-instruct-v1:0 tool calling (#8512) * fix(converse_transformation.py): fixing bedrock meta.llama3-3-70b tool calling * test(test_bedrock_completion.py): adding llama3.3 tool compatibility check * add TestBedrockTestSuite * add bedrock llama 3.3 to base llm class * us.meta.llama3-3-70b-instruct-v1:0 * test_basic_tool_calling * TestAzureOpenAIO1 * test_basic_tool_calling * test_basic_tool_calling --------- Co-authored-by: miraclebakelaser <65143272+miraclebakelaser@users.noreply.github.com>
krrishdholakia
pushed a commit
that referenced
this pull request
Feb 15, 2025
…support + cost tracking + base llm unit test for tool calling (#8545) * Add support for bedrock meta.llama3-3-70b-instruct-v1:0 tool calling (#8512) * fix(converse_transformation.py): fixing bedrock meta.llama3-3-70b tool calling * test(test_bedrock_completion.py): adding llama3.3 tool compatibility check * add TestBedrockTestSuite * add bedrock llama 3.3 to base llm class * us.meta.llama3-3-70b-instruct-v1:0 * test_basic_tool_calling * TestAzureOpenAIO1 * test_basic_tool_calling * test_basic_tool_calling --------- Co-authored-by: miraclebakelaser <65143272+miraclebakelaser@users.noreply.github.com>
krrishdholakia
added a commit
that referenced
this pull request
Feb 15, 2025
* refactor(organization.tsx): initial commit with orgs tab refactor make it similar to 'Teams' tab - simplifies org management actions * fix(page.tsx): pass user orgs to component * fix(organization_view.tsx): fix to pull info from org info endpoint * feat(organization_endpoints.py): return org members when calling /org/info * fix(organization_view.tsx): show org members on info page * feat(organization_view.tsx): allow adding user to org via user email Resolves #8330 * fix(organization_endpoints.py): raise better error when duplicate user_email found in db * fix(organization_view.tsx): cleanup user_email for now not in initial org info - will need to prefetch * fix(page.tsx): fix getting user models on page load allows passing down the user models to org * fix(organizations.tsx): fix creating org on ui * fix(proxy/_types.py): include org created at and updated at cleans up ui * fix(navbar.tsx): cleanup * fix(organizations.tsx): fix tpm/rpm limits on org * fix(organizations.tsx): fix linting error * fix(organizations.tsx): fix linting \ * (Feat) - Add `/bedrock/meta.llama3-3-70b-instruct-v1:0` tool calling support + cost tracking + base llm unit test for tool calling (#8545) * Add support for bedrock meta.llama3-3-70b-instruct-v1:0 tool calling (#8512) * fix(converse_transformation.py): fixing bedrock meta.llama3-3-70b tool calling * test(test_bedrock_completion.py): adding llama3.3 tool compatibility check * add TestBedrockTestSuite * add bedrock llama 3.3 to base llm class * us.meta.llama3-3-70b-instruct-v1:0 * test_basic_tool_calling * TestAzureOpenAIO1 * test_basic_tool_calling * test_basic_tool_calling --------- Co-authored-by: miraclebakelaser <65143272+miraclebakelaser@users.noreply.github.com> * fix(general_settings.tsx): filter out empty dictionaries post fallback delete (#8550) Fixes #8331 * bump: version 1.61.3 → 1.61.4 * (perf) Fix memory leak on `/completions` route (#8551) * initial mem util test * fix _cached_get_model_info_helper * test memory usage * fix tests * fix mem usage --------- Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> Co-authored-by: miraclebakelaser <65143272+miraclebakelaser@users.noreply.github.com>
ishaan-jaff
added a commit
that referenced
this pull request
Feb 15, 2025
…support + cost tracking + base llm unit test for tool calling (#8545) * Add support for bedrock meta.llama3-3-70b-instruct-v1:0 tool calling (#8512) * fix(converse_transformation.py): fixing bedrock meta.llama3-3-70b tool calling * test(test_bedrock_completion.py): adding llama3.3 tool compatibility check * add TestBedrockTestSuite * add bedrock llama 3.3 to base llm class * us.meta.llama3-3-70b-instruct-v1:0 * test_basic_tool_calling * TestAzureOpenAIO1 * test_basic_tool_calling * test_basic_tool_calling --------- Co-authored-by: miraclebakelaser <65143272+miraclebakelaser@users.noreply.github.com>
abhijitherekar
pushed a commit
to acuvity/litellm
that referenced
this pull request
Feb 20, 2025
…support + cost tracking + base llm unit test for tool calling (BerriAI#8545) * Add support for bedrock meta.llama3-3-70b-instruct-v1:0 tool calling (BerriAI#8512) * fix(converse_transformation.py): fixing bedrock meta.llama3-3-70b tool calling * test(test_bedrock_completion.py): adding llama3.3 tool compatibility check * add TestBedrockTestSuite * add bedrock llama 3.3 to base llm class * us.meta.llama3-3-70b-instruct-v1:0 * test_basic_tool_calling * TestAzureOpenAIO1 * test_basic_tool_calling * test_basic_tool_calling --------- Co-authored-by: miraclebakelaser <65143272+miraclebakelaser@users.noreply.github.com>
abhijitherekar
pushed a commit
to acuvity/litellm
that referenced
this pull request
Feb 20, 2025
* refactor(organization.tsx): initial commit with orgs tab refactor make it similar to 'Teams' tab - simplifies org management actions * fix(page.tsx): pass user orgs to component * fix(organization_view.tsx): fix to pull info from org info endpoint * feat(organization_endpoints.py): return org members when calling /org/info * fix(organization_view.tsx): show org members on info page * feat(organization_view.tsx): allow adding user to org via user email Resolves BerriAI#8330 * fix(organization_endpoints.py): raise better error when duplicate user_email found in db * fix(organization_view.tsx): cleanup user_email for now not in initial org info - will need to prefetch * fix(page.tsx): fix getting user models on page load allows passing down the user models to org * fix(organizations.tsx): fix creating org on ui * fix(proxy/_types.py): include org created at and updated at cleans up ui * fix(navbar.tsx): cleanup * fix(organizations.tsx): fix tpm/rpm limits on org * fix(organizations.tsx): fix linting error * fix(organizations.tsx): fix linting \ * (Feat) - Add `/bedrock/meta.llama3-3-70b-instruct-v1:0` tool calling support + cost tracking + base llm unit test for tool calling (BerriAI#8545) * Add support for bedrock meta.llama3-3-70b-instruct-v1:0 tool calling (BerriAI#8512) * fix(converse_transformation.py): fixing bedrock meta.llama3-3-70b tool calling * test(test_bedrock_completion.py): adding llama3.3 tool compatibility check * add TestBedrockTestSuite * add bedrock llama 3.3 to base llm class * us.meta.llama3-3-70b-instruct-v1:0 * test_basic_tool_calling * TestAzureOpenAIO1 * test_basic_tool_calling * test_basic_tool_calling --------- Co-authored-by: miraclebakelaser <65143272+miraclebakelaser@users.noreply.github.com> * fix(general_settings.tsx): filter out empty dictionaries post fallback delete (BerriAI#8550) Fixes BerriAI#8331 * bump: version 1.61.3 → 1.61.4 * (perf) Fix memory leak on `/completions` route (BerriAI#8551) * initial mem util test * fix _cached_get_model_info_helper * test memory usage * fix tests * fix mem usage --------- Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> Co-authored-by: miraclebakelaser <65143272+miraclebakelaser@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Title
Add support for bedrock meta.llama3-3-70b-instruct-v1:0 tool calling
Relevant issues
Fixes #8094
Type
🐛 Bug Fix
Changes
litellm/llms/bedrock/chat/converse_transformation.py
def get_supported_openai_params(self, model: str) -> List[str]: supported_params = [ "max_tokens", "max_completion_tokens", "stream", "stream_options", "stop", "temperature", "top_p", "extra_headers", "response_format", ] ## Filter out 'cross-region' from model name base_model = BedrockModelInfo.get_base_model(model) if ( base_model.startswith("anthropic") or base_model.startswith("mistral") or base_model.startswith("cohere") or base_model.startswith("meta.llama3-1") or base_model.startswith("meta.llama3-2") + or base_model.startswith("meta.llama3-3") or base_model.startswith("amazon.nova") ): supported_params.append("tools") if litellm.utils.supports_tool_choice( model=model, custom_llm_provider=self.custom_llm_provider ): # only anthropic and mistral support tool choice config. otherwise (E.g. cohere) will fail the call - https: supported_params.append("tool_choice") return supported_paramsTests