v1.18.9

What's Changed

• docs: Fix import statement for provider's sample code by @kihaya in #1535
• Litellm GitHub action build admin UI by @ShaunMaher in #1505
• [Feat] Proxy Auth - Use custom_key_generate by @ishaan-jaff in #1538
• [Fix] Router - Usage Based Routing with fallbacks (Track the correct tpm/rpm) by @ishaan-jaff in #1555
• [Feat] support custom cost tracking per second by @krrishdholakia in #1551

New Contributors

• @kihaya made their first contribution in #1535

Full Changelog: v1.18.7...v1.18.9

v1.18.8

What's Changed

• [Feat] Add typehints for litellm.Router by @ishaan-jaff in #1529
• [Feat] Litellm.Router set custom cooldown times by @ishaan-jaff in #1534

Full Changelog: v1.18.6...v1.18.8

v1.18.7

What's Changed

1. Improved litellm.Router logging for fallbacks

Example Log for one call

LiteLLM Router - INFO: get_available_deployment for model: azure/gpt-4-fast, No deployment available
LiteLLM Router - INFO: litellm.completion(model=None) Exception No models available.
LiteLLM Router - INFO: get_available_deployment for model: azure/gpt-4-basic, No deployment available
LiteLLM Router - INFO: litellm.completion(model=None) Exception No models available.
LiteLLM Router - INFO: get_available_deployment for model: openai-gpt-4, Selected deployment: {'model_name': 'openai-gpt-4', 'litellm_params': {'model': 'gpt-3.5-turbo', 'api_key': 'sk-PhEM****', 'tpm': 2000}, 'tpm': 2000, 'model_info': {'id': '5a4b95fa-c018-4767-85c2-c4851c57cf34'}} for model: openai-gpt-4
LiteLLM Router - INFO: litellm.completion(model=gpt-3.5-turbo) 200 OK

How to use in python

router = litellm.Router(
            model_list=model_list,
            fallbacks=fallbacks_list,
            set_verbose=True,
            debug_level="DEBUG" # optional, default=INFO
)

2. Improvements to Usage Based Routing - litellm.Router

Before making the first call, we check if any of the deployments have TPM to make the call, Thanks @georgeseifada for this!

3. [Feat] Add typehints for litellm.Router by @ishaan-jaff in #1529

Full Changelog: v1.18.6...v1.18.7

v1.18.6

What's Changed

1.[Feat] litellm.acompletion() make Langfuse success handler non blocking by @ishaan-jaff in #1519

• The Langfuse Success Callback was blocking running litellm.acompletion() calls. fixed on this release
• Support for logging Cache Hits on Langfuse:
  support for tagging cache_hits on Langfuse
  (note you need langfuse>=2.6.3

2. Langsmith: Add envs for project/run names; fix bug with None metadata by @timothyasp in #1524

[Feat] Router improvements by @ishaan-jaff in #1525

3. Allow overriding headers for anthropic by @keeganmccallum in #1513

• fix(utils.py): add metadata to logging obj on setup, if exists (fixes max parallel request bug) by @krrishdholakia in #1531

4. test(tests/): add unit testing for proxy server endpoints by @krrishdholakia in f5ced08

• fix(proxy_server.py): users can now see their key info by @krrishdholakia in f5ced08
• fix(proxy_server.py): model info now restricts for user access by @krrishdholakia in f5ced08

New Contributors

• @timothyasp made their first contribution in #1524
• @keeganmccallum made their first contribution in #1513

Full Changelog: v1.18.5...v1.18.6

v1.18.5

What's Changed

• add headers to budget manager by @HaruHunab1320 in #1506
• Adds s3_path prefix so that we can save objects in predifined location in s3 bucket by @duarteocarmo in #1499
• nit: switch to valid SPDX license identifier MIT in pyproject.toml by @ErikBjare in #1515

New Contributors

• @HaruHunab1320 made their first contribution in #1506

Full Changelog: v1.18.4...v1.18.5

v1.18.4

What's Changed

[Feat] Proxy - Add Spend tracking logs by @ishaan-jaff in #1498

New SpendTable when Using LiteLLM Virtual Keys - Logs API Key, CreatedAt Date + Time, Model, Spend, Messages, Response
Docs to get started: https://docs.litellm.ai/docs/proxy/virtual_keys

[Feat] Proxy - Track Cost Per User (Using user passed to requests) by @ishaan-jaff in #1509

• Proxy Server Track Cost Per User
  Request:

curl --location 'http://0.0.0.0:8000/chat/completions' \
        --header 'Content-Type: application/json' \
        --header 'Authorization: Bearer sk-RwPq' \
        --data ' {
        "model": "BEDROCK_GROUP",
        "user": "litellm-is-awesome-user",
        "messages": [
            {
            "role": "user",
            "content": "what llm are you-444"
            }
        ],
        }'

Cost Tracked in LiteLLM Spend Tracking DB

Notes:

• If a user is passed to the request the proxy tracks cost for it
• If the user does not exist in the User Table, we make a new user with the spend

feat(parallel_request_limiter.py): add support for tpm/rpm rate limits for keys by @krrishdholakia in #1501

Full Changelog: v1.18.3...v1.18.4

v1.18.3

What's Changed

• [Feat] /key/generate - create keys withteam_id by @ishaan-jaff in #1500
  Set team_id when creating users, keys
  docs: https://docs.litellm.ai/docs/proxy/virtual_keys#request

curl 'http://0.0.0.0:8000/key/generate' \
--header 'Authorization: Bearer <your-master-key>' \
--header 'Content-Type: application/json' \
--data-raw '{
  "models": ["gpt-3.5-turbo", "gpt-4", "claude-2"],
  "team_id": "core-infra"
}'

• (fix) read azure ad token from optional params extra body by @krrishdholakia in e0aaa94

Full Changelog: v1.18.2...v1.18.3

v1.18.2

What's Changed

• [Test+Fix] /Key/Info, /Key/Update - Litellm unit test key endpoints by @ishaan-jaff in #1496
• fix(ollama_chat.py): use tiktoken as backup for prompt token counting by @puffo in #1495
• fix(parallel_request_limiter.py): decrement count for failed llm calls by @krrishdholakia in 1ea3833
• fix(proxy_server.py): show all models user has access to in /models by @krrishdholakia in c8dd36d

New Contributors

• @puffo made their first contribution in #1495

Full Changelog: v1.18.1...v1.18.2

v1.18.1

What's Changed

• Altered requirements.txt to require pyyaml 6.0.1 which will resolve #1488 by @ShaunMaher in #1489
• Update s3 cache to support folder by @duarteocarmo in #1494

New Contributors

• @ShaunMaher made their first contribution in #1489
• @duarteocarmo made their first contribution in #1494

Full Changelog: v1.18.0...v1.18.1

v1.18.0

What's Changed

https://docs.litellm.ai/docs/simple_proxy

• [Feat] Proxy - Access Key metadata in callbacks by @ishaan-jaff in #1484
  • Access Proxy Key metadata in callbacks
  • Access Endpoint URL in calbacks - you can see if /chat/completions, /embeddings, /image/generation etc is called
  • Support for Langfuse Tags, We log request metadata as langfuse tags

PS. no keys leaked - these are keys to my local proxy

Support for model access groups

Use this if you have keys with access to specific models, and you want to give all them access to a new model.

You can now assign keys access to model groups, and add new models to that group via the config.yaml - https://docs.litellm.ai/docs/proxy/users#grant-access-to-new-model

curl --location 'http://localhost:8000/key/generate' \
-H 'Authorization: Bearer <your-master-key>' \
-H 'Content-Type: application/json' \
-d '{"models": ["beta-models"], # 👈 Model Access Group
            "max_budget": 0,}'

Langfuse Tags logged:

* feat(proxy_server.py): support model access groups by @krrishdholakia in https://github.com//pull/1483

Full Changelog: v1.17.18...v1.18.0

What's Changed

• [Feat] Proxy - Access Key metadata in callbacks by @ishaan-jaff in #1484
• feat(proxy_server.py): support model access groups by @krrishdholakia in #1483

Full Changelog: v1.17.18...v1.18.0