[Feat] litellm.acompletion() make Langfuse success handler non blocking #1519
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
|
This PR is being deployed to Railway 🚅 litellm: ◻️ REMOVED |
litellm/utils.py
Outdated
| threading.Thread( | ||
| target=logging_obj.success_handler, args=(result, start_time, end_time) | ||
| ).start() | ||
| threading.Thread( |
There was a problem hiding this comment.
@ishaan-jaff how does switching create task for threads solve the issue?
i'm also concerned about creating too many threads here, which would cause issues in high-traffic
There was a problem hiding this comment.
@krrishdholakia I update the conversation of this PR with notes
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
litellm.acompletion()callsWhy was this caused ?
langfuse.flush()in the callback, this is a blocking I/O Operation https://langfuse.com/docs/sdk/python#shutdown-behaviorasyncio.create_task, we NEED to use threads for langfuse logging, sincelangfuse.flush()is blockingThe async callbacks, (langfuse in this case) was not truly async, which was blocking execution
Solution
Move back to using Threads for running langfuse logging, instead of using
asyncio.create_taskHow to Prevent this in the future
Added a test to make 5
litellm.acompletion()calls to langfuse logger, and 5 to non langfuse. Asserting the delta is less than 1 secondNotes:
asyncio.create_task(coro)is used for scheduling coroutines for execution on the event loop. Langfuse logging was not truly async, due to which execution would get blockedThis PR also Adds support for tagging
cache_hitson Langfuse(note you need langfuse>=2.6.3